Rclone v1.68.0 release

Rclone 1.68.0 has been released. Find it in the rclone downloads or use rclone selfupdate to upgrade.

Some highlights of the release:

Thank you to all the contributors to this release, those who contributed code or doc fixes (39 people) or made issues or answered questions in the forum - your help is much appreciated!

If you’re working in the industry and you feel your company could benefit from a support contract please get in touch at sales@rclone.com or if you’re interested in advertising with us then please drop us a message to sponsorship@rclone.com.

Please consider donating or sponsoring to keep the project sustainable and a big thank you to our existing sponsors.

v1.68.0 - 2024-09-08

See commits

  • New backends
  • Changed backends
    • S3 backend updated to use AWS SDKv2 as v1 is now unsupported.
      • The matrix of providers and auth methods is huge and there could be problems with obscure combinations.
      • Please report problems in a new issue on Github.
  • New commands
    • config encryption: set, remove and check to manage config file encryption (Nick Craig-Wood)
  • New Features
    • build
      • Update to go1.23 and make go1.21 the minimum required version (Nick Craig-Wood)
      • Update all dependencies (Nick Craig-Wood)
      • Disable wasm/js build due to go bug #64856 (Nick Craig-Wood)
      • Enable custom linting rules with ruleguard via gocritic (albertony)
      • Update logging statements to make --use-json-log work always (albertony)
      • Adding new code quality tests and fixing the fallout (albertony)
    • config
      • Internal config re-organised to be more consistent and make it available from the rc (Nick Craig-Wood)
      • Avoid remotes with empty names from the environment (albertony)
      • Make listing of remotes more consistent (albertony)
      • Make getting config values more consistent (albertony)
      • Use --password-command to set config file password if supplied (Nick Craig-Wood)
    • doc fixes (albertony, crystalstall, David Seifert, Eng Zer Jun, Ernie Hershey, Florian Klink, John Oxley, kapitainsky, Mathieu Moreau, Nick Craig-Wood, nipil, Pétr Bozsó, Russ Bubley, Sam Harrison, Thearas, URenko, Will Miles, yuval-cloudinary)
    • fs: Allow semicolons as well as spaces in --bwlimit timetable parsing (Kyle Reynolds)
    • help
      • Global flags help command now takes glob filter (albertony)
      • Make help command output less distracting (albertony)
    • lib/encoder: Add Raw encoding for use where no encoding at all is required, eg --local-encoding Raw (URenko)
    • listremotes: Added options for filtering, ordering and json output (albertony)
    • nfsmount
      • Make the --sudo flag work for umount as well as mount (Nick Craig-Wood)
      • Add -o tcp option to NFS mount options to fix mounting under Linux (Nick Craig-Wood)
    • operations: copy: generate stable partial suffix (Georg Welzel)
    • rc
      • Add options/info call to enumerate options (Nick Craig-Wood)
      • Add option blocks parameter to options/get and options/info (Nick Craig-Wood)
      • Add vfs/queue to show the status of the upload queue (Nick Craig-Wood)
      • Add vfs/queue-set-expiry to adjust expiry of items in the VFS queue (Nick Craig-Wood)
      • Add --unix-socket option to rc command (Florian Klink)
      • Prevent unmount rc command from sending a STOPPING=1 sd-notify message (AThePeanut4)
    • rcserver: Implement prometheus metrics on a dedicated port (Oleg Kunitsyn)
    • serve dlna
      • Also look at "Subs" subdirectory (Florian Klink)
      • Don't swallow video.{idx,sub} (Florian Klink)
      • Set more correct mime type (Florian Klink)
    • serve nfs
      • Implement on disk cache for file handles selected with --nfs-cache-type (Nick Craig-Wood)
      • Add tracing to filesystem calls (Nick Craig-Wood)
      • Mask unimplemented error from chmod (Nick Craig-Wood)
      • Unify the nfs library logging with rclone's logging better (Nick Craig-Wood)
      • Fix incorrect user id and group id exported to NFS (Nick Craig-Wood)
    • serve s3
      • Implement --auth-proxy (Sawjan Gurung)
      • Update to AWS SDKv2 by updating github.com/rclone/gofakes3 (Nick Craig-Wood)
  • Bug Fixes
    • bisync: Fix sync time problems with backends that round time (eg Dropbox) (nielash)
    • serve dlna: Fix panic: invalid argument to Int63n (Nick Craig-Wood)
  • VFS
    • Add --vfs-read-chunk-streams to parallel read chunks from files (Nick Craig-Wood)
      • This can increase mount performance on high bandwidth or large latency links
    • Fix cache encoding with special characters (URenko)
  • Local
    • Fix encoding of root path fix (URenko)
    • Add server-side copy (using clone) with xattrs on macOS (nielash)
      • --local-no-clone flag to disable cloning for server-side copies (nielash)
    • Support setting custom --metadata during server-side Copy (nielash)
  • Azure Blob
    • Allow anonymous access for public resources (Nick Craig-Wood)
  • B2
    • Include custom upload headers in large file info (Pat Patterson)
  • Drive
    • Fix copying Google Docs to a backend which only supports SHA1 (Nick Craig-Wood)
  • Fichier
    • Fix detection of Flood Detected error (Nick Craig-Wood)
    • Fix server side move (Nick Craig-Wood)
  • HTTP
    • Reload client certificates on expiry (Saleh Dindar)
    • Support listening on passed FDs (Florian Klink)
  • Jottacloud
    • Fix setting of metadata on server side move (albertony)
  • Onedrive
    • Fix nil pointer error when uploading small files (Nick Craig-Wood)
  • Pcloud
    • Implement SetModTime (Georg Welzel)
    • Implement OpenWriterAt feature to enable multipart uploads (Georg Welzel)
  • Pikpak
    • Improve data consistency by ensuring async tasks complete (wiserain)
    • Implement custom hash to replace wrong sha1 (wiserain)
    • Fix error with copyto command (wiserain)
    • Optimize file move by removing unnecessary readMetaData() call (wiserain)
    • Non-buffered hash calculation for local source files (wiserain)
    • Optimize upload by pre-fetching gcid from API (wiserain)
    • Correct file transfer progress for uploads by hash (wiserain)
    • Update to using AWS SDK v2 (wiserain)
  • S3
    • Update to using AWS SDK v2 (Nick Craig-Wood)
      • Add --s3-sdk-log-mode to control SDKv2 debugging (Nick Craig-Wood)
    • Fix incorrect region for Magalu provider (Filipe Herculano)
    • Allow restoring from intelligent-tiering storage class (Pawel Palucha)
  • SFTP
    • Use uint32 for mtime to save memory (Tomasz Melcer)
    • Ignore useless errors when closing the connection pool (Nick Craig-Wood)
    • Support listening on passed FDs (Florian Klink)
  • Swift
    • Add workarounds for bad listings in Ceph RGW (Paul Collins)
    • Add total/free space info in about command. (fsantagostinobietti)
  • Ulozto
    • Fix upload of > 2GB files on 32 bit platforms (Tobias Markus)
  • WebDAV
    • Add --webdav-unix-socket-path to connect to a unix socket (Florian Klink)
  • Yandex
    • Implement custom user agent to help with upload speeds (Sebastian Bünger)
  • Zoho
    • Fix inefficiencies uploading with new API to avoid throttling (Nick Craig-Wood)
13 Likes

Such a great tool, thank you!

4 Likes

Looking forward to running it from the Docker container latest :slight_smile:
Thank you for this great work!

1 Like

Ah yes, looks like the docker build failed. Will have to remember how to do it manually :grimacing:

1 Like

Thank you! I was able to pull the image. Thanks!

1 Like

Interesting; I jumped from 1.64.2 to 1.68.0 and then sync to my crypt with personal OneDrive looks like it re-uploaded everything (lots of "copy replaced existing" messages). The next run worked as expected.

I'm now running a cryptcheck to check it all looks good... so far so good.

I wonder if this was related to the metadata support added in 1.66

Odd. What rclone parameters are you using? It should say in the debug log why it is uploading stuff. Most likely cause is time shift. I think metadata shouldn't make any difference here.

It only did it the first time after the upgrade and I don't habitually run my overnight cronjobs with debug enabled, so I don't have a log entry for it. Just the log lines telling me it replaced everything.

Unlikely to be time shift 'cos some of the files it re-uploaded haven't changed for a year so the stat value won't have changed. My server is also ntp sync'd

The command used was rclone --delete-before --stats=0 sync . OneDriveBackups: --exclude dir1 --exclude dir2 --exclude dir3 ...

The relevant config entries:

[OneDrive]
type = onedrive
token = ...
drive_id = ...
drive_type = personal

[OneDriveBackups]
type = crypt
remote = OneDrive:Backups
filename_encryption = off
directory_name_encryption = false
password = ...
password2 = 

Here's one of the files in question; we can see from stat that it hasn't changed, but OneDrive thinks a new version was uploaded

  File: 'SYNC.onedrive.tst'
  Size: 1026            Blocks: 8          IO Block: 1048576 regular file
Device: 2ah/42d Inode: 73735       Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-09-15 10:18:53.253107066 -0400
Modify: 2022-07-15 18:18:34.389621555 -0400
Change: 2022-07-15 18:18:34.568621751 -0400
 Birth: -

image

Hmm, definitely an anomaly. Can't say for sure what happened without seeing the debug logs though, alas!

I guess it is possible onedrive had a glitch and didn't list those files so rclone thought they were missing and uploaded them.

Could it be a timezone problem? Again, unlikely I think.

I wondered about a Onedrive glitch, but it would have had to been a long glitch 'cos it did it over 6 different rclone runs, referencing 5 different crypt remotes which are on 3 different OneDrive accounts and the whole thing took 14 hours to complete... and coincidentally happened the first run after I updated :slight_smile:

I also wondered if Microsoft had rolled out some change which caused rclone to think things needed re-uploading, but that would likely have impacted other people.

Ah well. Just an oddity!

Are you using rclone union by any chance?

Nope. Just a simple crypt on a single OneDrive. I just have a lot of them set up :slight_smile:

The precision for the onedrive personal backend changed from 1s to 1ms. Normally this wouldn't be a problem but if you are using crypt then it will exhibit this upload everything again problem. See Issue #8090 for more info.

However you should be able to fix with a one time sync using the --refresh-times flag which will update the timestamps.

Thank you to everyone for this update, especially the update to pCloud - comes just to the right time.

If it is now possible to set the modTime, can we use --refresh-times flag together with sync to only update the times?

It also means the overview page can be updated to RW for pCloud and ModTime, can't it?

1 Like

A sync will set the mod times anyway I think, though --refresh-times will work with crypt so that is a good plan.

Well spotted!