Rclone v1.68.0 release

Releases
1

Rclone 1.68.0 has been released. Find it in the rclone downloads or use rclone selfupdate to upgrade.

Some highlights of the release:

Thank you to all the contributors to this release, those who contributed code or doc fixes (39 people) or made issues or answered questions in the forum - your help is much appreciated!

If you’re working in the industry and you feel your company could benefit from a support contract please get in touch at sales@rclone.com or if you’re interested in advertising with us then please drop us a message to sponsorship@rclone.com.

Please consider donating or sponsoring to keep the project sustainable and a big thank you to our existing sponsors.

v1.68.0 - 2024-09-08

See commits

  • New backends
  • Changed backends
    • S3 backend updated to use AWS SDKv2 as v1 is now unsupported.
      • The matrix of providers and auth methods is huge and there could be problems with obscure combinations.
      • Please report problems in a new issue on Github.
  • New commands
    • config encryption: set, remove and check to manage config file encryption (Nick Craig-Wood)
  • New Features
    • build
      • Update to go1.23 and make go1.21 the minimum required version (Nick Craig-Wood)
      • Update all dependencies (Nick Craig-Wood)
      • Disable wasm/js build due to go bug #64856 (Nick Craig-Wood)
      • Enable custom linting rules with ruleguard via gocritic (albertony)
      • Update logging statements to make --use-json-log work always (albertony)
      • Adding new code quality tests and fixing the fallout (albertony)
    • config
      • Internal config re-organised to be more consistent and make it available from the rc (Nick Craig-Wood)
      • Avoid remotes with empty names from the environment (albertony)
      • Make listing of remotes more consistent (albertony)
      • Make getting config values more consistent (albertony)
      • Use --password-command to set config file password if supplied (Nick Craig-Wood)
    • doc fixes (albertony, crystalstall, David Seifert, Eng Zer Jun, Ernie Hershey, Florian Klink, John Oxley, kapitainsky, Mathieu Moreau, Nick Craig-Wood, nipil, Pétr Bozsó, Russ Bubley, Sam Harrison, Thearas, URenko, Will Miles, yuval-cloudinary)
    • fs: Allow semicolons as well as spaces in --bwlimit timetable parsing (Kyle Reynolds)
    • help
      • Global flags help command now takes glob filter (albertony)
      • Make help command output less distracting (albertony)
    • lib/encoder: Add Raw encoding for use where no encoding at all is required, eg --local-encoding Raw (URenko)
    • listremotes: Added options for filtering, ordering and json output (albertony)
    • nfsmount
      • Make the --sudo flag work for umount as well as mount (Nick Craig-Wood)
      • Add -o tcp option to NFS mount options to fix mounting under Linux (Nick Craig-Wood)
    • operations: copy: generate stable partial suffix (Georg Welzel)
    • rc
      • Add options/info call to enumerate options (Nick Craig-Wood)
      • Add option blocks parameter to options/get and options/info (Nick Craig-Wood)
      • Add vfs/queue to show the status of the upload queue (Nick Craig-Wood)
      • Add vfs/queue-set-expiry to adjust expiry of items in the VFS queue (Nick Craig-Wood)
      • Add --unix-socket option to rc command (Florian Klink)
      • Prevent unmount rc command from sending a STOPPING=1 sd-notify message (AThePeanut4)
    • rcserver: Implement prometheus metrics on a dedicated port (Oleg Kunitsyn)
    • serve dlna
      • Also look at "Subs" subdirectory (Florian Klink)
      • Don't swallow video.{idx,sub} (Florian Klink)
      • Set more correct mime type (Florian Klink)
    • serve nfs
      • Implement on disk cache for file handles selected with --nfs-cache-type (Nick Craig-Wood)
      • Add tracing to filesystem calls (Nick Craig-Wood)
      • Mask unimplemented error from chmod (Nick Craig-Wood)
      • Unify the nfs library logging with rclone's logging better (Nick Craig-Wood)
      • Fix incorrect user id and group id exported to NFS (Nick Craig-Wood)
    • serve s3
      • Implement --auth-proxy (Sawjan Gurung)
      • Update to AWS SDKv2 by updating github.com/rclone/gofakes3 (Nick Craig-Wood)
  • Bug Fixes
    • bisync: Fix sync time problems with backends that round time (eg Dropbox) (nielash)
    • serve dlna: Fix panic: invalid argument to Int63n (Nick Craig-Wood)
  • VFS
    • Add --vfs-read-chunk-streams to parallel read chunks from files (Nick Craig-Wood)
      • This can increase mount performance on high bandwidth or large latency links
    • Fix cache encoding with special characters (URenko)
  • Local
    • Fix encoding of root path fix (URenko)
    • Add server-side copy (using clone) with xattrs on macOS (nielash)
      • --local-no-clone flag to disable cloning for server-side copies (nielash)
    • Support setting custom --metadata during server-side Copy (nielash)
  • Azure Blob
    • Allow anonymous access for public resources (Nick Craig-Wood)
  • B2
    • Include custom upload headers in large file info (Pat Patterson)
  • Drive
    • Fix copying Google Docs to a backend which only supports SHA1 (Nick Craig-Wood)
  • Fichier
    • Fix detection of Flood Detected error (Nick Craig-Wood)
    • Fix server side move (Nick Craig-Wood)
  • HTTP
    • Reload client certificates on expiry (Saleh Dindar)
    • Support listening on passed FDs (Florian Klink)
  • Jottacloud
    • Fix setting of metadata on server side move (albertony)
  • Onedrive
    • Fix nil pointer error when uploading small files (Nick Craig-Wood)
  • Pcloud
    • Implement SetModTime (Georg Welzel)
    • Implement OpenWriterAt feature to enable multipart uploads (Georg Welzel)
  • Pikpak
    • Improve data consistency by ensuring async tasks complete (wiserain)
    • Implement custom hash to replace wrong sha1 (wiserain)
    • Fix error with copyto command (wiserain)
    • Optimize file move by removing unnecessary readMetaData() call (wiserain)
    • Non-buffered hash calculation for local source files (wiserain)
    • Optimize upload by pre-fetching gcid from API (wiserain)
    • Correct file transfer progress for uploads by hash (wiserain)
    • Update to using AWS SDK v2 (wiserain)
  • S3
    • Update to using AWS SDK v2 (Nick Craig-Wood)
      • Add --s3-sdk-log-mode to control SDKv2 debugging (Nick Craig-Wood)
    • Fix incorrect region for Magalu provider (Filipe Herculano)
    • Allow restoring from intelligent-tiering storage class (Pawel Palucha)
  • SFTP
    • Use uint32 for mtime to save memory (Tomasz Melcer)
    • Ignore useless errors when closing the connection pool (Nick Craig-Wood)
    • Support listening on passed FDs (Florian Klink)
  • Swift
    • Add workarounds for bad listings in Ceph RGW (Paul Collins)
    • Add total/free space info in about command. (fsantagostinobietti)
  • Ulozto
    • Fix upload of > 2GB files on 32 bit platforms (Tobias Markus)
  • WebDAV
    • Add --webdav-unix-socket-path to connect to a unix socket (Florian Klink)
  • Yandex
    • Implement custom user agent to help with upload speeds (Sebastian Bünger)
  • Zoho
    • Fix inefficiencies uploading with new API to avoid throttling (Nick Craig-Wood)
14 Likes
3

Such a great tool, thank you!

4 Likes
4

Looking forward to running it from the Docker container latest :slight_smile:
Thank you for this great work!

1 Like
5

Ah yes, looks like the docker build failed. Will have to remember how to do it manually :grimacing:

1 Like
6

Thank you! I was able to pull the image. Thanks!

1 Like
7

Interesting; I jumped from 1.64.2 to 1.68.0 and then sync to my crypt with personal OneDrive looks like it re-uploaded everything (lots of "copy replaced existing" messages). The next run worked as expected.

I'm now running a cryptcheck to check it all looks good... so far so good.

I wonder if this was related to the metadata support added in 1.66

8

Odd. What rclone parameters are you using? It should say in the debug log why it is uploading stuff. Most likely cause is time shift. I think metadata shouldn't make any difference here.

9

It only did it the first time after the upgrade and I don't habitually run my overnight cronjobs with debug enabled, so I don't have a log entry for it. Just the log lines telling me it replaced everything.

Unlikely to be time shift 'cos some of the files it re-uploaded haven't changed for a year so the stat value won't have changed. My server is also ntp sync'd

The command used was rclone --delete-before --stats=0 sync . OneDriveBackups: --exclude dir1 --exclude dir2 --exclude dir3 ...

The relevant config entries:

[OneDrive]
type = onedrive
token = ...
drive_id = ...
drive_type = personal

[OneDriveBackups]
type = crypt
remote = OneDrive:Backups
filename_encryption = off
directory_name_encryption = false
password = ...
password2 =

Here's one of the files in question; we can see from stat that it hasn't changed, but OneDrive thinks a new version was uploaded

  File: 'SYNC.onedrive.tst'
  Size: 1026            Blocks: 8          IO Block: 1048576 regular file
Device: 2ah/42d Inode: 73735       Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-09-15 10:18:53.253107066 -0400
Modify: 2022-07-15 18:18:34.389621555 -0400
Change: 2022-07-15 18:18:34.568621751 -0400
 Birth: -

image

10

Hmm, definitely an anomaly. Can't say for sure what happened without seeing the debug logs though, alas!

I guess it is possible onedrive had a glitch and didn't list those files so rclone thought they were missing and uploaded them.

Could it be a timezone problem? Again, unlikely I think.

11

I wondered about a Onedrive glitch, but it would have had to been a long glitch 'cos it did it over 6 different rclone runs, referencing 5 different crypt remotes which are on 3 different OneDrive accounts and the whole thing took 14 hours to complete... and coincidentally happened the first run after I updated :slight_smile:

I also wondered if Microsoft had rolled out some change which caused rclone to think things needed re-uploading, but that would likely have impacted other people.

Ah well. Just an oddity!

12

Are you using rclone union by any chance?

13

Nope. Just a simple crypt on a single OneDrive. I just have a lot of them set up :slight_smile:

14

The precision for the onedrive personal backend changed from 1s to 1ms. Normally this wouldn't be a problem but if you are using crypt then it will exhibit this upload everything again problem. See Issue #8090 for more info.

However you should be able to fix with a one time sync using the --refresh-times flag which will update the timestamps.

15

Thank you to everyone for this update, especially the update to pCloud - comes just to the right time.

If it is now possible to set the modTime, can we use --refresh-times flag together with sync to only update the times?

It also means the overview page can be updated to RW for pCloud and ModTime, can't it?

1 Like
16

A sync will set the mod times anyway I think, though --refresh-times will work with crypt so that is a good plan.

Well spotted!

17

The pCloud OpenWriterAt feature has caused a serious issue with my copy jobs:

  • Transfer speed significantly reduced: 1.2 MiB/s compared to around 4.5 MiB/s with v1.67.
  • Failure to copy large (>20GB) files. This seems to occur when the files are closed, where rclone reports a size of 0 even though the copy was reported as complete.

I need to do a bit more analysis, but I have had to roll back to v1.67 in the meantime.

18

For reference. There are multiple reports of this issue emerging:

19

Thanks.... I drafted my comment last night, but only just got around to posting it. I should have done a quick search first!

20

I did not mean to criticize you posting. More reports better:) It makes bug case stronger.