Rclone v1.64.0 release

Releases
1

Rclone 1.64.0 has been released. Find it in the rclone downloads or use rclone selfupdate to upgrade.

Some highlights of the release:

  • New backends: Proton Drive, Quatrix
  • Rework of multi-thread transfers
    • Support for s3, azureblob, b2, oracleobjectstorage and smb
  • rclone config redacted to make support easier!

Thank you to all the contributors to this release, those who contributed code or doc fixes (49 people) or made issues or answered questions in the forum - your help is much appreciated!

In particular thank you to our new maintainers @henrybear327 (Proton Drive), @nielash (bisync) and @boukendesho (snap packaging).

Thank you also to our new sponsor Storj - please check out their innovative and well priced offerings.

If you’re working in the industry and you feel your company could benefit from a support contract please get in touch at sales@rclone.com or if you’re interested in advertising with us then please drop us a message to sponsorship@rclone.com.

Please consider donating or sponsoring to keep the project sustainable.

v1.64.0 - 2023-09-11

See commits

  • New backends
  • Major changes
    • Multi-thread transfers (Vitor Gomes, Nick Craig-Wood, Manoj Ghosh, Edwin Mackenzie-Owen)
      • Multi-thread transfers are now available when transferring to:
        • local, s3, azureblob, b2, oracleobjectstorage and smb
      • This greatly improves transfer speed between two network sources.
      • In memory buffering has been unified between all backends and should share memory better.
      • See --multi-thread docs for more info
  • New commands
    • rclone config redacted support mechanism for showing redacted config (Nick Craig-Wood)
  • New Features
    • accounting
      • Show server side stats in own lines and not as bytes transferred (Nick Craig-Wood)
    • bisync
      • Add new --ignore-listing-checksum flag to distinguish from --ignore-checksum (nielash)
      • Add experimental --resilient mode to allow recovery from self-correctable errors (nielash)
      • Add support for --create-empty-src-dirs (nielash)
      • Dry runs no longer commit filter changes (nielash)
      • Enforce --check-access during --resync (nielash)
      • Apply filters correctly during deletes (nielash)
      • Equality check before renaming (leave identical files alone) (nielash)
      • Fix dryRun rc parameter being ignored (nielash)
    • build
      • Update to go1.21 and make go1.19 the minimum required version (Anagh Kumar Baranwal, Nick Craig-Wood)
      • Update dependencies (Nick Craig-Wood)
      • Add snap installation (hideo aoyama)
      • Change Winget Releaser job to ubuntu-latest (sitiom)
    • cmd: Refactor and use sysdnotify in more commands (eNV25)
    • config: Add --multi-thread-chunk-size flag (Vitor Gomes)
    • doc updates (antoinetran, Benjamin, Bjørn Smith, Dean Attali, gabriel-suela, James Braza, Justin Hellings, kapitainsky, Mahad, Masamune3210, Nick Craig-Wood, Nihaal Sangha, Niklas Hambüchen, Raymond Berger, r-ricci, Sawada Tsunayoshi, Tiago Boeing, Vladislav Vorobev)
    • fs
      • Use atomic types everywhere (Roberto Ricci)
      • When --max-transfer limit is reached exit with code (10) (kapitainsky)
      • Add rclone completion powershell - basic implementation only (Nick Craig-Wood)
    • http servers: Allow CORS to be set with --allow-origin flag (yuudi)
    • lib/rest: Remove unnecessary nil check (Eng Zer Jun)
    • ncdu: Add keybinding to rescan filesystem (eNV25)
    • rc
      • Add executeId to job listings (yuudi)
      • Add core/du to measure local disk usage (Nick Craig-Wood)
      • Add operations/settier to API (Drew Stinnett)
    • rclone test info: Add --check-base32768 flag to check can store all base32768 characters (Nick Craig-Wood)
    • rmdirs: Remove directories concurrently controlled by --checkers (Nick Craig-Wood)
  • Bug Fixes
    • accounting: Don't stop calculating average transfer speed until the operation is complete (Jacob Hands)
    • fs: Fix transferTime not being set in JSON logs (Jacob Hands)
    • fshttp: Fix --bind 0.0.0.0 allowing IPv6 and --bind ::0 allowing IPv4 (Nick Craig-Wood)
    • operations: Fix overlapping check on case insensitive file systems (Nick Craig-Wood)
    • serve dlna: Fix MIME type if backend can't identify it (Nick Craig-Wood)
    • serve ftp: Fix race condition when using the auth proxy (Nick Craig-Wood)
    • serve sftp: Fix hash calculations with --vfs-cache-mode full (Nick Craig-Wood)
    • serve webdav: Fix error: Expecting fs.Object or fs.Directory, got nil (Nick Craig-Wood)
    • sync: Fix lockup with --cutoff-mode=soft and --max-duration (Nick Craig-Wood)
  • Mount
    • fix: Mount parsing for linux (Anagh Kumar Baranwal)
  • VFS
    • Add --vfs-cache-min-free-space to control minimum free space on the disk containing the cache (Nick Craig-Wood)
    • Added cache cleaner for directories to reduce memory usage (Anagh Kumar Baranwal)
    • Update parent directory modtimes on vfs actions (David Pedersen)
    • Keep virtual directory status accurate and reduce deadlock potential (Anagh Kumar Baranwal)
    • Make sure struct field is aligned for atomic access (Roberto Ricci)
  • Local
    • Rmdir return an error if the path is not a dir (zjx20)
  • Azure Blob
    • Implement OpenChunkWriter and multi-thread uploads (Nick Craig-Wood)
    • Fix creation of directory markers (Nick Craig-Wood)
    • Fix purging with directory markers (Nick Craig-Wood)
  • B2
    • Implement OpenChunkWriter and multi-thread uploads (Nick Craig-Wood)
    • Fix rclone link when object path contains special characters (Alishan Ladhani)
  • Box
    • Add polling support (David Sze)
    • Add --box-impersonate to impersonate a user ID (Nick Craig-Wood)
    • Fix unhelpful decoding of error messages into decimal numbers (Nick Craig-Wood)
  • Chunker
    • Update documentation to mention issue with small files (Ricardo D'O. Albanus)
  • Compress
    • Fix ChangeNotify (Nick Craig-Wood)
  • Drive
    • Add --drive-fast-list-bug-fix to control ListR bug workaround (Nick Craig-Wood)
  • Fichier
    • Implement DirMove (Nick Craig-Wood)
    • Fix error code parsing (alexia)
  • FTP
    • Add socks_proxy support for SOCKS5 proxies (Zach)
    • Fix 425 "TLS session of data connection not resumed" errors (Nick Craig-Wood)
  • Hdfs
    • Retry "replication in progress" errors when uploading (Nick Craig-Wood)
    • Fix uploading to the wrong object on Update with overriden remote name (Nick Craig-Wood)
  • HTTP
    • CORS should not be sent if not set (yuudi)
    • Fix webdav OPTIONS response (yuudi)
  • Opendrive
    • Fix List on a just deleted and remade directory (Nick Craig-Wood)
  • Oracleobjectstorage
    • Use rclone's rate limiter in mutipart transfers (Manoj Ghosh)
    • Implement OpenChunkWriter and multi-thread uploads (Manoj Ghosh)
  • S3
    • Refactor multipart upload to use OpenChunkWriter and ChunkWriter (Vitor Gomes)
    • Factor generic multipart upload into lib/multipart (Nick Craig-Wood)
    • Fix purging of root directory with --s3-directory-markers (Nick Craig-Wood)
    • Add rclone backend set command to update the running config (Nick Craig-Wood)
    • Add rclone backend restore-status command (Nick Craig-Wood)
  • SFTP
    • Stop uploads re-using the same ssh connection to improve performance (Nick Craig-Wood)
    • Add --sftp-ssh to specify an external ssh binary to use (Nick Craig-Wood)
    • Add socks_proxy support for SOCKS5 proxies (Zach)
    • Support dynamic --sftp-path-override (nielash)
    • Fix spurious warning when using --sftp-ssh (Nick Craig-Wood)
  • Smb
    • Implement multi-threaded writes for copies to smb (Edwin Mackenzie-Owen)
  • Storj
    • Performance improvement for large file uploads (Kaloyan Raev)
  • Swift
    • Fix HEADing 0-length objects when --swift-no-large-objects set (Julian Lepinski)
  • Union
    • Add :writback to act as a simple cache (Nick Craig-Wood)
  • WebDAV
    • Nextcloud: fix segment violation in low-level retry (Paul)
  • Zoho
    • Remove Range requests workarounds to fix integration tests (Nick Craig-Wood)
14 Likes
4

PS: Proton Drive beta testers - I changed the config key name for the mailbox password at the very last minute before the release to make it consistent with rclone's other config keys - so you'll need to rename it in your config.

This also applies to 2FA so you'll need to do 2FA again - sorry!

I think it should just work if you rename the keys in the config file like this

old new
clientUID client_uid
clientAccessToken client_access_token
clientRefreshToken client_refresh_token
clientSaltedKeyPass client_salted_key_pass
mailboxPassword mailbox_password
1 Like
5

Nice big list! :heart_eyes:

1 Like
7

@ncw

I was using rclone beta with proton support. It worked fine. I just updated to this version. I tried renaming the config as you mentioned. rclone was giving the error I see below. I went ahead and deleted the protondrive config completely and started over with a new config.

First time trying to use protondrive with the new config, the same way I had previously set it up, gives me the following error on first try.

rclone lsd Proton: --protondrive-2fa=xxxxxx
2023/09/13 08:18:11.119855 ERROR RESTY 401 GET https://mail.proton.me/api/core/v4/users: Invalid access token (Code=401, Status=401), Attempt 1
2023/09/13 08:18:11.334837 ERROR RESTY 422 POST https://mail.proton.me/api/auth/v4/refresh: Invalid refresh token (Code=10013, Status=422), Attempt 1

It hangs at this point. I get the notification from Proton that rclone has logged into my account yet I do not get a directory listing. If I ctrl-c out of it and just do an rclone -ls Proton: it hangs as well.

rclone ls Proton: --log-level DEBUG
2023/09/13 08:24:32 DEBUG : rclone: Version "v1.64.0" starting with parameters ["rclone" "ls" "Proton:" "--log-level" "DEBUG"]
2023/09/13 08:24:32 DEBUG : Creating backend with remote "Proton:"
2023/09/13 08:24:32 DEBUG : Using config file from "/home/jim/.config/rclone/rclone.conf"
2023/09/13 08:24:32 DEBUG : proton drive root link ID '': Has cached credentials
2023/09/13 08:24:35 DEBUG : proton drive root link ID '': Used cached credential to initialize the ProtonDrive API

But it hangs here.

8

Not sure what has happened there.

The old config should work if you rename things as shown in this commit

Maybe @henrybear327 has an idea?

9

@ncw

I got it working again. I deleted the remote again. Used a completely different name for it. Hopefully, it was just a me issue.

1 Like
10

i know i'm kind of the only person waiting for this, but the new VFS can't function as a cached remote yet, can it?

11

Not sure what you mean as I use full mode cache without a problem for some time now. Best to start a new post if you have a new request or probably one that's already on out there.

13

the caching of a remote locally so it doesn't need to be scanned for each sync. nick has a prototype, so i'm guessing if it were released it would be one of the major points on the release :slight_smile: it used to be doable with the cache remote. but as per your post here, the current cache remote should no longer be used.

14

@jared alas this is still a prototype. Hopefully one day soon it will make it out the dev incubator :slight_smile:

15

thanks for stopping by. i'll keep waiting. :crossed_fingers:

16

Yep, that is correct.

There a few issues that you may want to follow:

Is probably the best I'd imagine if I'm following.

17

Sorry for the late reply.

No, I don't have an idea either. I can't reproduce this case on my side for some reason. I will try again tomorrow to see if there is anything that I might have missed :slight_smile:

18

Hi,

I noticed that after an initial usage of proton drive session refresh keys are generated and stored. I removed the username, password and 2fa code (expired anyway) and everything keeps on working.

I do not know what happens after an expiry time but would it not be best to have an initial connection when creating the config to get the keys and never even store password and 2fa code?

Rgrds A

19

I think @henrybear327 has plan to remove the username and password from the config file. It is only used at the start to get the token.

20

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.