Rclone 1.54 release

Rclone 1.54 has been released. Find it here https://rclone.org/downloads/

Some highlights of a particularly large release:

Thank you to all the contributors to this release, those who contributed code or doc fixes (71 people) or made issues or answered questions in the forum - your help is much appreciated!

Thank you also to Storage Made Easy for sponsoring the File Fabric backend and to Digitalis for sponsoring deglobalising the config.

A special thank you to @ivandeex who has been organising the Pull Request backlog - we are getting there :slight_smile:

If you (or your company) is enjoying rclone, please consider donating to keep the project sustainable, or hire me to add a feature you really want!

v1.54.0 - 2021-02-02

See commits

  • New backends
    • Compression remote (experimental) (buengese)
    • Enterprise File Fabric (Nick Craig-Wood)
    • HDFS (Hadoop Distributed File System) (Yury Stankevich)
    • Zoho workdrive (buengese)
  • New Features
    • Deglobalise the config (Nick Craig-Wood)
      • Global config now read from the context
      • This will enable passing of global config via the rc
      • This work was sponsored by Digitalis
    • Add --bwlimit for upload and download (Nick Craig-Wood)
      • Obey bwlimit in http Transport for better limiting
    • Enhance systemd integration (Hekmon)
      • log level identification, manual activation with flag, automatic systemd launch detection
      • Don't compile systemd log integration for non unix systems (Benjamin Gustin)
    • Add a --download flag to md5sum/sha1sum/hashsum to force rclone to download and hash files locally (lostheli)
    • Add --progress-terminal-title to print ETA to terminal title (LaSombra)
    • Make backend env vars show in help as the defaults for backend flags (Nick Craig-Wood)
    • build
      • Raise minimum go version to go1.12 (Nick Craig-Wood)
    • dedupe
      • Add --by-hash to dedupe on content hash not file name (Nick Craig-Wood)
      • Add --dedupe-mode list to just list dupes, changing nothing (Nick Craig-Wood)
      • Add warning if used on a remote which can't have duplicate names (Nick Craig-Wood)
    • fs
      • Add Shutdown optional method for backends (Nick Craig-Wood)
      • When using --files-from check files concurrently (zhucan)
      • Accumulate stats when using --dry-run (Ingo Weiss)
      • Always show stats when using --dry-run or --interactive (Nick Craig-Wood)
      • Add support for flag --no-console on windows to hide the console window (albertony)
    • genautocomplete: Add support to output to stdout (Ingo)
    • ncdu
      • Highlight read errors instead of aborting (Claudio Bantaloukas)
      • Add sort by average size in directory (Adam Plánský)
      • Add toggle option for average s3ize in directory - key 'a' (Adam Plánský)
      • Add empty folder flag into ncdu browser (Adam Plánský)
      • Add ! (errror) and . (unreadable) file flags to go with e (empty) (Nick Craig-Wood)
    • obscure: Make rclone osbcure - ignore newline at end of line (Nick Craig-Wood)
    • operations
      • Add logs when need to upload files to set mod times (Nick Craig-Wood)
      • Move and copy log name of the destination object in verbose (Adam Plánský)
      • Add size if known to skipped items and JSON log (Nick Craig-Wood)
    • rc
      • Prefer actual listener address if using ":port" or "addr:0" only (Nick Craig-Wood)
      • Add listener for finished jobs (Aleksandar Jankovic)
    • serve ftp: Add options to enable TLS (Deepak Sah)
    • serve http/webdav: Redirect requests to the base url without the / (Nick Craig-Wood)
    • serve restic: Implement object cache (Nick Craig-Wood)
    • stats: Add counter for deleted directories (Nick Craig-Wood)
    • sync: Only print "There was nothing to transfer" if no errors (Nick Craig-Wood)
    • webui
      • Prompt user for updating webui if an update is available (Chaitanya Bankanhal)
      • Fix plugins initialization (negative0)
  • Bug Fixes
    • fs
      • Fix nil pointer on copy & move operations directly to remote (Anagh Kumar Baranwal)
      • Fix parsing of .. when joining remotes (Nick Craig-Wood)
    • log: Fix enabling systemd logging when using --log-file (Nick Craig-Wood)
    • check
      • Make the error count match up in the log message (Nick Craig-Wood)
    • move: Fix data loss when source and destination are the same object (Nick Craig-Wood)
    • operations
      • Fix --cutof-mode hard not cutting off immediately (Nick Craig-Wood)
      • Fix --immutable error message (Nick Craig-Wood)
    • sync
      • Fix --cutoff-mode soft & cautious so it doesn't end the transfer early (Nick Craig-Wood)
      • Fix --immutable errors retrying many times (Nick Craig-Wood)
  • Docs
    • Many fixes and a rewrite of the filtering docs (edwardxml)
    • Many spelling and grammar fixes (Josh Soref)
    • Doc fixes for commands delete, purge, rmdir, rmdirs and mount (albertony)
    • And thanks to these people for many doc fixes too numerous to list
      • Ameer Dawood, Antoine GIRARD, Bob Bagwill, Christopher Stewart
      • CokeMine, David, Dov Murik, Durval Menezes, Evan Harris, gtorelly
      • Ilyess Bachiri, Janne Johansson, Kerry Su, Marcin Zelent,
      • Martin Michlmayr, Milly, Sơn Trần-Nguyễn
  • Mount
    • Update systemd status with cache stats (Hekmon)
    • Disable bazil/fuse based mount on macOS (Nick Craig-Wood)
      • Make rclone mount actually run rclone cmount under macOS (Nick Craig-Wood)
    • Implement mknod to make NFS file creation work (Nick Craig-Wood)
    • Make sure we don't call umount more than once (Nick Craig-Wood)
    • More user friendly mounting as network drive on windows (albertony)
    • Detect if uid or gid are set in same option string: -o uid=123,gid=456 (albertony)
    • Don't attempt to unmount if fs has been destroyed already (Nick Craig-Wood)
  • VFS
    • Fix virtual entries causing deleted files to still appear (Nick Craig-Wood)
    • Fix "file already exists" error for stale cache files (Nick Craig-Wood)
    • Fix file leaks with --vfs-cache-mode full and --buffer-size 0 (Nick Craig-Wood)
    • Fix invalid cache path on windows when using :backend: as remote (albertony)
  • Local
    • Continue listing files/folders when a circular symlink is detected (Manish Gupta)
    • New flag --local-zero-size-links to fix sync on some virtual filesystems (Riccardo Iaconelli)
  • Azure Blob
    • Add support for service principals (James Lim)
    • Add support for managed identities (Brad Ackerman)
    • Add examples for access tier (Bob Pusateri)
    • Utilize the streaming capabilities from the SDK for multipart uploads (Denis Neuling)
    • Fix setting of mime types (Nick Craig-Wood)
    • Fix crash when listing outside a SAS URL's root (Nick Craig-Wood)
    • Delete archive tier blobs before update if --azureblob-archive-tier-delete (Nick Craig-Wood)
    • Fix crash on startup (Nick Craig-Wood)
    • Fix memory usage by upgrading the SDK to v0.13.0 and implementing a TransferManager (Nick Craig-Wood)
    • Require go1.14+ to compile due to SDK changes (Nick Craig-Wood)
  • B2
    • Make NewObject use less expensive API calls (Nick Craig-Wood)
      • This will improve --files-from and restic serve in particular
    • Fixed crash on an empty file name (lluuaapp)
  • Box
    • Fix NewObject for files that differ in case (Nick Craig-Wood)
    • Fix finding directories in a case insentive way (Nick Craig-Wood)
  • Chunker
    • Skip long local hashing, hash in-transit (fixes) (Ivan Andreev)
    • Set Features ReadMimeType to false as Object.MimeType not supported (Nick Craig-Wood)
    • Fix case-insensitive NewObject, test metadata detection (Ivan Andreev)
  • Drive
    • Implement rclone backend copyid command for copying files by ID (Nick Craig-Wood)
    • Added flag --drive-stop-on-download-limit to stop transfers when the download limit is exceeded (Anagh Kumar Baranwal)
    • Implement CleanUp workaround for team drives (buengese)
    • Allow shortcut resolution and creation to be retried (Nick Craig-Wood)
    • Log that emptying the trash can take some time (Nick Craig-Wood)
    • Add xdg office icons to xdg desktop files (Pau Rodriguez-Estivill)
  • Dropbox
    • Add support for viewing shared files and folders (buengese)
    • Enable short lived access tokens (Nick Craig-Wood)
    • Implement IDer on Objects so rclone lsf etc can read the IDs (buengese)
    • Set Features ReadMimeType to false as Object.MimeType not supported (Nick Craig-Wood)
    • Make malformed_path errors from too long files not retriable (Nick Craig-Wood)
    • Test file name length before upload to fix upload loop (Nick Craig-Wood)
  • Fichier
    • Set Features ReadMimeType to true as Object.MimeType is supported (Nick Craig-Wood)
  • FTP
    • Add --ftp-disable-msld option to ignore MLSD for really old servers (Nick Craig-Wood)
    • Make --tpslimit apply (Nick Craig-Wood)
  • Google Cloud Storage
    • Storage class object header support (Laurens Janssen)
    • Fix anonymous client to use rclone's HTTP client (Nick Craig-Wood)
    • Fix Entry doesn't belong in directory "" (same as directory) - ignoring (Nick Craig-Wood)
  • Googlephotos
    • New flag --gphotos-include-archived to show archived photos as well (Nicolas Rueff)
  • Jottacloud
    • Don't erroneously report support for writing mime types (buengese)
    • Add support for Telia Cloud (Patrik Nordlén)
  • Mailru
    • Accept special folders eg camera-upload (Ivan Andreev)
    • Avoid prehashing of large local files (Ivan Andreev)
    • Fix uploads after recent changes on server (Ivan Andreev)
    • Fix range requests after June 2020 changes on server (Ivan Andreev)
    • Fix invalid timestamp on corrupted files (fixes) (Ivan Andreev)
    • Remove deprecated protocol quirks (Ivan Andreev)
  • Memory
    • Fix setting of mime types (Nick Craig-Wood)
  • Onedrive
    • Add support for China region operated by 21vianet and other regional suppliers (NyaMisty)
    • Warn on gateway timeout errors (Nick Craig-Wood)
    • Fall back to normal copy if server-side copy unavailable (Alex Chen)
    • Fix server-side copy completely disabled on OneDrive for Business (Cnly)
    • (business only) workaround to replace existing file on server-side copy (Alex Chen)
    • Enhance link creation with expiry, scope, type and password (Nick Craig-Wood)
    • Remove % and # from the set of encoded characters (Alex Chen)
    • Support addressing site by server-relative URL (kice)
  • Opendrive
    • Fix finding directories in a case insensitive way (Nick Craig-Wood)
  • Pcloud
    • Fix setting of mime types (Nick Craig-Wood)
  • Premiumizeme
    • Fix finding directories in a case insensitive way (Nick Craig-Wood)
  • Qingstor
    • Fix error propagation in CleanUp (Nick Craig-Wood)
    • Fix rclone cleanup (Nick Craig-Wood)
  • S3
    • Added --s3-disable-http2 to disable http/2 (Anagh Kumar Baranwal)
    • Complete SSE-C implementation (Nick Craig-Wood)
      • Fix hashes on small files with AWS:KMS and SSE-C (Nick Craig-Wood)
      • Add MD5 metadata to objects uploaded with SSE-AWS/SSE-C (Nick Craig-Wood)
    • Add --s3-no-head parameter to minimise transactions on upload (Nick Craig-Wood)
    • Update docs with a Reducing Costs section (Nick Craig-Wood)
    • Added error handling for error code 429 indicating too many requests (Anagh Kumar Baranwal)
    • Add requester pays option (kelv)
    • Fix copy multipart with v2 auth failing with 'SignatureDoesNotMatch' (Louis Koo)
  • SFTP
    • Allow cert based auth via optional pubkey (Stephen Harris)
    • Allow user to optionally check server hosts key to add security (Stephen Harris)
    • Defer asking for user passwords until the SSH connection succeeds (Stephen Harris)
    • Remember entered password in AskPass mode (Stephen Harris)
    • Implement Shutdown method (Nick Craig-Wood)
    • Implement keyboard interactive authentication (Nick Craig-Wood)
    • Make --tpslimit apply (Nick Craig-Wood)
    • Implement --sftp-use-fstat for unusual SFTP servers (Nick Craig-Wood)
  • Sugarsync
    • Fix NewObject for files that differ in case (Nick Craig-Wood)
    • Fix finding directories in a case insentive way (Nick Craig-Wood)
  • Swift
    • Fix deletion of parts of Static Large Object (SLO) (Nguyễn Hữu Luân)
    • Ensure partially uploaded large files are uploaded unless --swift-leave-parts-on-error (Nguyễn Hữu Luân)
  • Tardigrade
    • Upgrade to uplink v1.4.1 (Caleb Case)
  • WebDAV
    • Updated docs to show streaming to nextcloud is working (Durval Menezes)
  • Yandex
    • Set Features WriteMimeType to false as Yandex ignores mime types (Nick Craig-Wood)

OK, that's impressive.


i am sure there is something for everyone.

the SSE-C support is great for my use-case.

thanks much,

1 Like

There is lots of stuff in there certainly! I'm sorry this release is so long since the last stable release. I'm going to try to get them out every 2 months from now on...

I was asking few months ago about option, details are here:

and I wonder if that has been added or it is on to do list.

thanks for this big update and the many improvements!
The new metric counter for deleted directories rclone_dirs_deleted_total is really good for my grafana dashboard. Now I can see that a file OR a folder was deleted.


I think this is the relevant issue: https://github.com/rclone/rclone/issues/157

Yeah that is it. So as far as I understood feature is still in development ?

@lostheli, @ncw: you are mind readers of future needs. I just really needed rclone hashsum MD5 aws_s3_compatible:bucket --download (to the point that I was going to do a script with Python for my usage quickly). I saw it on rclone online help, I didn't work with my rclone v.1.53... and it's just released. Thanks very much! :slight_smile:

I'm testing it: [quote="carles, post:10, topic:22051"]
rclone hashsum MD5 aws_s3_compatible:bucket --download
I suspect that it downloads the files sequentially. Ideally I would like in parallel. For some of my buckets we have many, many small files and it would be great al in parallel. Not a big deal, when it finishes these directories it should speed up :slight_smile: if there is a hidden flag or version somewhere let me know of course! :slight_smile:

Checking with --progress it seems that files are downloaded in parallel, so all good! I might increase the number of simultaneous files for one of the buckets but... SUPER! :100:

We are still thinking about MD5 sum files and how we might fit them in.

Yes it should download them in parallel; how many is set with --transfers.

Glad you are finding this useful :slight_smile:

good news :grinning:
where can i find the document for the feature “ *This will enable passing of global config via the rc”?
another question, Are the two or more config from rc independent ?

I didn't quite manage to merge that feature yet.

I will merge it in a few days. You'll be able to pass _config into the rc.

You can see more here in this unmerged commit

Yes, the config will just be for that rc session.

Great job. Thanks for your endless efforts!

1 Like

I don't think it's a good solution。
Backend should be similar to Windows service,not just hide the window。
I also have such application scenarios,and use another application(NSSM)address my needs

Well it is a solution, but maybe not the best one

Hmm, so make a version of rclone which can run as a Windows service? Not a bad idea!

1 Like

For mount, it kind of is already.. You simply have to register it with PowerShell command New-Service. Think it gets the service integration from the WinFsp library, but have not tested it enough yet to say more about how it behaves.