Rclone 1.52 release

Releases
1

Rclone 1.52 has been released. Find it here https://rclone.org/downloads/

This is a bumper release - the 1.52 release has been delayed for some time due to general uncertainty! However I'm getting back to normal; adjusting to not having a "proper" job any more. I'm going to concentrate on being an Open Source developer so I'm accepting sponsorship at the new donate page.

Highlights - too many! Here is the top few:

  • Tardigrade backend for use with storj.io
  • Seafile backend for Seafile server
  • Union backend re-write to have multiple writable remotes
  • Full support for shortcuts on Google Drive
  • --header-download, --header-upload and --header flags for setting HTTP headers everywhere

Thank you to all the contributors to this release, those who contributed code or doc fixes (68 people!) or made issues or answered questions in the forum - your help is much appreciated!

Special thanks to Martin Michlmayr for proof reading and correcting all the docs and Edward Barker for helping re-write the front page.

v1.52.0 - 2020-05-27

See commits

  • New backends
  • New commands
    • backend: command for backend specific commands (see backends) (Nick Craig-Wood)
    • cachestats: Deprecate in favour of rclone backend stats cache: (Nick Craig-Wood)
    • dbhashsum: Deprecate in favour of rclone hashsum DropboxHash (Nick Craig-Wood)
  • New Features
    • Add --header-download and --header-upload flags for setting HTTP headers when uploading/downloading (Tim Gallant)
    • Add --header flag to add HTTP headers to every HTTP transaction (Nick Craig-Wood)
    • Add --check-first to do all checking before starting transfers (Nick Craig-Wood)
    • Add --track-renames-strategy for configurable matching criteria for --track-renames (Bernd Schoolmann)
    • Add --cutoff-mode hard,soft,catious (Shing Kit Chan & Franklyn Tackitt)
    • Filter flags (eg --files-from -) can read from stdin (fishbullet)
    • Add --error-on-no-transfer option (Jon Fautley)
    • Implement --order-by xxx,mixed for copying some small and some big files (Nick Craig-Wood)
    • Allow --max-backlog to be negative meaning as large as possible (Nick Craig-Wood)
    • Added --no-unicode-normalization flag to allow Unicode filenames to remain unique (Ben Zenker)
    • Allow --min-age/--max-age to take a date as well as a duration (Nick Craig-Wood)
    • Add rename statistics for file and directory renames (Nick Craig-Wood)
    • Add statistics output to JSON log (reddi)
    • Make stats be printed on non-zero exit code (Nick Craig-Wood)
    • When running --password-command allow use of stdin (SĂ©bastien Gross)
    • Stop empty strings being a valid remote path (Nick Craig-Wood)
    • accounting: support WriterTo for less memory copying (Nick Craig-Wood)
    • build
      • Update to use go1.14 for the build (Nick Craig-Wood)
      • Add -trimpath to release build for reproduceable builds (Nick Craig-Wood)
      • Remove GOOS and GOARCH from Dockerfile (Brandon Philips)
    • config
      • Fsync the config file after writing to save more reliably (Nick Craig-Wood)
      • Add --obscure and --no-obscure flags to config create/update (Nick Craig-Wood)
      • Make config show take remote: as well as remote (Nick Craig-Wood)
    • copyurl: Add --no-clobber flag (Denis)
    • delete: Added --rmdirs flag to delete directories as well (Kush)
    • filter: Added --files-from-raw flag (Ankur Gupta)
    • genautocomplete: Add support for fish shell (Matan Rosenberg)
    • log: Add support for syslog LOCAL facilities (Patryk Jakuszew)
    • lsjson: Add --hash-type parameter and use it in lsf to speed up hashing (Nick Craig-Wood)
    • rc
      • Add -o/--opt and -a/--arg for more structured input (Nick Craig-Wood)
      • Implement backend/command for running backend specific commands remotely (Nick Craig-Wood)
      • Add mount/mount command for starting rclone mount via the API (Chaitanya)
    • rcd: Add Prometheus metrics support (Gary Kim)
    • serve http
      • Added a --template flag for user defined markup (Rob Calistri)
      • Add Last-Modified headers to files and directories (Nick Craig-Wood)
    • serve sftp: Add support for multiple host keys by repeating --key flag (Maxime Suret)
    • touch: Add --localtime flag to make --timestamp localtime not UTC (Nick Craig-Wood)
  • Bug Fixes
    • accounting
      • Restore "Max number of stats groups reached" log line (MichaĹ‚ Matczuk)
      • Correct exitcode on Transfer Limit Exceeded flag. (Anuar Serdaliyev)
      • Reset bytes read during copy retry (Ankur Gupta)
      • Fix race clearing stats (Nick Craig-Wood)
    • copy: Only create empty directories when they don't exist on the remote (Ishuah Kariuki)
    • dedupe: Stop dedupe deleting files with identical IDs (Nick Craig-Wood)
    • oauth
      • Use custom http client so that --no-check-certificate is honored by oauth token fetch (Mark Spieth)
      • Replace deprecated oauth2.NoContext (Lars Lehtonen)
    • operations
      • Fix setting the timestamp on Windows for multithread copy (Nick Craig-Wood)
      • Make rcat obey --ignore-checksum (Nick Craig-Wood)
      • Make --max-transfer more accurate (Nick Craig-Wood)
    • rc
      • Fix dropped error (Lars Lehtonen)
      • Fix misplaced http server config (Xiaoxing Ye)
      • Disable duplicate log (ElonH)
    • serve dlna
      • Cds: don't specify childCount at all when unknown (Dan Walters)
      • Cds: use modification time as date in dlna metadata (Dan Walters)
    • serve restic: Fix tests after restic project removed vendoring (Nick Craig-Wood)
    • sync
      • Fix incorrect "nothing to transfer" message using --delete-before (Nick Craig-Wood)
      • Only create empty directories when they don't exist on the remote (Ishuah Kariuki)
  • Mount
    • Add --async-read flag to disable asynchronous reads (Nick Craig-Wood)
    • Ignore --allow-root flag with a warning as it has been removed upstream (Nick Craig-Wood)
    • Warn if --allow-non-empty used on Windows and clarify docs (Nick Craig-Wood)
    • Constrain to go1.13 or above otherwise bazil.org/fuse fails to compile (Nick Craig-Wood)
    • Fix fail because of too long volume name (evileye)
    • Report 1PB free for unknown disk sizes (Nick Craig-Wood)
    • Map more rclone errors into file systems errors (Nick Craig-Wood)
    • Fix disappearing cwd problem (Nick Craig-Wood)
    • Use ReaddirPlus on Windows to improve directory listing performance (Nick Craig-Wood)
    • Send a hint as to whether the filesystem is case insensitive or not (Nick Craig-Wood)
    • Add rc command mount/types (Nick Craig-Wood)
    • Change maximum leaf name length to 1024 bytes (Nick Craig-Wood)
  • VFS
    • Add --vfs-read-wait and --vfs-write-wait flags to control time waiting for a sequential read/write (Nick Craig-Wood)
    • Change default --vfs-read-wait to 20ms (it was 5ms and not configurable) (Nick Craig-Wood)
    • Make df output more consistent on a rclone mount. (Yves G)
    • Report 1PB free for unknown disk sizes (Nick Craig-Wood)
    • Fix race condition caused by unlocked reading of Dir.path (Nick Craig-Wood)
    • Make File lock and Dir lock not overlap to avoid deadlock (Nick Craig-Wood)
    • Implement lock ordering between File and Dir to eliminate deadlocks (Nick Craig-Wood)
    • Factor the vfs cache into its own package (Nick Craig-Wood)
    • Pin the Fs in use in the Fs cache (Nick Craig-Wood)
    • Add SetSys() methods to Node to allow caching stuff on a node (Nick Craig-Wood)
    • Ignore file not found errors from Hash in Read.Release (Nick Craig-Wood)
    • Fix hang in read wait code (Nick Craig-Wood)
  • Local
    • Speed up multi thread downloads by using sparse files on Windows (Nick Craig-Wood)
    • Implement --local-no-sparse flag for disabling sparse files (Nick Craig-Wood)
    • Implement rclone backend noop for testing purposes (Nick Craig-Wood)
    • Fix "file not found" errors on post transfer Hash calculation (Nick Craig-Wood)
  • Cache
    • Implement rclone backend stats command (Nick Craig-Wood)
    • Fix Server Side Copy with Temp Upload (Brandon McNama)
    • Remove Unused Functions (Lars Lehtonen)
    • Disable race tests until bbolt is fixed (Nick Craig-Wood)
    • Move methods used for testing into test file (greatroar)
    • Add Pin and Unpin and canonicalised lookup (Nick Craig-Wood)
    • Use proper import path go.etcd.io/bbolt (Robert-AndrĂ© Mauchin)
  • Crypt
    • Calculate hashes for uploads from local disk (Nick Craig-Wood)
      • This allows crypted Jottacloud uploads without using local disk
      • This means crypted s3/b2 uploads will now have hashes
    • Added rclone backend decode/encode commands to replicate functionality of cryptdecode (Anagh Kumar Baranwal)
    • Get rid of the unused Cipher interface as it obfuscated the code (Nick Craig-Wood)
  • Azure Blob
    • Implement streaming of unknown sized files so rcat is now supported (Nick Craig-Wood)
    • Implement memory pooling to control memory use (Nick Craig-Wood)
    • Add --azureblob-disable-checksum flag (Nick Craig-Wood)
    • Retry InvalidBlobOrBlock error as it may indicate block concurrency problems (Nick Craig-Wood)
    • Remove unused Object.parseTimeString() (Lars Lehtonen)
    • Fix permission error on SAS URL limited to container (Nick Craig-Wood)
  • B2
    • Add support for --header-upload and --header-download (Tim Gallant)
    • Ignore directory markers at the root also (Nick Craig-Wood)
    • Force the case of the SHA1 to lowercase (Nick Craig-Wood)
    • Remove unused largeUpload.clearUploadURL() (Lars Lehtonen)
  • Box
    • Add support for --header-upload and --header-download (Tim Gallant)
    • Implement About to read size used (Nick Craig-Wood)
    • Add token renew function for jwt auth (David Bramwell)
    • Added support for interchangeable root folder for Box backend (Sunil Patra)
    • Remove unnecessary iat from jws claims (David)
  • Drive
    • Follow shortcuts by default, skip with --drive-skip-shortcuts (Nick Craig-Wood)
    • Implement rclone backend shortcut command for creating shortcuts (Nick Craig-Wood)
    • Added rclone backend command to change service_account_file and chunk_size (Anagh Kumar Baranwal)
    • Fix missing files when using --fast-list and --drive-shared-with-me (Nick Craig-Wood)
    • Fix duplicate items when using --drive-shared-with-me (Nick Craig-Wood)
    • Extend --drive-stop-on-upload-limit to respond to teamDriveFileLimitExceeded. (harry)
    • Don't delete files with multiple parents to avoid data loss (Nick Craig-Wood)
    • Server side copy docs use default description if empty (Nick Craig-Wood)
  • Dropbox
    • Make error insufficient space to be fatal (harry)
    • Add info about required redirect url (Elan Ruusamäe)
  • Fichier
    • Add support for --header-upload and --header-download (Tim Gallant)
    • Implement custom pacer to deal with the new rate limiting (buengese)
  • FTP
    • Fix lockup when using concurrency limit on failed connections (Nick Craig-Wood)
    • Fix lockup on failed upload when using concurrency limit (Nick Craig-Wood)
    • Fix lockup on Close failures when using concurrency limit (Nick Craig-Wood)
    • Work around pureftp sending spurious 150 messages (Nick Craig-Wood)
  • Google Cloud Storage
    • Add support for --header-upload and --header-download (Nick Craig-Wood)
    • Add ARCHIVE storage class to help (Adam Stroud)
    • Ignore directory markers at the root (Nick Craig-Wood)
  • Googlephotos
    • Make the start year configurable (Daven)
    • Add support for --header-upload and --header-download (Tim Gallant)
    • Create feature/favorites directory (Brandon Philips)
    • Fix "concurrent map write" error (Nick Craig-Wood)
    • Don't put an image in error message (Nick Craig-Wood)
  • HTTP
    • Improved directory listing with new template from Caddy project (Rob Calistri)
  • Jottacloud
    • Implement --jottacloud-trashed-only (buengese)
    • Add support for --header-upload and --header-download (Tim Gallant)
    • Use RawURLEncoding when decoding base64 encoded login token (buengese)
    • Implement cleanup (buengese)
    • Update docs regarding cleanup, removed remains from old auth, and added warning about special mountpoints. (albertony)
  • Mailru
    • Describe 2FA requirements (valery1707)
  • Onedrive
    • Implement --onedrive-server-side-across-configs (Nick Craig-Wood)
    • Add support for --header-upload and --header-download (Tim Gallant)
    • Fix occasional 416 errors on multipart uploads (Nick Craig-Wood)
    • Added maximum chunk size limit warning in the docs (Harry)
    • Fix missing drive on config (Nick Craig-Wood)
    • Make error quotaLimitReached to be fatal (harry)
  • Opendrive
    • Add support for --header-upload and --header-download (Tim Gallant)
  • Pcloud
    • Added support for interchangeable root folder for pCloud backend (Sunil Patra)
    • Add support for --header-upload and --header-download (Tim Gallant)
    • Fix initial config "Auth state doesn't match" message (Nick Craig-Wood)
  • Premiumizeme
    • Add support for --header-upload and --header-download (Tim Gallant)
    • Prune unused functions (Lars Lehtonen)
  • Putio
    • Add support for --header-upload and --header-download (Nick Craig-Wood)
    • Make downloading files use the rclone http Client (Nick Craig-Wood)
    • Fix parsing of remotes with leading and trailing / (Nick Craig-Wood)
  • Qingstor
    • Make rclone cleanup remove pending multipart uploads older than 24h (Nick Craig-Wood)
    • Try harder to cancel failed multipart uploads (Nick Craig-Wood)
    • Prune multiUploader.list() (Lars Lehtonen)
    • Lint fix (Lars Lehtonen)
  • S3
    • Add support for --header-upload and --header-download (Tim Gallant)
    • Use memory pool for buffer allocations (Maciej Zimnoch)
    • Add SSE-C support for AWS, Ceph, and MinIO (Jack Anderson)
    • Fail fast multipart upload (MichaĹ‚ Matczuk)
    • Report errors on bucket creation (mkdir) correctly (Nick Craig-Wood)
    • Specify that Minio supports URL encoding in listings (Nick Craig-Wood)
    • Added 500 as retryErrorCode (MichaĹ‚ Matczuk)
    • Use --low-level-retries as the number of SDK retries (Aleksandar Janković)
    • Fix multipart abort context (Aleksandar Jankovic)
    • Replace deprecated session.New() with session.NewSession() (Lars Lehtonen)
    • Use the provided size parameter when allocating a new memory pool (Joachim Brandon LeBlanc)
    • Use rclone's low level retries instead of AWS SDK to fix listing retries (Nick Craig-Wood)
    • Ignore directory markers at the root also (Nick Craig-Wood)
    • Use single memory pool (MichaĹ‚ Matczuk)
    • Do not resize buf on put to memBuf (MichaĹ‚ Matczuk)
    • Improve docs for --s3-disable-checksum (Nick Craig-Wood)
    • Don't leak memory or tokens in edge cases for multipart upload (Nick Craig-Wood)
  • Seafile
    • Implement 2FA (Fred)
  • SFTP
    • Added --sftp-pem-key to support inline key files (Rob Calistri)
    • Fix post transfer copies failing with 0 size when using set_modtime=false (Nick Craig-Wood)
  • Sharefile
    • Add support for --header-upload and --header-download (Tim Gallant)
  • Sugarsync
    • Add support for --header-upload and --header-download (Tim Gallant)
  • Swift
    • Add support for --header-upload and --header-download (Nick Craig-Wood)
    • Fix cosmetic issue in error message (Martin Michlmayr)
  • Union
    • Implement multiple writable remotes (Max Sum)
    • Fix server-side copy (Max Sum)
    • Implement ListR (Max Sum)
    • Enable ListR when upstreams contain local (Max Sum)
  • WebDAV
    • Add support for --header-upload and --header-download (Tim Gallant)
    • Fix X-OC-Mtime header for Transip compatibility (Nick Craig-Wood)
    • Report full and consistent usage with about (Yves G)
  • Yandex
    • Add support for --header-upload and --header-download (Tim Gallant)
16 Likes
2

That is one massive release. :slight_smile:

2 Likes
3

this is great news for rclone.
i just made a donation, my second donation since i started to use rclone.

lets' make this happen!

3 Likes
4

It amazes me the sheer amount of code changes that go into these releases. Its impressive really. and this is a point release.

2 Likes
5

Thanks! Did you see the new donate page BTW? Not sure if the old one is being cached...

6

I can see it (after a shift+refresh in chrome)

7

i have one request, as per this post.
can you add the '--magic' flag :upside_down_face:

3 Likes
8

I'm sure that is in the just-make-it-work branch :wink:

9

i clicked the link you shared in this post.
https://rclone.org/donate/
"However, maintaining rclone is a lot of work - easily the equivalent of a full time job - for me"

is that correct?

10

That is the correct one - thanks for confirming :slight_smile:

11

Going to need a week to simply read about all the changes. A month to play with them all :joy:

Thank you ncw

3 Likes
12

Fantastic work, everyone. Donation coming your way, Nick.

2 Likes
13

Thank you for the effort everyone.
Like the new overhaul of the site. Made it easy for me to spot the donate button and donate! :slight_smile:

1 Like
14

Thanks for the excellent work!

It appears that the (I think official) docker image has not been updated yet at https://hub.docker.com/r/rclone/rclone

1 Like
15

Excellent work as always !

Just one quick question. For iowait issue, no flags are needed now in .52 ? Thank you

1 Like
16

Yes the docker build broke for some reason... Will try to fix

1 Like
17

I fixed a major bug and tweaked the defaults so no flags should be needed. You'll still get iowait but that is because rclone is making the kernel wait for stuff to be fetched over the network with async fetches. You can use --async-read=false to turn off async reading which will get rid of the IOWAIT at the cost of some performance.

18

Hi does union support hardlinks?

1 Like
19

On local filesystems it should see hardlinks just as a duplicate of the file so I wouldn't have thought there would be a problem with that.

20

w-z4eOb
w-z4eOb2560Ă—1056 124 KB