Rclone v1.63.0 release

Releases
1

Rclone 1.63.0 has been released. Find it in the rclone downloads or use rclone selfupdate to upgrade.

Some highlights of the release:

  • New backends: Pikpak, petabox.io, GCS via s3, Fastmail.
  • Partial files: transfers to a temporary file for local,ftp,sftp file which is renamed when complete. Disable with the --inplace flag.
  • The s3, azureblob and gcs backends now support directory markers so empty directories are supported.
  • The --default-time flag now controls the unknown modification time of files/dirs.

Thank you to all the contributors to this release, those who contributed code or doc fixes (58 people) or made issues or answered questions in the forum - your help is much appreciated!

I am pleased to say we have started offering support contracts for businesses through our new company Rclone Services Ltd. Over the years I've had a lot of requests for support and sponsorship from businesses, universities, research labs and other organisations all over the world so it seemed natural to form a limited company to service these requirements.

I have employed my son (@edcw) who has been doing sales and account management for the past 5 years to help me with customer interactions so I can spend more time developing and delivering updates.

We're not going to change the open source nature of the project, it will still be available for free under the generous MIT licence and I'm hoping the community will continue to grow around rclone. In the future I hope to be able to recruit from the rclone experts that congregate here.

If youā€™re working in the industry and you feel your company could benefit from a support contract please get in touch at sales@rclone.com or if youā€™re interested in advertising with us then please drop us a message to sponsorship@rclone.com.

Please consider donating or sponsoring to keep the project sustainable.

v1.63.0 - 2023-06-30

See commits

  • New backends
  • Major changes
    • Files will be copied to a temporary name ending in .partial when copying to local,ftp,sftp then renamed at the end of the transfer. (Janne Hellsten, Nick Craig-Wood)
      • This helps with data integrity as we don't delete the existing file until the new one is complete.
      • It can be disabled with the --inplace flag.
      • This behaviour will also happen if the backend is wrapped, for example sftp wrapped with crypt.
    • The s3, azureblob and gcs backends now support directory markers so empty directories are supported (Jānis BebrÄ«tis, Nick Craig-Wood)
    • The --default-time flag now controls the unknown modification time of files/dirs (Nick Craig-Wood)
      • If a file or directory does not have a modification time rclone can read then rclone will display this fixed time instead.
      • For the old behaviour use --default-time 0s which will set this time to the time rclone started up.
  • New Features
    • build
      • Modernise linters in use and fixup all affected code (albertony)
      • Push docker beta to GHCR (GitHub container registry) (Richard Tweed)
    • cat: Add --separator option to cat command (Loren Gordon)
    • config
      • Do not remove/overwrite other files during config file save (albertony)
      • Do not overwrite config file symbolic link (albertony)
      • Stop config create making invalid config files (Nick Craig-Wood)
    • doc updates (Adam K, Aditya Basu, albertony, asdffdsazqqq, Damo, danielkrajnik, Dimitri Papadopoulos, dlitster, Drew Parsons, jumbi77, kapitainsky, mac-15, Mariusz Suchodolski, Nick Craig-Wood, NickIAm, Rintze Zelle, Stanislav Gromov, Tareq Sharafy, URenko, yuudi, Zach Kipp)
    • fs
      • Add size to JSON logs when moving or copying an object (Nick Craig-Wood)
      • Allow boolean features to be enabled with --disable !Feature (Nick Craig-Wood)
    • genautocomplete: Rename to completion with alias to the old name (Nick Craig-Wood)
    • librclone: Added example on using librclone with Go (alankrit)
    • lsjson: Make --stat more efficient (Nick Craig-Wood)
    • operations
      • Implement --multi-thread-write-buffer-size for speed improvements on downloads (Paulo Schreiner)
      • Reopen downloads on error when using check --download and cat (Nick Craig-Wood)
    • rc: config/listremotes includes remotes defined with environment variables (kapitainsky)
    • selfupdate: Obey --no-check-certificate flag (Nick Craig-Wood)
    • serve restic: Trigger systemd notify (Shyim)
    • serve webdav: Implement owncloud checksum and modtime extensions (WeidiDeng)
    • sync: --suffix-keep-extension preserve 2 part extensions like .tar.gz (Nick Craig-Wood)
  • Bug Fixes
    • accounting
      • Fix Prometheus metrics to be the same as core/stats (Nick Craig-Wood)
      • Bwlimit signal handler should always start (Sam Lai)
    • bisync: Fix maxDelete parameter being ignored via the rc (Nick Craig-Wood)
    • cmd/ncdu: Fix screen corruption when logging (eNV25)
    • filter: Fix deadlock with errors on --files-from (douchen)
    • fs
      • Fix interaction between --progress and --interactive (Nick Craig-Wood)
      • Fix infinite recursive call in pacer ModifyCalculator (fixes issue reported by the staticcheck linter) (albertony)
    • lib/atexit: Ensure OnError only calls cancel function once (Nick Craig-Wood)
    • lib/rest: Fix problems re-using HTTP connections (Nick Craig-Wood)
    • rc
      • Fix operations/stat with trailing / (Nick Craig-Wood)
      • Fix missing --rc flags (Nick Craig-Wood)
      • Fix output of Time values in options/get (Nick Craig-Wood)
    • serve dlna: Fix potential data race (Nick Craig-Wood)
    • version: Fix reported os/kernel version for windows (albertony)
  • Mount
    • Add --mount-case-insensitive to force the mount to be case insensitive (Nick Craig-Wood)
    • Removed unnecessary byte slice allocation for reads (Anagh Kumar Baranwal)
    • Clarify rclone mount error when installed via homebrew (Nick Craig-Wood)
    • Added _netdev to the example mount so it gets treated as a remote-fs rather than local-fs (Anagh Kumar Baranwal)
  • Mount2
    • Updated go-fuse version (Anagh Kumar Baranwal)
    • Fixed statfs (Anagh Kumar Baranwal)
    • Disable xattrs (Anagh Kumar Baranwal)
  • VFS
    • Add MkdirAll function to make a directory and all beneath (Nick Craig-Wood)
    • Fix reload: failed to add virtual dir entry: file does not exist (Nick Craig-Wood)
    • Fix writing to a read only directory creating spurious directory entries (WeidiDeng)
    • Fix potential data race (Nick Craig-Wood)
    • Fix backends being Shutdown too early when startup takes a long time (Nick Craig-Wood)
  • Local
    • Fix filtering of symlinks with -l/--links flag (Nick Craig-Wood)
    • Fix /path/to/file.rclonelink when -l/--links is in use (Nick Craig-Wood)
    • Fix crash with --metadata on Android (Nick Craig-Wood)
  • Cache
    • Fix backends shutting down when in use when used via the rc (Nick Craig-Wood)
  • Crypt
    • Add --crypt-suffix option to set a custom suffix for encrypted files (jladbrook)
    • Add --crypt-pass-bad-blocks to allow corrupted file output (Nick Craig-Wood)
    • Fix reading 0 length files (Nick Craig-Wood)
    • Try not to return "unexpected EOF" error (Nick Craig-Wood)
    • Reduce allocations (albertony)
    • Recommend Dropbox for base32768 encoding (Nick Craig-Wood)
  • Azure Blob
    • Empty directory markers (Nick Craig-Wood)
    • Support azure workload identities (Tareq Sharafy)
    • Fix azure blob uploads with multiple bits of metadata (Nick Craig-Wood)
    • Fix azurite compatibility by sending nil tier if set to empty string (Roel Arents)
  • Combine
    • Implement missing methods (Nick Craig-Wood)
    • Fix goroutine stack overflow on bad object (Nick Craig-Wood)
  • Drive
    • Add --drive-env-auth to get IAM credentials from runtime (Peter Brunner)
    • Update drive service account guide (Juang, Yi-Lin)
    • Fix change notify picking up files outside the root (Nick Craig-Wood)
    • Fix trailing slash mis-identificaton of folder as file (Nick Craig-Wood)
    • Fix incorrect remote after Update on object (Nick Craig-Wood)
  • Dropbox
    • Implement --dropbox-pacer-min-sleep flag (Nick Craig-Wood)
    • Fix the dropbox batcher stalling (Misty)
  • Fichier
    • Add --ficicher-cdn option to use the CDN for download (Nick Craig-Wood)
  • FTP
    • Lower log message priority when SetModTime is not supported to debug (Tobias Gion)
    • Fix "unsupported LIST line" errors on startup (Nick Craig-Wood)
    • Fix "501 Not a valid pathname." errors when creating directories (Nick Craig-Wood)
  • Google Cloud Storage
    • Empty directory markers (Jānis BebrÄ«tis, Nick Craig-Wood)
    • Added --gcs-user-project needed for requester pays (Christopher Merry)
  • HTTP
    • Add client certificate user auth middleware. This can auth serve restic from the username in the client cert. (Peter Fern)
  • Jottacloud
    • Fix vfs writeback stuck in a failed upload loop with file versioning disabled (albertony)
  • Onedrive
    • Add --onedrive-av-override flag to download files flagged as virus (Nick Craig-Wood)
    • Fix quickxorhash on 32 bit architectures (Nick Craig-Wood)
    • Report any list errors during rclone cleanup (albertony)
  • Putio
    • Fix uploading to the wrong object on Update with overriden remote name (Nick Craig-Wood)
    • Fix modification times not being preserved for server side copy and move (Nick Craig-Wood)
    • Fix server side copy failures (400 errors) (Nick Craig-Wood)
  • S3
    • Empty directory markers (Jānis BebrÄ«tis, Nick Craig-Wood)
    • Update Scaleway storage classes (Brian Starkey)
    • Fix --s3-versions on individual objects (Nick Craig-Wood)
    • Fix hang on aborting multpart upload with iDrive e2 (Nick Craig-Wood)
    • Fix missing "tier" metadata (Nick Craig-Wood)
    • Fix V3sign: add missing subresource delete (cc)
    • Fix Arvancloud Domain and region changes and alphabetise the provider (Ehsan Tadayon)
    • Fix Qiniu KODO quirks virtualHostStyle is false (zzq)
  • SFTP
    • Add --sftp-host-key-algorithms to allow specifying SSH host key algorithms (Joel)
    • Fix using --sftp-key-use-agent and --sftp-key-file together needing private key file (Arnav Singh)
    • Fix move to allow overwriting existing files (Nick Craig-Wood)
    • Don't stat directories before listing them (Nick Craig-Wood)
    • Don't check remote points to a file if it ends with / (Nick Craig-Wood)
  • Sharefile
    • Disable streamed transfers as they no longer work (Nick Craig-Wood)
  • Smb
    • Code cleanup to avoid overwriting ctx before first use (fixes issue reported by the staticcheck linter) (albertony)
  • Storj
    • Fix "uplink: too many requests" errors when uploading to the same file (Nick Craig-Wood)
    • Fix uploading to the wrong object on Update with overriden remote name (Nick Craig-Wood)
  • Swift
    • Ignore 404 error when deleting an object (Nick Craig-Wood)
  • Union
    • Implement missing methods (Nick Craig-Wood)
    • Allow errors to be unwrapped for inspection (Nick Craig-Wood)
  • Uptobox
    • Add --uptobox-private flag to make all uploaded files private (Nick Craig-Wood)
    • Fix improper regex (Aaron Gokaslan)
    • Fix Update returning the wrong object (Nick Craig-Wood)
    • Fix rmdir declaring that directories weren't empty (Nick Craig-Wood)
  • WebDAV
    • nextcloud: Add support for chunked uploads (Paul)
    • Set modtime using propset for owncloud and nextcloud (WeidiDeng)
    • Make pacer minSleep configurable with --webdav-pacer-min-sleep (ed)
    • Fix server side copy/move not overwriting (WeidiDeng)
    • Fix modtime on server side copy for owncloud and nextcloud (Nick Craig-Wood)
  • Yandex
    • Fix 400 Bad Request on transfer failure (Nick Craig-Wood)
  • Zoho
    • Fix downloads with Range: header returning the wrong data (Nick Craig-Wood)
12 Likes
3

I am way more excited for this release than anyone really should be for some software!

For my own tools, it means I get to rip out a lot of machinery around atomic transfers and links. (The former is done. The latter is something I'll work on next week or so).

For my own uses, I really worried about Make rclone more robust to false-negatives with local copies and --size-only Ā· Issue #6206 Ā· rclone/rclone Ā· GitHub and am glad I can put those concerns to rest.

But speaking of size-based transfers, I am so happy that rclone's serve webdav supports serving and setting ModTime. I currently use SFTP for rclone to rclone communication but it is SLOW. WebDAV is so much faster and I get to use that now! (even over an SSH tunnel, it is way faster)

Thanks for everyone's hard work!

2 Likes
4

wow, what an update.

about the weblinks, the url do not work, as they point to the forum, not the main website
for example,
https://forum.rclone.org/s3/#s3-directory-markers
should be
https://rclone.org/s3/#s3-directory-markers

5

Is it normal that docs are not updated yet?

e.g.:

https://forum.rclone.org/docs/#inplace

6

:slight_smile: Rather overdue, but events conspired against me. That is my excuse!

Foo. Have updated the post now - thanks.

I've fixed the links so they should work now ( eg Documentation )

1 Like
7

Yes it works now.

So we have new flag --inplace which applies to only few remotes. As an rclone user who for example tries sftp it is almost impossible to find it. As what would I do? I would read rclone docs about sftp. Nothing mentioned there.

There are flags like --track-renames useful only for sync operation but nothing is mentioned on rclone sync

Would it make sense to add to every command/remote help page section with flags which apply? And slowly improve overall docs usefulness.

8

Great!

Noted.

Yes you are right there.

The docs are, as forever, a work in progress :slight_smile:

Which commands work with which backends tends to be based on the feature flags. --inplace requires the PartialUploads feature flags on a backend and --track-renames requires Move or Copy.

If you want to see what features a backend has use rclone backend features remote:.

So some of this work could potentially be automated.

It might be worth a mention of --inplace on the local, ftp, sftp backends. I'd probably say something like "This backend can have partial uploads so normally rclone will upload to a .partial file. This can be disabled with --inplace.

As for --track-renames that should really be an option on the sync command itself, but isn't for historical reasons. Making it an option on the sync command would make it appear in the help properly and would mean it only worked on rclone sync.

Putting a link to the docs for --track-renames on the sync page would probably be a good idea. There are a whole pile of flags which are relevant to move/sync/copy/moveto/copyto which could have links too.

In the docs I try not to duplicate stuff as it makes maintaining the docs harder and I try to automate stuff which can be automated, so maybe this can be improved with a bit of automation. Maybe each command could have a list of flags that it would like the user to refer to and the doc creator could take that list, look up the help for each one and add them to the docs page.

2 Likes
9

Very happy about the support contracts for businesses, I hope it does very well : ) And "welcome" to your son in the Rclone world ; )

4 Likes
10

Can I get some clarifications on the difference between tpslimit and the dropbox-pacer-min-sleep?

11

They do pretty much the same thing but the pacer is for the Dropbox backend only whereas tpslimit affects everything.

12

This is good news. Only commercialization can make the road of open source go further. I have seen many good open source software start commercialization. I am very supportive and hope to go better in the future.

2 Likes
13

i see the cache remote got an update. wow. i thought it was deprecated.

on that note. question. @ncw could you shed a little light on the roadmap for using cache on windows? i see you're putting a lot of development into the new VFS-cache, and triggering a cache sync recommends using the new VFS cache, which requires installing additional software. is that the planned roadmap for cached syncs? just trying to plan ahead a bit here.

14

It gets a bit of maintenance every now and again!

The plan is to make the vfs cache into the one true caching system which you can use like the cache backend. I have a prototype of this working. You won't need to install WinFSP for this to work - everything will be built into rclone.

1 Like
15

awesome :man_dancing:

i'll wait :slight_smile:

16

Nice.
I hope we will one day get an option to use local drive instead of memory to store dir-cache-time.

17

That is on the roadmap :slight_smile:

1 Like
18

When you use the new copy module on windows, if i do something like :
C:\rclone\rclone copy "D:\source" "H:\dest" does it use the local C drive as cache in any way ?

I want to be sure to not wear out my nvme with data transfer.

19

No. The copy goes straight to the destination drive. It is just renamed at the end.

20

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.