Rclone 1.55 release

Releases
1

Rclone 1.55 has been released. Find it here Rclone downloads

Some highlights of the release:

  • rclone selfupdate for updating rclone
  • macOS ARM64 is now fully supported
  • Connection strings
  • Config system re-work
  • Lots of work on getting cancellation to work everywhere (eg for --max-duration)
  • Lots of little but important fixes in the VFS layer for rclone mount

Thank you to all the contributors to this release, those who contributed code or doc fixes (33 people) or made issues or answered questions in the forum - your help is much appreciated!

I'd like to welcome our new maintainer @albertony who has being doing a fantastic job fixing up lots of things and improving the documents.

Thank you also to CERN and the CS3MESH4EOSC Project for sponsoring the "Connection strings" improvement - see below for details.

If you (or your company) is enjoying rclone, please consider donating to keep the project sustainable, or hire me to add a feature you really want!

v1.55.0 - 2021-03-31

See commits

  • New commands
    • selfupdate (Ivan Andreev)
      • Allows rclone to update itself in-place or via a package (using --package flag)
      • Reads cryptographically signed signatures for non beta releases
      • Works on all OSes.
    • test - these are test commands - use with care!
      • histogram - Makes a histogram of file name characters.
      • info - Discovers file name or other limitations for paths.
      • makefiles - Make a random file hierarchy for testing.
      • memory - Load all the objects at remote:path into memory and report memory stats.
  • New Features
    • Connection strings
      • Config parameters can now be passed as part of the remote name as a connection string.
      • For example to do the equivalent of --drive-shared-with-me use drive,shared_with_me:
      • Make sure we don't save on the fly remote config to the config file (Nick Craig-Wood)
      • Make sure backends with additional config have a different name for caching (Nick Craig-Wood)
      • This work was sponsored by CERN, through the CS3MESH4EOSC Project.
        • CS3MESH4EOSC has received funding from the European Union’s Horizon 2020
        • research and innovation programme under Grant Agreement no. 863353.
    • build
      • Update go build version to go1.16 and raise minimum go version to go1.13 (Nick Craig-Wood)
      • Make a macOS ARM64 build to support Apple Silicon (Nick Craig-Wood)
      • Install macfuse 4.x instead of osxfuse 3.x (Nick Craig-Wood)
      • Use GO386=softfloat instead of deprecated GO386=387 for 386 builds (Nick Craig-Wood)
      • Disable IOS builds for the time being (Nick Craig-Wood)
      • Androids builds made with up to date NDK (x0b)
      • Add an rclone user to the Docker image but don't use it by default (cynthia kwok)
    • dedupe: Make largest directory primary to minimize data moved (Saksham Khanna)
    • config
      • Wrap config library in an interface (Fionera)
      • Make config file system pluggable (Nick Craig-Wood)
      • --config "" or "/notfound" for in memory config only (Nick Craig-Wood)
      • Clear fs cache of stale entries when altering config (Nick Craig-Wood)
    • copyurl: Add option to print resulting auto-filename (albertony)
    • delete: Make --rmdirs obey the filters (Nick Craig-Wood)
    • docs - many fixes and reworks from edwardxml, albertony, pvalls, Ivan Andreev, Evan Harris, buengese, Alexey Tabakman
    • encoder/filename - add SCSU as tables (Klaus Post)
    • Add multiple paths support to --compare-dest and --copy-dest flag (K265)
    • filter: Make --exclude "dir/" equivalent to --exclude "dir/**" (Nick Craig-Wood)
    • fshttp: Add DSCP support with --dscp for QoS with differentiated services (Max Sum)
    • lib/cache: Add Delete and DeletePrefix methods (Nick Craig-Wood)
    • lib/file
      • Make pre-allocate detect disk full errors and return them (Nick Craig-Wood)
      • Don't run preallocate concurrently (Nick Craig-Wood)
      • Retry preallocate on EINTR (Nick Craig-Wood)
    • operations: Made copy and sync operations obey a RetryAfterError (Ankur Gupta)
    • rc
      • Add string alternatives for setting options over the rc (Nick Craig-Wood)
      • Add options/local to see the options configured in the context (Nick Craig-Wood)
      • Add _config parameter to set global config for just this rc call (Nick Craig-Wood)
      • Implement passing filter config with _filter parameter (Nick Craig-Wood)
      • Add fscache/clear and fscache/entries to control the fs cache (Nick Craig-Wood)
      • Avoid +Inf value for speed in core/stats (albertony)
      • Add a full set of stats to core/stats (Nick Craig-Wood)
      • Allow fs= params to be a JSON blob (Nick Craig-Wood)
    • rcd: Added systemd notification during the rclone rcd command. (Naveen Honest Raj)
    • rmdirs: Make --rmdirs obey the filters (Nick Craig-Wood)
    • version: Show build tags and type of executable (Ivan Andreev)
  • Bug Fixes
    • install.sh: make it fail on download errors (Ivan Andreev)
    • Fix excessive retries missing --max-duration timeout (Nick Craig-Wood)
    • Fix crash when --low-level-retries=0 (Nick Craig-Wood)
    • Fix failed token refresh on mounts created via the rc (Nick Craig-Wood)
    • fshttp: Fix bandwidth limiting after bad merge (Nick Craig-Wood)
    • lib/atexit
      • Unregister interrupt handler once it has fired so users can interrupt again (Nick Craig-Wood)
      • Fix occasional failure to unmount with CTRL-C (Nick Craig-Wood)
      • Fix deadlock calling Finalise while Run is running (Nick Craig-Wood)
    • lib/rest: Fix multipart uploads not stopping on context cancel (Nick Craig-Wood)
  • Mount
    • Allow mounting to root directory on windows (albertony)
    • Improved handling of relative paths on windows (albertony)
    • Fix unicode issues with accented characters on macOS (Nick Craig-Wood)
    • Docs: document the new FileSecurity option in WinFsp 2021 (albertony)
    • Docs: add note about volume path syntax on windows (albertony)
    • Fix caching of old directories after renaming them (Nick Craig-Wood)
    • Update cgofuse to the latest version to bring in macfuse 4 fix (Nick Craig-Wood)
  • VFS
    • --vfs-used-is-size to report used space using recursive scan (tYYGH)
    • Don't set modification time if it was already correct (Nick Craig-Wood)
    • Fix Create causing windows explorer to truncate files on CTRL-C CTRL-V (Nick Craig-Wood)
    • Fix modtimes not updating when writing via cache (Nick Craig-Wood)
    • Fix modtimes changing by fractional seconds after upload (Nick Craig-Wood)
    • Fix modtime set if --vfs-cache-mode writes/full and no write (Nick Craig-Wood)
    • Rename files in cache and cancel uploads on directory rename (Nick Craig-Wood)
    • Fix directory renaming by renaming dirs cached in memory (Nick Craig-Wood)
  • Local
    • Add flag --local-no-preallocate (David Sze)
    • Make nounc an advanced option except on Windows (albertony)
    • Don't ignore preallocate disk full errors (Nick Craig-Wood)
  • Cache
    • Add --fs-cache-expire-duration to control the fs cache (Nick Craig-Wood)
  • Crypt
    • Add option to not encrypt data (Vesnyx)
    • Log hash ok on upload (albertony)
  • Azure Blob
    • Add container public access level support. (Manish Kumar)
  • B2
    • Fix HTML files downloaded via cloudflare (Nick Craig-Wood)
  • Box
    • Fix transfers getting stuck on token expiry after API change (Nick Craig-Wood)
  • Chunker
    • Partially implement no-rename transactions (Maxwell Calman)
  • Drive
    • Don't stop server side copy if couldn't read description (Nick Craig-Wood)
    • Pass context on to drive SDK - to help with cancellation (Nick Craig-Wood)
  • Dropbox
    • Add polling for changes support (Robert Thomas)
    • Make --timeout 0 work properly (Nick Craig-Wood)
    • Raise priority of rate limited message to INFO to make it more noticeable (Nick Craig-Wood)
  • Fichier
    • Implement copy & move (buengese)
    • Implement public link (buengese)
  • FTP
    • Implement Shutdown method (Nick Craig-Wood)
    • Close idle connections after --ftp-idle-timeout (1m by default) (Nick Craig-Wood)
    • Make --timeout 0 work properly (Nick Craig-Wood)
    • Add --ftp-close-timeout flag for use with awkward ftp servers (Nick Craig-Wood)
    • Retry connections and logins on 421 errors (Nick Craig-Wood)
  • Hdfs
    • Fix permissions for when directory is created (Lucas Messenger)
  • Onedrive
    • Make --timeout 0 work properly (Nick Craig-Wood)
  • S3
    • Fix --s3-profile which wasn't working (Nick Craig-Wood)
  • SFTP
    • Close idle connections after --sftp-idle-timeout (1m by default) (Nick Craig-Wood)
    • Fix "file not found" errors for read once servers (Nick Craig-Wood)
    • Fix SetModTime stat failed: object not found with --sftp-set-modtime=false (Nick Craig-Wood)
  • Swift
    • Update github.com/ncw/swift to v2.0.0 (Nick Craig-Wood)
    • Implement copying large objects (nguyenhuuluan434)
  • Union
    • Fix crash when using epff policy (Nick Craig-Wood)
    • Fix union attempting to update files on a read only file system (Nick Craig-Wood)
    • Refactor to use fspath.SplitFs instead of fs.ParseRemote (Nick Craig-Wood)
    • Fix initialisation broken in refactor (Nick Craig-Wood)
  • WebDAV
    • Add support for sharepoint with NTLM authentication (Rauno Ots)
    • Make sharepoint-ntlm docs more consistent (Alex Chen)
    • Improve terminology in sharepoint-ntlm docs (Ivan Andreev)
    • Disable HTTP/2 for NTLM authentication (georne)
    • Fix sharepoint-ntlm error 401 for parallel actions (Ivan Andreev)
    • Check that purged directory really exists (Ivan Andreev)
  • Yandex
    • Make --timeout 0 work properly (Nick Craig-Wood)
  • Zoho
    • Replace client id - you will need to rclone config reconnect after this (buengese)
    • Add forgotten setupRegion() to NewFs - this finally fixes regions other than EU (buengese)
10 Likes
4

Looks like there might be a problem with the OSX ARM64 version install script. I'm running MacOS 11.3 on my M1 MacBook Pro, and the script returns 'arm64' for the OS type, but the script expects 'arm'

% uname  -m
arm64

Want to fix it so the case is arm64, rather than 'arm'? I'd submit a PR but it's probably quicker for you to tweak it!

5

I've patched it - can you see if it works for you @webreaper ? Thanks

2 Likes
6

Works a treat. I raised an issue (here: OS type incorrect on ARM64/M1 Macs, means install.sh fails. · Issue #5183 · rclone/rclone · GitHub) but you're too fast! :smiley:

7

first of all thanks for this update! :heart:

Just a little question on this. Can anybody describe how to use an "in memory config"?

8
1 Like
9

The --config "" or "/notfound" are relevant when you have an encrypted configuration file, which typically means you have to enter a password on startup when rclone loads the configuration (though there are some other possibilities such as --password-command), but you are executing commands that don't really need the stored configuration, such as genautocomplete or on-the-fly remotes, and therefore want to skip the password prompt (see #5125).

1 Like
10

looks like it is time to encrypt the config. :slight_smile:
I will read the docs. Thank you guys.

11

Also check out the wiki, there are some related examples there you may find useful (e.g. this and this).

12

What is the use case for connection strings?

13

Autoupdate throws this error…
Error: updating would discard the mount FUSE capability, aborting

I’m not using the fuse mount (as far as I know!), I guess it’s user error, any suggestions?

14

Either it's my mistake in the selfupdate code or probably you use a custom build of rclone that was built with libfuse (you might not be using it but according to detection it's in). What is output of rclone version and of rclone version --check? You can manually follow https://rclone.org/install once to work around this safety measure.

15

They are so you can have all the config for a backend in the string you give to rclone. Very useful for people using rclone via the api to create backends on the fly.

It is also very useful for a little tweak to a backend that you want to apply to one backend only - this is my fave example from the docs

The major advantage to using the connection string style syntax is
that it only applies the the remote, not to all the remotes of that
type of the command line. A common confusion is this attempt to copy a
file shared on google drive to the normal drive which does not
work because the --drive-shared-with-me flag applies to both the
source and the destination.

rclone copy --drive-shared-with-me gdrive:shared-file.txt gdrive:

However using the connection string syntax, this does work.

rclone copy "gdrive,shared_with_me:shared-file.txt" gdrive:
16

I am getting the same error when using selfupdate.
Have previously updated using install.sh script.

rclone selfupdate
2021/04/01 20:07:25 Error: updating would discard the mount FUSE capability, aborting

rclone v1.55.0

  • os/type: linux
  • os/arch: amd64
  • go/version: go1.16.2
  • go/linking: static
  • go/tags: cmount

yours: 1.55.0
latest: 1.55.0 (released 2021-03-31)
beta: 1.56.0-beta.5356.9e2fbe0f1 (released 2021-03-31)
upgrade: (wouldn't let me include links in my post)

17

@jpd
Thanks for info, I understand now.
Will be fixed in 1.55.1. For the time being please fall back to install.sh.

@ncw
The detector treats combination static+cmount as libfuse-aware, although in fact it isn't.

1 Like
18

How did we ever get static and cmount - I don't understand?

19

The build is beyond my department :wink:
The fact is that rclone/cmount.go at v1.55.0 · rclone/rclone · GitHub was compiled in
But rclone/cgo.go at v1.55.0 · rclone/rclone · GitHub wasn't
I think the problem doesn't stem from xgo. I use it on my fork and it produces the cgo tag.
I don't yet understand either.
Maybe it's a good idea to add a unit test failing for the static+cmount combination after a solution is found...

20

We'll get to the bottom of it I'm sure!

Is there an issue about this? I didn't see one.

21