Rclone v1.61.0 release

[ncw (Nick Craig-Wood)]

Rclone 1.61.0 has been released. Find it in the rclone downloads or use rclone selfupdate to upgrade.

Some highlights of the release:

• New S3 provider: Liara LOS
• Rework of azureblob backend to use the new SDK which supports all the Azure authentication methods.
• Very large reduction in memory use for s3 backend.

This release also includes a major refactor of the http servers (rclone serve http, rclone serve webdav, rclone serve restic, rclone rcd) and as a result of this they support multiple addresses and unix sockets.

Thank you to all the contributors to this release, those who contributed code or doc fixes (35 people) or made issues or answered questions in the forum - your help is much appreciated! Many thanks also to Stonebranch for sponsoring the Azureblob re-write.

If you (or your company) is enjoying rclone, please consider donating or sponsoring to keep the project sustainable, or hire me to add a feature you really want!

v1.61.0 - 2022-12-20

See commits

• New backends
  • New S3 providers
    • Liara LOS (MohammadReza)
• New Features
  • build: Add vulnerability testing using govulncheck (albertony)
  • cmd: Enable SIGINFO (Ctrl-T) handler on FreeBSD, NetBSD, OpenBSD and Dragonfly BSD (x3-apptech)
  • config: Add config/setpath for setting config path via rc/librclone (Nick Craig-Wood)
  • dedupe
    • Count Checks in the stats while scanning for duplicates (Nick Craig-Wood)
    • Make dedupe obey the filters (Nick Craig-Wood)
  • dlna: Properly attribute code used from anacrolix/dms (Nick Craig-Wood)
  • docs
    • Add minimum versions and status badges to backend and command docs (Nick Craig-Wood, albertony)
    • Remote names may not start or end with space (albertony)
  • filter: Add metadata filters --metadata-include/exclude/filter and friends (Nick Craig-Wood)
  • fs
    • Make all duration flags take y, M, w, d etc suffixes (Nick Craig-Wood)
    • Add global flag --color to control terminal colors (Kevin Verstaen)
  • fspath: Allow unicode numbers and letters in remote names (albertony)
  • lib/file: Improve error message for creating dir on non-existent network host on windows (albertony)
  • lib/http: Finish port of rclone servers to lib/http (Tom Mombourquette, Nick Craig-Wood)
  • lib/oauthutil: Improved usability of config flows needing web browser (Ole Frost)
  • ncdu
    • Add support for modification time (albertony)
    • Fallback to sort by name also for sort by average size (albertony)
    • Rework to use tcell directly instead of the termbox wrapper (eNV25)
  • rc: Add commands to set GC Percent & Memory Limit (go 1.19+) (Anagh Kumar Baranwal)
  • rcat: Preserve metadata when Copy falls back to Rcat (Nick Craig-Wood)
  • rcd: Refactor rclone rc server to use lib/http (Nick Craig-Wood)
  • rcserver: Avoid generating default credentials with htpasswd (Kamui)
  • restic: Refactor to use lib/http (Nolan Woods)
  • serve http: Support unix sockets and multiple listeners (Tom Mombourquette)
  • serve webdav: Refactor to use lib/http (Nick Craig-Wood)
  • test: Replace defer cleanup with t.Cleanup (Eng Zer Jun)
  • test memory: Read metadata if -M flag is specified (Nick Craig-Wood)
  • wasm: Comply with wasm_exec.js licence terms (Matthew Vernon)
• Bug Fixes
  • build: Update golang.org/x/net/http2 to fix GO-2022-1144 (Nick Craig-Wood)
  • restic: Fix typo in docs 'remove' should be 'remote' (asdffdsazqqq)
  • serve dlna: Fix panic: Logger uninitialized. (Nick Craig-Wood)
• Mount
  • Update cgofuse for FUSE-T support for mounting volumes on Mac (Nick Craig-Wood)
• VFS
  • Windows: fix slow opening of exe files by not truncating files when not necessary (Nick Craig-Wood)
  • Fix IO Error opening a file with O_CREATE|O_RDONLY in --vfs-cache-mode not full (Nick Craig-Wood)
• Crypt
  • Fix compress wrapping crypt giving upload errors (Nick Craig-Wood)
• Azure Blob
  • Port to new SDK (Nick Craig-Wood)
    • Revamp authentication to include all methods and docs (Nick Craig-Wood)
    • Port old authentication methods to new SDK (Nick Craig-Wood, Brad Ackerman)
    • Thanks to Stonebranch for sponsoring this work.
  • Add --azureblob-no-check-container to assume container exists (Nick Craig-Wood)
  • Add --use-server-modtime support (Abdullah Saglam)
  • Add support for custom upload headers (rkettelerij)
  • Allow emulator account/key override (Roel Arents)
  • Support simple "environment credentials" (Nathaniel Wesley Filardo)
  • Ignore AuthorizationFailure when trying to create a create a container (Nick Craig-Wood)
• Box
  • Added note on Box API rate limits (Ole Frost)
• Drive
  • Handle shared drives with leading/trailing space in name (related to) (albertony)
• FTP
  • Update help text of implicit/explicit TLS options to refer to FTPS instead of FTP (ycdtosa)
  • Improve performance to speed up --files-from and NewObject (Anthony Pessy)
• HTTP
  • Parse GET responses when no_head is set (Arnie97)
  • Do not update object size based on Range requests (Arnie97)
  • Support Content-Range response header (Arnie97)
• Onedrive
  • Document workaround for shared with me files (vanplus)
• S3
  • Add Liara LOS to provider list (MohammadReza)
  • Add DigitalOcean Spaces regions sfo3, fra1, syd1 (Jack)
  • Avoid privileged GetBucketLocation to resolve s3 region (Anthony Pessy)
  • Stop setting object and bucket ACL to private if it is an empty string (Philip Harvey)
  • If bucket or object ACL is empty string then don't add X-Amz-Acl: header (Nick Craig-Wood)
  • Reduce memory consumption for s3 objects (Erik Agterdenbos)
  • Fix listing loop when using v2 listing on v1 server (Nick Craig-Wood)
  • Fix nil pointer exception when using Versions (Nick Craig-Wood)
  • Fix excess memory usage when using versions (Nick Craig-Wood)
  • Ignore versionIDs from uploads unless using --s3-versions or --s3-versions-at (Nick Craig-Wood)
• SFTP
  • Add configuration options to set ssh Ciphers / MACs / KeyExchange (dgouju)
  • Auto-detect shell type for fish (albertony)
  • Fix NewObject with leading / (Nick Craig-Wood)
• Smb
  • Fix issue where spurious dot directory is created (albertony)
• Storj
  • Implement server side Copy (Kaloyan Raev)

[El_Rico]

@ncw

lib/http: Finish port of rclone servers to lib/http

This change had an unexpected impact for me. I am already running another daemon on port 8080, so when restic tried to start the rclone REST API, it failed. Prior to this change, rclone and restic communicated via a stdin/stdout pipe:

@ncw

Rclone runs rclone serve restic over a stdin/stdout pipe.

Not a big issue, I just set the RCLONE_ADDR environment variable to unix:///tmp/rclone.sock. However, it seems that rclone is not cleaning up the socket, because on the next restic run, I am seeing this error message:

rclone: Failed to restic: failed to init server: listen unix /tmp/rclone.sock: bind: address already in use

I have worked around this for now, by cleaning up the socket file in the bash script, that I use to manage my restic backup. I do know what the conventions for sockets are, but from my perspective it would be cleaner if rclone managed this automatically.

If this does not break any conventions, should I create a feature request over on Github?

[ncw (Nick Craig-Wood)]

@El_Rico

If this does not break any conventions, should I create a feature request over on Github?

Yes, please report this on GitHub. I'm not 100% sure I understand your explanation so if you could give an example with command line flags there - that would be great - thanks.

[thesethguy]

After upgrading to the latest version from v1.60.1, I seem to be experiencing a similar issue and my old mount will no longer work.

As soon as I try to remount w/ rclone, I get this error:

Failed to start remote control: failed to init server: listen tcp :5572: bind: address already in use. Here's my systemd. What's weird is the mount functions normally, I just get nagged about my rc already being in use.

[Unit]
Description=RClone Service
Wants=network-online.target
After=network-online.target

[Service]
Type=notify
Environment=RCLONE_CONFIG=/home/username/.config/rclone/rclone.conf
KillMode=none
RestartSec=5
ExecStart=/usr/bin/rclone mount teamdrive:td1 /home/username/.mount \
--allow-other \
--dir-cache-time 48h \
--attr-timeout 48h \
--log-file /home/username/logs/rclone.log \
--log-level INFO \
--poll-interval 10s \
#--umask 002 \
--user-agent rclone_mount \
--rc \
--rc-addr :5572 \
--rc-no-auth \
--cache-dir=/home/username/.cache/cache \
--drive-pacer-min-sleep 10ms \
--drive-pacer-burst 200 \
--vfs-cache-mode full \
--vfs-cache-max-size 256G \
--vfs-cache-max-age 4h \
--vfs-cache-poll-interval 5m \
--vfs-read-ahead 2G \
--bwlimit-file 32M
ExecStop=/bin/fusermount -uz /home/username/.mount
ExecStartPost=/usr/bin/rclone rc vfs/refresh recursive=true --rc-addr 127.0.0.1:5572 _async=true
Restart=on-failure
User=username
Group=username

[Install]
WantedBy=multi-user.target

Uninstalling rclone and then reinstalling v1.60.1 resolves the issue and my mount works normally.

[ncw (Nick Craig-Wood)]

@thesethguy can you open a new issue on Github about this please?

[Animosity022]

Known issue already ->

Failing to run rc commands against a specific rc address

[thesethguy]

Thanks! Almost didn't see this, .

May all of you be blessed with the most dank stuff for the things you do.

[Animosity022]

I made a quick update to my service script.

ExecStartPost=/usr/bin/rclone rc vfs/refresh recursive=true --url 127.0.0.1:5575 _async=true

That will fix it until the issue is fixed as you can replace the rc-addr with url and that works.

[thesethguy]

Cool, I had tried changing the port to 5573 in both the ExecStart and ExecStartPost sections, and still had the same issue. This totally makes sense after reading the GitHub issue, changing it over to use --url instead. Sounds like they're working on reverting it back later.

Cheers

[El_Rico]

While creating the logs, it dawned on me that there is another issue, because restic uses the --stdio option by default. That means it should not matter if I am already using port 8080 elsewhere, since rclone should not be using tcp/ip - unless I am misunderstanding the docs. I have created #6646 for that.

While testing, I noticed that tcp/ip connections from restic will fail, unless I use the --stdio option. I have created #6647 for that. Alas I cannot link to this and the next issue, since this forum only allows two links per post for new users.

Finally I created #6648, since rclone does not seem to close unix sockets cleanly.

[footlooseboss]

#6647

#6648

[ncw (Nick Craig-Wood)]

I've just uploaded v1.61.1 which should fix all the issues mentioned in this thread so far!

(Forgot to update the website - should be OK now!)

[thesethguy]

You're a beast! Cheers

Quick edit: Just updated from 1.60.1 to 1.61.1 successfully with the selfupdate command and the systemctl start rclone command worked flawlessly afterwards. No errors. The logs also look clean. Thanks again!

[El_Rico]

Thank you for the fixes! #6647 does not seem to fixed on my system though.

Now that #6646 is fixed, I am wondering if I should revert back to --stdio, or stick with sockets. Does anybody know which is more efficient?

[ncw (Nick Craig-Wood)]

@El_Rico

Now that #6646 is fixed, I am wondering if I should revert back to --stdio, or stick with sockets. Does anybody know which is more efficient?

There probably isn't a lot in it. I use --stdio as it saves opening a socket.