Rclone 1.51 release

Releases
1

Rclone 1.51 has been released. Find it here https://rclone.org/downloads/

Highlights:

Thank you to all the contributors to this release, those who contributed code or doc fixes (46 people!) or made issues or answered questions in the forum - your help is much appreciated!

Thanks in particular to @Animosity022 for helping with the backlog of issues in GitHub.

v1.51.0 - 2020-02-01

  • New backends
  • New Features
    • Adjust all backends to have --backend-encoding parameter (Nick Craig-Wood)
      • this enables the encoding for special characters to be adjusted or disabled
    • Add --max-duration flag to control the maximum duration of a transfer session (boosh)
    • Add --expect-continue-timeout flag, default 1s (Nick Craig-Wood)
    • Add --no-check-dest flag for copying without testing the destination (Nick Craig-Wood)
    • Implement --order-by flag to order transfers (Nick Craig-Wood)
    • accounting
      • Don't show entries in both transferring and checking (Nick Craig-Wood)
      • Add option to delete stats (Aleksandar Jankovic)
    • build
      • Compress the test builds with gzip (Nick Craig-Wood)
      • Implement a framework for starting test servers during tests (Nick Craig-Wood)
    • cmd: Always print elapsed time to tenth place seconds in progress (Gary Kim)
    • config
      • Add --password-command to allow dynamic config password (Damon Permezel)
      • Give config questions default values (Nick Craig-Wood)
      • Check a remote exists when creating a new one (Nick Craig-Wood)
    • copyurl: Add --stdout flag to write to stdout (Nick Craig-Wood)
    • dedupe: Implement keep smallest too (Nick Craig-Wood)
    • hashsum: Add flag --base64 flag (landall)
    • lsf: Speed up on s3/swift/etc by not reading mimetype by default (Nick Craig-Wood)
    • lsjson: Add --no-mimetype flag (Nick Craig-Wood)
    • rc: Add methods to turn on blocking and mutex profiling (Nick Craig-Wood)
    • rcd
      • Adding group parameter to stats (Chaitanya)
      • Move webgui apart; option to disable browser (Xiaoxing Ye)
    • serve sftp: Add support for public key with auth proxy (Paul Tinsley)
    • stats: Show deletes in stats and hide zero stats (anuar45)
  • Bug Fixes
    • accounting
      • Fix error counter counting multiple times (Ankur Gupta)
      • Fix error count shown as checks (Cnly)
      • Clear finished transfer in stats-reset (Maciej Zimnoch)
      • Added StatsInfo locking in statsGroups sum function (Michał Matczuk)
    • asyncreader: Fix EOF error (buengese)
    • check: Fix --one-way recursing more directories than it needs to (Nick Craig-Wood)
    • chunkedreader: Disable hash calculation for first segment (Nick Craig-Wood)
    • config
      • Do not open browser on headless on drive/gcs/google photos (Xiaoxing Ye)
      • SetValueAndSave ignore error if config section does not exist yet (buengese)
    • cmd: Fix completion with an encrypted config (Danil Semelenov)
    • dbhashsum: Stop it returning UNSUPPORTED on dropbox (Nick Craig-Wood)
    • dedupe: Add missing modes to help string (Nick Craig-Wood)
    • operations
      • Fix dedupe continuing on errors like insufficientFilePermisson (SezalAgrawal)
      • Clear accounting before low level retry (Maciej Zimnoch)
      • Write debug message when hashes could not be checked (Ole Schütt)
      • Move interface assertion to tests to remove pflag dependency (Nick Craig-Wood)
      • Make NewOverrideObjectInfo public and factor uses (Nick Craig-Wood)
    • proxy: Replace use of bcrypt with sha256 (Nick Craig-Wood)
    • vendor
      • Update bazil.org/fuse to fix FreeBSD 12.1 (Nick Craig-Wood)
      • Update github.com/t3rm1n4l/go-mega to fix mega "illegal base64 data at input byte 22" (Nick Craig-Wood)
      • Update termbox-go to fix ncdu command on FreeBSD (Kuang-che Wu)
      • Update t3rm1n4l/go-mega - fixes mega: couldn't login: crypto/aes: invalid key size 0 (Nick Craig-Wood)
  • Mount
    • Enable async reads for a 20% speedup (Nick Craig-Wood)
    • Replace use of WriteAt with Write for cache mode >= writes and O_APPEND (Brett Dutro)
    • Make sure we call unmount when exiting (Nick Craig-Wood)
    • Don't build on go1.10 as bazil/fuse no longer supports it (Nick Craig-Wood)
    • When setting dates discard out of range dates (Nick Craig-Wood)
  • VFS
    • Add a newly created file straight into the directory (Nick Craig-Wood)
    • Only calculate one hash for reads for a speedup (Nick Craig-Wood)
    • Make ReadAt for non cached files work better with non-sequential reads (Nick Craig-Wood)
    • Fix edge cases when reading ModTime from file (Nick Craig-Wood)
    • Make sure existing files opened for write show correct size (Nick Craig-Wood)
    • Don't cache the path in RW file objects to fix renaming (Nick Craig-Wood)
    • Fix rename of open files when using the VFS cache (Nick Craig-Wood)
    • When renaming files in the cache, rename the cache item in memory too (Nick Craig-Wood)
    • Fix open file renaming on drive when using --vfs-cache-mode writes (Nick Craig-Wood)
    • Fix incorrect modtime for mv into mount with --vfs-cache-modes writes (Nick Craig-Wood)
    • On rename, rename in cache too if the file exists (Anagh Kumar Baranwal)
  • Local
    • Make source file being updated errors be NoLowLevelRetry errors (Nick Craig-Wood)
    • Fix update of hidden files on Windows (Nick Craig-Wood)
  • Cache
  • Crypt
    • Reorder the filename encryption options (Thomas Eales)
    • Correctly handle trailing dot (buengese)
  • Chunker
    • Reduce length of temporary suffix (Ivan Andreev)
  • Drive
    • Add --drive-stop-on-upload-limit flag to stop syncs when upload limit reached (Nick Craig-Wood)
    • Add --drive-use-shared-date to use date file was shared instead of modified date (Garry McNulty)
    • Make sure invalid auth for teamdrives always reports an error (Nick Craig-Wood)
    • Fix --fast-list when using appDataFolder (Nick Craig-Wood)
    • Use multipart resumable uploads for streaming and uploads in mount (Nick Craig-Wood)
    • Log an ERROR if an incomplete search is returned (Nick Craig-Wood)
    • Hide dangerous config from the configurator (Nick Craig-Wood)
  • Dropbox
    • Treat insufficient_space errors as non retriable errors (Nick Craig-Wood)
  • Jottacloud
    • Use new auth method used by official client (buengese)
    • Add URL to generate Login Token to config wizard (Nick Craig-Wood)
    • Add support whitelabel versions (buengese)
  • Koofr
    • Use rclone HTTP client. (jaKa)
  • Onedrive
    • Add Sites.Read.All permission (Benjamin Richter)
    • Add support "Retry-After" header (Motonori IWAMURO)
  • Opendrive
    • Implement --opendrive-chunk-size (Nick Craig-Wood)
  • S3
    • Re-implement multipart upload to fix memory issues (Nick Craig-Wood)
    • Add --s3-copy-cutoff for size to switch to multipart copy (Nick Craig-Wood)
    • Add new region Asia Patific (Hong Kong) (Outvi V)
    • Reduce memory usage streaming files by reducing max stream upload size (Nick Craig-Wood)
    • Add --s3-list-chunk option for bucket listing (Thomas Kriechbaumer)
    • Force path style bucket access to off for AWS deprecation (Nick Craig-Wood)
    • Use AWS web identity role provider if available (Tennix)
    • Add StackPath Object Storage Support (Dave Koston)
    • Fix ExpiryWindow value (Aleksandar Jankovic)
    • Fix DisableChecksum condition (Aleksandar Janković)
    • Fix URL decoding of NextMarker (Nick Craig-Wood)
  • SFTP
    • Add --sftp-skip-links to skip symlinks and non regular files (Nick Craig-Wood)
    • Retry Creation of Connection (Sebastian Brandt)
    • Fix "failed to parse private key file: ssh: not an encrypted key" error (Nick Craig-Wood)
    • Open files for update write only to fix AWS SFTP interop (Nick Craig-Wood)
  • Swift
    • Reserve segments of dynamic large object when delete objects in container what was enabled versioning. (Nguyễn Hữu Luân)
    • Fix parsing of X-Object-Manifest (Nick Craig-Wood)
    • Update OVH API endpoint (unbelauscht)
  • WebDAV
    • Make nextcloud only upload SHA1 checksums (Nick Craig-Wood)
    • Fix case of "Bearer" in Authorization: header to agree with RFC (Nick Craig-Wood)
    • Add Referer header to fix problems with WAFs (Nick Craig-Wood)
12 Likes
2

Thanks for the BIG update!

1 Like
3

Huge. Awesome. And thank you as always.

1 Like
4

This is really a huge update, awesome work. :heart_eyes:

1 Like
5

Great update, Nick! Thanks!!

I just noticed .deb packages are being released (probably for some time, but just noticed it today), so I've removed the previous version I had manually installed on /usr/local/* from the release ZIP file, and installed the deb on one of my Ubuntu (18.04) and on one of my Devuan (2.0 ASCII) machines. Worked perfectly on both instances.

I then examined dpkg -L rclone and noticed it includes a /usr/share/rclone/README.html (OK, the ZIP included one too, but I never installed it :stuck_out_tongue:), so I opened with a browser (eg, firefox file:///usr/share/rclone/README.html for a look).

I found some discrepancies/oddities I think might be issues:

  1. At:

     Changelog
 v1.51.0 - 2020-02-01

 New backends
     Memory (Nick Craig-Wood)
     Sugarsync (Nick Craig-Wood)

"Memory" and "Sugarsync" above are shown as links, but pointing to "file://memory" and "file://sugarsync" respectively, which of course don't work.

  1. At:

     Configure
 [...]
 See the following for detailed instructions for
 1Fichier
 Alias
 Amazon Drive
 [... lots of other REMOTEs  ...]

Here, the issue is that each REMOTE ("1Fichier", "Alias", etc) links to https://rclone.org/REMOTE, and not to #REMOTE so it could be read inside the same local README.html document instead of going on-line to the Rclone website (which I think kinda defeats the purpose of having a local README.html file).

I'm not sure whether this is something 'fixable' (perhaps it's an unavoidable "feature" of whatever tool is used to generate the README.html file, or something similar), but I thought it would be better to point it out.

@ncw, if you want me to open an issue for this, please tell.

Cheers,
-- Durval.

6

I use pandoc to make the html file. I view it as a last resort document!

It would be possible to fix these fby setting a base URL: https://github.com/jgm/pandoc/issues/1751 which will mean the links go to the website at least. I'm not sure how to make the links work internally - that would probably need some rewriting of the markdown.

If you think that is worth doing then please make an issue. Even better figure out which parameter to pass to pandoc (it is called from the Makefile) and send a PR :smile:

7

Hello Nick,

Thanks for the pointers. I investigated pandoc and the standard Markdown and it seems to be possible:

https://pandoc.org/MANUAL.html#pandocs-markdown

Whether it's worth or not I'm not sure, but I know this is just the thing to tickle my OCD :grin:-) so I'm going ahead and doing some tests.

If everything goes right, I hope to be sending you a PR soon :slight_smile:

Cheers,
-- Durval.

8

Great!

Look forward to receiving it :slight_smile:

1 Like
9

@ncw this release causes problems IMHO.

  1. Constantly high IOWAIT during heavy access to rclone mount
    See: https://imgur.com/a/GBGz11u
    I have only two process accessing my rclone mount, local Plex server and samba clients. I never have problem with iowait before this upgrade. Right after updating rclone, iowait becomes constantly high, as high as 50-80% checking from netdata. This is while I'm doing local plex scanning and kodi remote scanning via smbclient.
    When the rclone directory is in vfs cache, iowait jumps to constantly 20% per process, when not in vfs cache, such as new files and folders, iowait jumps even higher.

  2. rclone mount is significantly more CPU hungry. Compared to before 1.50, this latest version cause rclone to eat more CPU for every simple task. For batch tasks such as ffprobe/Plex scans, each rclone mount process use 50-80% of vCPU more often than older version.

So for example, I have 6 rclone mount points, 1 localhost Plex server, 1 remote Plex server and 1 remote Kodi that both scans the rclone mount via samba and these all scanning or batch files access eats up a lot of CPU and constantly high iowait. Compare to before1.50 version, that same tasks doesn't cause high iowait and consume less than 20% of one vCPU.

r these bugs ?
I think I will downgrade back.

Edit:

This is my iowait executing the same tasks and batch process after downgrading back to version 1.50.2

[Album] Imgur
10

@JamesY - can you open up a question template and fill out all the info. Thanks!

11

Howdy @ncw,

so I'm going ahead and doing some tests.

Test results are GO! :wink: WIth this simple modification, I was able to make internal references on the MANUAL.md file that got properly translated to internal links in MANUAL.html by pandoc:

@ncw, is this satisfatory? If so, I will start to go through the rest of MANUAL.md and fix the other references.

Cheers,
-- Durval.

12

I had the same problem in the following environment.
mount googledrive -> cache -> crypt for plex media sever

What is the problem you are having with rclone?
Constantly high IOWAIT

What is your rclone version (output from rclone version)
v1.51

Which OS you are using and how many bits (eg Windows 7, 64 bit)
Ubuntu 18.04.3 LTS / 64 Bit

Which cloud storage system are you using? (eg Google Drive)
Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)
/usr/bin/rclone mount gdrive_cache_crypt: /mnt/gc
--config /mnt/hdd/rclone/rclone.conf
--use-mmap
--fast-list
--vfs-cache-mode writes
--buffer-size 0M
--cache-dir /mnt/hdd/tmp/gc/cache
--cache-chunk-size 5M
--cache-chunk-total-size 100G
--cache-chunk-no-memory
--cache-db-path /mnt/hdd/tmp/gc/chunk
--cache-chunk-path /mnt/hdd/tmp/gc/chunk
--allow-other
--umask 0
--tpslimit 10
--tpslimit-burst 20
--rc
--rc-user XXXX
--rc-pass XXXX

13

@Animosity022

Sure here

Constantly high IOWAIT

14

Ah, probably the bit I forgot to explain is that MANUAL.md is made by bin/make_manual.py from the individual md pages. So it might be that we do the substitutions in that python program instead.

15

Hello @ncw,

Sure thing, I will have a look at it (and them).

Cheers,
-- Durval.

16

Is this a very spesific bug, or do you think this might be related to the corrupted modtimes issues I reported to you a while back? Most of it turned out to be the damned antivirus interfering - but there were still some lingering issues even then.

Maybe I should run a thorough re-test on that so see if it still happens... I have kind of avoided large uploads via mount since then.

17
  • Enable async reads for a 20% speedup (Nick Craig-Wood)

Does this mean we can now use async_read=true on mergerfs configs ?

18

Hmm, don't know. I spent a lot of time fixing rename stuff in the VFS layer for 1.51 - including the modtimes, so it could be a fix!

19

Rclone is essentially setting its own version of async_read=true now. I'm not sure what relation that would have with mergerfs use though...

20

Hello @ncw,

So I did, and here's the result:

With the above modifications to bin/make_manual.py, it's now a simple matter of inserting the proper (#PROVIDER.md) links on the necessary md pages, most notably about.md, as I started doing here:

So, what do you think?

Cheers,
-- Durval.