Rclone 1.51 release

Great update, Nick! Thanks!!

I just noticed .deb packages are being released (probably for some time, but just noticed it today), so I've removed the previous version I had manually installed on /usr/local/* from the release ZIP file, and installed the deb on one of my Ubuntu (18.04) and on one of my Devuan (2.0 ASCII) machines. Worked perfectly on both instances.

I then examined dpkg -L rclone and noticed it includes a /usr/share/rclone/README.html (OK, the ZIP included one too, but I never installed it :stuck_out_tongue:), so I opened with a browser (eg, firefox file:///usr/share/rclone/README.html for a look).

I found some discrepancies/oddities I think might be issues:

  1. At:

     Changelog
     v1.51.0 - 2020-02-01
    
     New backends
         Memory (Nick Craig-Wood)
         Sugarsync (Nick Craig-Wood)
    

"Memory" and "Sugarsync" above are shown as links, but pointing to "file://memory" and "file://sugarsync" respectively, which of course don't work.

  1. At:

     Configure
     [...]
     See the following for detailed instructions for
     1Fichier
     Alias
     Amazon Drive
     [... lots of other REMOTEs  ...] 
    

Here, the issue is that each REMOTE ("1Fichier", "Alias", etc) links to https://rclone.org/REMOTE, and not to #REMOTE so it could be read inside the same local README.html document instead of going on-line to the Rclone website (which I think kinda defeats the purpose of having a local README.html file).

I'm not sure whether this is something 'fixable' (perhaps it's an unavoidable "feature" of whatever tool is used to generate the README.html file, or something similar), but I thought it would be better to point it out.

@ncw, if you want me to open an issue for this, please tell.

Cheers,
-- Durval.

I use pandoc to make the html file. I view it as a last resort document!

It would be possible to fix these fby setting a base URL: https://github.com/jgm/pandoc/issues/1751 which will mean the links go to the website at least. I'm not sure how to make the links work internally - that would probably need some rewriting of the markdown.

If you think that is worth doing then please make an issue. Even better figure out which parameter to pass to pandoc (it is called from the Makefile) and send a PR :smile:

Hello Nick,

Thanks for the pointers. I investigated pandoc and the standard Markdown and it seems to be possible:

https://pandoc.org/MANUAL.html#pandocs-markdown

Whether it's worth or not I'm not sure, but I know this is just the thing to tickle my OCD :grin:-) so I'm going ahead and doing some tests.

If everything goes right, I hope to be sending you a PR soon :slight_smile:

Cheers,
-- Durval.

Great!

Look forward to receiving it :slight_smile:

1 Like

@ncw this release causes problems IMHO.

  1. Constantly high IOWAIT during heavy access to rclone mount
    See: https://imgur.com/a/GBGz11u
    I have only two process accessing my rclone mount, local Plex server and samba clients. I never have problem with iowait before this upgrade. Right after updating rclone, iowait becomes constantly high, as high as 50-80% checking from netdata. This is while I'm doing local plex scanning and kodi remote scanning via smbclient.
    When the rclone directory is in vfs cache, iowait jumps to constantly 20% per process, when not in vfs cache, such as new files and folders, iowait jumps even higher.

  2. rclone mount is significantly more CPU hungry. Compared to before 1.50, this latest version cause rclone to eat more CPU for every simple task. For batch tasks such as ffprobe/Plex scans, each rclone mount process use 50-80% of vCPU more often than older version.

So for example, I have 6 rclone mount points, 1 localhost Plex server, 1 remote Plex server and 1 remote Kodi that both scans the rclone mount via samba and these all scanning or batch files access eats up a lot of CPU and constantly high iowait. Compare to before1.50 version, that same tasks doesn't cause high iowait and consume less than 20% of one vCPU.

r these bugs ?
I think I will downgrade back.

Edit:

This is my iowait executing the same tasks and batch process after downgrading back to version 1.50.2

@JamesY - can you open up a question template and fill out all the info. Thanks!

Howdy @ncw,

so I'm going ahead and doing some tests.

Test results are GO! :wink: WIth this simple modification, I was able to make internal references on the MANUAL.md file that got properly translated to internal links in MANUAL.html by pandoc:

@ncw, is this satisfatory? If so, I will start to go through the rest of MANUAL.md and fix the other references.

Cheers,
-- Durval.

I had the same problem in the following environment.
mount googledrive -> cache -> crypt for plex media sever

What is the problem you are having with rclone?
Constantly high IOWAIT

What is your rclone version (output from rclone version)
v1.51

Which OS you are using and how many bits (eg Windows 7, 64 bit)
Ubuntu 18.04.3 LTS / 64 Bit

Which cloud storage system are you using? (eg Google Drive)
Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)
/usr/bin/rclone mount gdrive_cache_crypt: /mnt/gc
--config /mnt/hdd/rclone/rclone.conf
--use-mmap
--fast-list
--vfs-cache-mode writes
--buffer-size 0M
--cache-dir /mnt/hdd/tmp/gc/cache
--cache-chunk-size 5M
--cache-chunk-total-size 100G
--cache-chunk-no-memory
--cache-db-path /mnt/hdd/tmp/gc/chunk
--cache-chunk-path /mnt/hdd/tmp/gc/chunk
--allow-other
--umask 0
--tpslimit 10
--tpslimit-burst 20
--rc
--rc-user XXXX
--rc-pass XXXX

@Animosity022

Sure here

Constantly high IOWAIT

Ah, probably the bit I forgot to explain is that MANUAL.md is made by bin/make_manual.py from the individual md pages. So it might be that we do the substitutions in that python program instead.

Hello @ncw,

Sure thing, I will have a look at it (and them).

Cheers,
-- Durval.

Is this a very spesific bug, or do you think this might be related to the corrupted modtimes issues I reported to you a while back? Most of it turned out to be the damned antivirus interfering - but there were still some lingering issues even then.

Maybe I should run a thorough re-test on that so see if it still happens... I have kind of avoided large uploads via mount since then.

  • Enable async reads for a 20% speedup (Nick Craig-Wood)

Does this mean we can now use async_read=true on mergerfs configs ?

Hmm, don't know. I spent a lot of time fixing rename stuff in the VFS layer for 1.51 - including the modtimes, so it could be a fix!

Rclone is essentially setting its own version of async_read=true now. I'm not sure what relation that would have with mergerfs use though...

Hello @ncw,

So I did, and here's the result:

With the above modifications to bin/make_manual.py, it's now a simple matter of inserting the proper (#PROVIDER.md) links on the necessary md pages, most notably about.md, as I started doing here:

So, what do you think?

Cheers,
-- Durval.

I think your patch to make_manual.py looks very good :slight_smile: I think it should be make_manual.py that changes the link references - was that what you were thinking? Otherwise the linking won't work on the website where the docs are separate pages.

@ncw, thanks for having a look at my patch, and sorry for taking so long to get back to you ("real life" intervened,as usual...).

My initial approach was to solve just the manual.md/manual.html linkings; I did not consider the individual HTML pages. I will get back to it and have a look at this as soon as I have some free time again.

Cheers,
-- Durval.

1 Like

Hi,

When adding Jottacloud to my new rClone installation on MacOS I get the error: "illegal base64 data at input byte 264".

Sample of my 'Login token': eyQ1c2VybmFtZSI6Imphbm5pY2tuaWpzb2x0IiwicmVhbG0iOiJqb3R0YWNsb3VkIiwid2VsbF9Abm93bl9saW5rIfoiaGR0cHM6Ly9pZC5qb3R0Y1Nsb0VkLmNvbS9hdXRoL3JlYWxtcy9qbQR0YWNsb3VkLy53ZZxsLWtub3d2L29wZW5pZC1jb25maWd1cmF0aW9uIiwiYXV0aF90b2tlbiI6IjQ1RTVFNkM0NTRG5DY0QTU3QjA2ODU1MEY0NURTMThDQn0 (changed a lot of letters and numbers, but same length)

Thanks!

Please create a new post and fill the question template.