Rclone 1.51 release

I use pandoc to make the html file. I view it as a last resort document!

It would be possible to fix these fby setting a base URL: https://github.com/jgm/pandoc/issues/1751 which will mean the links go to the website at least. I'm not sure how to make the links work internally - that would probably need some rewriting of the markdown.

If you think that is worth doing then please make an issue. Even better figure out which parameter to pass to pandoc (it is called from the Makefile) and send a PR :smile:

Hello Nick,

Thanks for the pointers. I investigated pandoc and the standard Markdown and it seems to be possible:

https://pandoc.org/MANUAL.html#pandocs-markdown

Whether it's worth or not I'm not sure, but I know this is just the thing to tickle my OCD :grin:-) so I'm going ahead and doing some tests.

If everything goes right, I hope to be sending you a PR soon :slight_smile:

Cheers,
-- Durval.

Great!

Look forward to receiving it :slight_smile:

1 Like

@ncw this release causes problems IMHO.

  1. Constantly high IOWAIT during heavy access to rclone mount
    See: https://imgur.com/a/GBGz11u
    I have only two process accessing my rclone mount, local Plex server and samba clients. I never have problem with iowait before this upgrade. Right after updating rclone, iowait becomes constantly high, as high as 50-80% checking from netdata. This is while I'm doing local plex scanning and kodi remote scanning via smbclient.
    When the rclone directory is in vfs cache, iowait jumps to constantly 20% per process, when not in vfs cache, such as new files and folders, iowait jumps even higher.

  2. rclone mount is significantly more CPU hungry. Compared to before 1.50, this latest version cause rclone to eat more CPU for every simple task. For batch tasks such as ffprobe/Plex scans, each rclone mount process use 50-80% of vCPU more often than older version.

So for example, I have 6 rclone mount points, 1 localhost Plex server, 1 remote Plex server and 1 remote Kodi that both scans the rclone mount via samba and these all scanning or batch files access eats up a lot of CPU and constantly high iowait. Compare to before1.50 version, that same tasks doesn't cause high iowait and consume less than 20% of one vCPU.

r these bugs ?
I think I will downgrade back.

Edit:

This is my iowait executing the same tasks and batch process after downgrading back to version 1.50.2

@JamesY - can you open up a question template and fill out all the info. Thanks!

Howdy @ncw,

so I'm going ahead and doing some tests.

Test results are GO! :wink: WIth this simple modification, I was able to make internal references on the MANUAL.md file that got properly translated to internal links in MANUAL.html by pandoc:

@ncw, is this satisfatory? If so, I will start to go through the rest of MANUAL.md and fix the other references.

Cheers,
-- Durval.

I had the same problem in the following environment.
mount googledrive -> cache -> crypt for plex media sever

What is the problem you are having with rclone?
Constantly high IOWAIT

What is your rclone version (output from rclone version)
v1.51

Which OS you are using and how many bits (eg Windows 7, 64 bit)
Ubuntu 18.04.3 LTS / 64 Bit

Which cloud storage system are you using? (eg Google Drive)
Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)
/usr/bin/rclone mount gdrive_cache_crypt: /mnt/gc
--config /mnt/hdd/rclone/rclone.conf
--use-mmap
--fast-list
--vfs-cache-mode writes
--buffer-size 0M
--cache-dir /mnt/hdd/tmp/gc/cache
--cache-chunk-size 5M
--cache-chunk-total-size 100G
--cache-chunk-no-memory
--cache-db-path /mnt/hdd/tmp/gc/chunk
--cache-chunk-path /mnt/hdd/tmp/gc/chunk
--allow-other
--umask 0
--tpslimit 10
--tpslimit-burst 20
--rc
--rc-user XXXX
--rc-pass XXXX

@Animosity022

Sure here

Constantly high IOWAIT

Ah, probably the bit I forgot to explain is that MANUAL.md is made by bin/make_manual.py from the individual md pages. So it might be that we do the substitutions in that python program instead.

Hello @ncw,

Sure thing, I will have a look at it (and them).

Cheers,
-- Durval.

Is this a very spesific bug, or do you think this might be related to the corrupted modtimes issues I reported to you a while back? Most of it turned out to be the damned antivirus interfering - but there were still some lingering issues even then.

Maybe I should run a thorough re-test on that so see if it still happens... I have kind of avoided large uploads via mount since then.

  • Enable async reads for a 20% speedup (Nick Craig-Wood)

Does this mean we can now use async_read=true on mergerfs configs ?

Hmm, don't know. I spent a lot of time fixing rename stuff in the VFS layer for 1.51 - including the modtimes, so it could be a fix!

Rclone is essentially setting its own version of async_read=true now. I'm not sure what relation that would have with mergerfs use though...

Hello @ncw,

So I did, and here's the result:

With the above modifications to bin/make_manual.py, it's now a simple matter of inserting the proper (#PROVIDER.md) links on the necessary md pages, most notably about.md, as I started doing here:

So, what do you think?

Cheers,
-- Durval.

I think your patch to make_manual.py looks very good :slight_smile: I think it should be make_manual.py that changes the link references - was that what you were thinking? Otherwise the linking won't work on the website where the docs are separate pages.

@ncw, thanks for having a look at my patch, and sorry for taking so long to get back to you ("real life" intervened,as usual...).

My initial approach was to solve just the manual.md/manual.html linkings; I did not consider the individual HTML pages. I will get back to it and have a look at this as soon as I have some free time again.

Cheers,
-- Durval.

1 Like

Hi,

When adding Jottacloud to my new rClone installation on MacOS I get the error: "illegal base64 data at input byte 264".

Sample of my 'Login token': eyQ1c2VybmFtZSI6Imphbm5pY2tuaWpzb2x0IiwicmVhbG0iOiJqb3R0YWNsb3VkIiwid2VsbF9Abm93bl9saW5rIfoiaGR0cHM6Ly9pZC5qb3R0Y1Nsb0VkLmNvbS9hdXRoL3JlYWxtcy9qbQR0YWNsb3VkLy53ZZxsLWtub3d2L29wZW5pZC1jb25maWd1cmF0aW9uIiwiYXV0aF90b2tlbiI6IjQ1RTVFNkM0NTRG5DY0QTU3QjA2ODU1MEY0NURTMThDQn0 (changed a lot of letters and numbers, but same length)

Thanks!

Please create a new post and fill the question template.

I heard that there was a --backend-encoding parameter added. By the way, I don't understand readme.html.

How do I take advantage of this option when mounting a GDrive on Windows? How can I get rid of the "'" character?

2020-04-14_132542