VFS-cache does not seem to work for mount when copying multiple files

What is the problem you are having with rclone?

I'm running rclone with the command line from below. If I copy a large single file (around 1GB) into the mounted directory this goes really fast as expected (because of the cache). According to the webgui this file is uploaded then and this upload finishes a few minutes after the copy ended. So everything fine here. What I noticed is that the copy command doesn't get back after that (I also tried to copy with pv file.img > /jottacloud-folder/file.img to see a status bar, it was really fast but at 100% it stopped and did not terminate) . I'm not sure if the reason for that is rclone or something else. But the file is copied correctly

But if I copy 50 files, about 3MB each, about 150MB overall they don't seem to hit the cache. The process of copying seemingly takes exactly as long as the upload takes (and the upload speed limit seems to be ignored). If I look into the cache folder I can see that the files only appear there one after another really slow. Looking at the webgui it almost looks like they hit the cache folder at the same time as they're uploaded.

The setup is the same in both scenarios. If I copy the files from one folder on the hard drive to another (both not rclone mounted) it goes really fast so my filesystem/SSD isn't the bottleneck here.

Is it possible that after copying the file into the cache rclone does not tell the program which issued the copy that it's finished now (=in cache) but only after it's uploaded?

What is your rclone version (output from rclone version)

rclone v1.52.3
- os/arch: linux/arm
- go version: go1.14.7

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Raspberry Pi OS, 32bit on a Raspberry Pi 4 (4GB Ram) booting from SSD attached via USB 3.0

Which cloud storage system are you using? (eg Google Drive)


The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount jottacloud:/backups /home/pi/mnt/jottacloud/backups --retries 5 --retries-sleep 5m --cache-dir ~/rcloneCache --vfs-cache-mode full --vfs-cache-max-size 350G -vv --log-file log.txt --rc --rc-web-gui --rc-htpasswd ./htpasswd --rc-serve --rc-web-gui-no-open-browser --bwlimit "04:00,off 07:00,600K 16:00,350K" --rc-addr :5572 --umask 770 --allow-other

Note: The cache directory lies on the (fast) SSD (not SD card). There is more than 400GB of free space on this SD so rclone could use the whole 350GB of cache that I allow it to use.

A log from the command with the -vv flag

Log was a bit too big for pastebin, so here as file:
Note: Don't get confused by the directory jottacloud. This is not a mounted folder, only backups inside of this is the mounted jottacloud. The non-cloud folder is the one with the data that I copy into the backups folder (which is the mounted jottacloud)

rclone-log.txt (499.5 KB)

Thanks in advance

i can't answer your question, but i thought you should know that your umssk value is odd. umask is not a chmod value, umask subtracts from whatever the file permission is.

022 or 002 is the most common. the way you have it could give too much permissions.

This bit of code has been rewritten it the beta which will shortly become 1.53

Can you try your test with the beta? It will behave quite differently uploading the cached goes in the background.

Oh thank you, I did not know that

Okay I will as soon as I have time. I'll get back to you.

1 Like

I just tried it with

rclone v1.52.3-309-g47d08ac1-beta
- os/arch: linux/arm
- go version: go1.15

And it seems to work just fine! Good job!

Great :smiley:

It should be released by the end of the month.

1 Like

I am also having this issue, does someone have any solution for this?

The solution is a few posts above. You need to use the beta or wait for the new release.

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.