"rclone mount" stops when uploading large set of small files

What is the problem you are having with rclone?

"rclone mount" stops when uploading large set of small files.

What is your rclone version (output from rclone version)

rclone v1.55.1

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows 10, 64 bit

Which cloud storage system are you using? (eg Google Drive)

AWS S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone.exe mount REMOTE:BUCKET * -o FileSecurity=\\"D:P(A;;FA;;;WD)\\" --use-server-modtime --update --vfs-cache-mode writes --vfs-cache-max-age 1s --stats 5s

The rclone config contents with secrets removed.

[REMOTE]
type = s3
provider = Other
env_auth = false
access_key_id = ****
secret_access_key = ****
endpoint = ****
storage_class = 
server_side_encryption =

A log from the command with the -vv flag

2021/05/27 14:15:44 INFO  : 
Transferred:   	  106.430M / 106.430 MBytes, 100%, 269.565 kBytes/s, ETA 0s
Transferred:        41262 / 41262, 100%
Elapsed time:     46m25.1s

2021/05/27 14:15:49 INFO  : 
Transferred:   	  106.430M / 106.430 MBytes, 100%, 269.565 kBytes/s, ETA 0s
Transferred:        41262 / 41262, 100%
Elapsed time:     46m30.1s

I am encountering an issue which forces me thinking about how rclone works with mount and writes cache enabled.

There are large set of small files needs to be uploading to remote, in my test case, it would be 100,000 small files (4KB each). After uploading starts for a while, the Windows copy progress bar stops with 0KB/s, the remote does not receive new uploading anymore, it just stops.

I am guessing the command line option --vfs-cache-max-age 1s is the root cause. Tried without this option and --vfs-cache-max-age 30s, both finishes uploading without stop (although the speed is very slow, which is 100+KB/s; with large files, the speed can reach 100+MB/s).

If --vfs-cache-max-age 1s is the root cause, how can I set this option safely? By default, this option is set to 1h as the doc says.

With --vfs-cache-max-age, is there a timer which responsible for removing cache files when max age exceeds? If this is true, there might be race conditions which related to file uploading and cache removing.

If you want to see what's going on, you need to run the mount with -vv and share the full debug log.

debug.1.log (2.9 MB)
debug.2.log (3.1 MB)
debug.3.log (3.0 MB)

Hello, here is the full debug log, I divided it into 3 log files as the maximum file size limit is 4096KB.
Hope this helps.

The cache expiry is quite crude in rclone at the moment. So it will be enumerating all the 100,000 files every second so using up all your CPU doing that.

Is that what you see?

The CPU usage is increased about 40%, however, not 100% in this case. After the uploading stops, the CPU usage decreases to normal state.

I think --vfs-cache-max-age 1s is not a good choice. Seems using --vfs-cache-max-size and keep --vfs-cache-max-age the default value will make the uploading works without stopping.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.