What is the problem you are having with rclone?
"rclone mount" stops when uploading large set of small files.
What is your rclone version (output from rclone version
)
rclone v1.55.1
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Windows 10, 64 bit
Which cloud storage system are you using? (eg Google Drive)
AWS S3
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
rclone.exe mount REMOTE:BUCKET * -o FileSecurity=\\"D:P(A;;FA;;;WD)\\" --use-server-modtime --update --vfs-cache-mode writes --vfs-cache-max-age 1s --stats 5s
The rclone config contents with secrets removed.
[REMOTE]
type = s3
provider = Other
env_auth = false
access_key_id = ****
secret_access_key = ****
endpoint = ****
storage_class =
server_side_encryption =
A log from the command with the -vv
flag
2021/05/27 14:15:44 INFO :
Transferred: 106.430M / 106.430 MBytes, 100%, 269.565 kBytes/s, ETA 0s
Transferred: 41262 / 41262, 100%
Elapsed time: 46m25.1s
2021/05/27 14:15:49 INFO :
Transferred: 106.430M / 106.430 MBytes, 100%, 269.565 kBytes/s, ETA 0s
Transferred: 41262 / 41262, 100%
Elapsed time: 46m30.1s
I am encountering an issue which forces me thinking about how rclone works with mount and writes cache enabled.
There are large set of small files needs to be uploading to remote, in my test case, it would be 100,000 small files (4KB each). After uploading starts for a while, the Windows copy progress bar stops with 0KB/s, the remote does not receive new uploading anymore, it just stops.
I am guessing the command line option --vfs-cache-max-age 1s
is the root cause. Tried without this option and --vfs-cache-max-age 30s
, both finishes uploading without stop (although the speed is very slow, which is 100+KB/s; with large files, the speed can reach 100+MB/s).
If --vfs-cache-max-age 1s
is the root cause, how can I set this option safely? By default, this option is set to 1h as the doc says.
With --vfs-cache-max-age
, is there a timer which responsible for removing cache files when max age exceeds? If this is true, there might be race conditions which related to file uploading and cache removing.