"rclone mount" does not work with multi thread downloading using "--vfs-cache-mode full" with version v1.55.1 and v1.54.1

Suspected Bug
1

What is the problem you are having with rclone?

"rclone mount" does not work with multi thread downloading using "--vfs-cache-mode full" with version v1.55.1 (v1.54.1 does not work neither).
By the way, v1.48.0 (the version this feature first introduced) works fine.

What is your rclone version (output from rclone version)

v1.55.1 and v1.54.1

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows 7, 64 bit

Which cloud storage system are you using? (eg Google Drive)

Amazon S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone.exe mount REMOTE:BUCKET Z: --no-check-certificate --vfs-cache-mode full --vfs-cache-max-age 1s --dir-cache-time 1s --s3-upload-cutoff 200M --s3-chunk-size 5M --s3-upload-concurrency 5 --multi-thread-cutoff 8M --multi-thread-streams 3 --rc --rc-addr=127.0.0.1:**** --rc-user=**** --stats 1s --stats-log-level "DEBUG" --log-level "DEBUG"

The rclone config contents with secrets removed.

[REMOTE]
type = s3
provider = Other
env_auth = false
access_key_id = ****
secret_access_key = ****
endpoint = ****
storage_class = 
server_side_encryption =

A log from the command with the -vv flag

I use keyword "multi" to grep the log file, when using v1.48.0, it appears to have something like below:

2021/04/28 14:18:08 DEBUG : FILENAME: multi-thread copy: stream 3/3 (38273024-57318912) size 18.164M starting
2021/04/28 14:18:08 DEBUG : FILENAME: multi-thread copy: stream 1/3 (0-19136512) size 18.250M starting
2021/04/28 14:18:08 DEBUG : FILENAME: multi-thread copy: stream 2/3 (19136512-38273024) size 18.250M starting

Using version v1.55.1 and v1.54.1, no DEBUG information like above logged (the sign of multi thread downloading not working?). When testing, the executable file has been changed to specific version, other configuration and command line remains the same.

1 Like
2

This is correct, multithread download was a casualty in making the vfs cache work without downloading the whole file first in full mode.

There is a plan to bring in back though quite soon (maybe for 1.56).

3

Thanks for the quick response!

As I tested, multi-thread downloading does not work in writes mode neither (v1.55.1, v1.54.1 and v1.48.0).

May I know which version does this casualty happened? For using it, I might considering use the version before that happened.

4

This feature was released in v1.53.0 so anything before that.

For my interest, can you tell my why this feature is important to you? Is it the total download speed, or getting the file into cache as quickly as possible? Or something else?

Also can you tell me the size of files you are using?

5

Thank you for this version confirmation!

This feature is important because of the ability to control the multi-thread concurrency. Say, we have very high speed networking and memory capacity, transferring big files as quickly as possible is an attractive feature.

I am not sure, for uploading/downloading files, is "vfs cache" a must have option to increasing the throughput? If we do not care about the cache, transferring files with reasonable concurrency and chunk size (suppose we have very good networking and memory capacity) would be sufficient for us to achieve this?

I cannot expect/limit the size of the file being downloading, maybe hundreds/thousands of GB.

Actually, one use case would be: mount the remote on a giant machine, and many clients connected to this guy via network, then we could transferring files happily once this mount setup has been configured.

1 Like
6

Here is the issue if you want to subscribe for updates

1 Like
7

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.