Improve transfer start time (tags: local, glusterfs, rclone cache)

Help and Support
1

What is the problem you are having with rclone?

I have setup rclone, with the following parameters:

  --vfs-cache-mode full \
  --vfs-cache-max-size 100G \
  --vfs-read-ahead 512M \
  --no-modtime \
  --no-checksum \
  --read-only \
  --vfs-fast-fingerprint \
  --cache-dir /rclone-cache \
  --poll-interval 30s \
  --dir-cache-time=1000h \
  --timeout=10m \
  --rc \
  --rc-addr=localhost:5572

However am having a minor issue and wanted to know if this can be resolved via tuning some parameters?
My setup is a glusterfs mount, of which I am using the "local" option in rclone to point to it, my reason for using rclone is the effectiveness of the VFS cache making directory traversals near instant.

Now, it seems if i make a full cache using the following command:
rc vfs/refresh recursive=true --rc-addr localhost:5572 _async=true

everything works fine, however it none of the directories are cached it seems it takes about 15 - 20 seconds to start a file transfer process (these are "linux iso's" ranging from 1 - 10GB), and when doing this directly on the glusterfs mount it's instant as expected, and after rclone has cached the directories (such as in the command above) the transfer starts instantly as expected.

So here I sort of have two questions, 1. is my current setup decent in general for my use case? It seems to be hard to find a consensus that applies to my setup on the forums as most people are using remote data stores such as google drive which has various API limits, where I obviously don't have any limits in place. Will the polling also still work with a local glusterfs mount for cache invalidation?

and then 2. Is it possible to fix the issue described above, as make it so that if the directory is not cached the file won't take 15 to 20 seconds to load up?

Run the command 'rclone version' and share the full output of the command.

rclone v1.62.2
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.4.0-146-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.20.2
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

local

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rsync --progress /file/on/rclone/mount/xyz.mkv /local/nvme/drive/

The rclone config contents with secrets removed.

rclone mount gluster:/mnt/data /mnt/rclone-data \
  --vfs-cache-mode full \
  --vfs-cache-max-size 100G \
  --vfs-read-ahead 512M \
  --no-modtime \
  --no-checksum \
  --read-only \
  --vfs-fast-fingerprint \
  --cache-dir /rclone-cache \
  --poll-interval 30s \
  --dir-cache-time=1000h \
  --timeout=10m \
  --rc \
  --rc-addr=localhost:5572

A log from the command with the -vv flag

Not a rclone command, it is rsync in this example

2

If it is helpful then I don't see a problem with it.

Polling for changes doesn't work with the local backend at the moment.

That sounds like a long time for a local directory.

Does any directory on the path to the file have a very large number of entries? That could cause the problem.

Perhaps you could get a log with -vv of what rclone is doing in that 20s?

3

So even though the local mount (glusterfs-client in this case) has a file change, e.g. radarr upgrades file, then the rclone location of said file will be stale unless manually refreshed via some externally built system that does a rc vfs/refresh on the directory the changed file is in? I do think a direct implementation within rclone for this would make sense.

This directory has 7574 files and directories:

Here is the rclone copyto command with the -vv flag and when the directory cache has not been built:

2023/05/01 21:28:25 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "copyto" "-P" "/mnt/rclone-data/movies/group1/When Did You Last See Your Father (2007) [imdb-tt0829098] [tmdb-45791]/When Did You Last See Your Father (2007) [imdb-tt0829098] [tmdb-45791] -aAF.mp4" "/root/testfile.test" "-vv"]
2023/05/01 21:28:25 DEBUG : Creating backend with remote "/mnt/rclone-data/movies/group1/When Did You Last See Your Father (2007) [imdb-tt0829098] [tmdb-45791]/When Did You Last See Your Father (2007) [imdb-tt0829098] [tmdb-45791] -aAF.mp4"
2023/05/01 21:28:25 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/05/01 21:28:37 DEBUG : fs cache: adding new entry for parent of "/mnt/rclone-data/movies/group1/When Did You Last See Your Father (2007) [imdb-tt0829098] [tmdb-45791]/When Did You Last See Your Father (2007) [imdb-tt0829098] [tmdb-45791] -aAF.mp4", "/mnt/rclone-data/movies/group1/When Did You Last See Your Father (2007) [imdb-tt0829098] [tmdb-45791]"
2023/05/01 21:28:37 DEBUG : Creating backend with remote "/root/"
2023/05/01 21:28:37 DEBUG : fs cache: renaming cache item "/root/" to be canonical "/root"
2023-05-01 21:28:37 DEBUG : When Did You Last See Your Father (2007) [imdb-tt0829098] [tmdb-45791] -aAF.mp4: Sizes differ (src 8750573627 vs dst 1628434432)
2023-05-01 21:30:16 DEBUG : When Did You Last See Your Father (2007) [imdb-tt0829098] [tmdb-45791] -aAF.mp4: md5 = f2df294d0a2d9b487654327aa0a1d130 OK
2023-05-01 21:30:16 INFO  : When Did You Last See Your Father (2007) [imdb-tt0829098] [tmdb-45791] -aAF.mp4: Copied (replaced existing) to: testfile.test
Transferred:        8.150 GiB / 8.150 GiB, 100%, 74.913 MiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:      1m50.6s
2023/05/01 21:30:16 INFO  : 
Transferred:        8.150 GiB / 8.150 GiB, 100%, 74.913 MiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:      1m50.6s

We can see that here it takes 12 seconds for the entry to be added to the fs cache and then start the transfer.
Doing this directly on the glusterfs-client mount, which is where the rclone local is pointing to is pretty much instant < 1second.

And then, if we run the command a second time, after the entry has been added to the cache, we can see the transfer starts instantly:

2023/05/01 21:38:50 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "copyto" "-P" "/mnt/rclone-data/movies/group1/When Did You Last See Your Father (2007) [imdb-tt0829098] [tmdb-45791]/When Did You Last See Your Father (2007) [imdb-tt0829098] [tmdb-45791] -aAF.mp4" "/root/testfile2.test" "-vv"]
2023/05/01 21:38:50 DEBUG : Creating backend with remote "/mnt/rclone-data/movies/group1/When Did You Last See Your Father (2007) [imdb-tt0829098] [tmdb-45791]/When Did You Last See Your Father (2007) [imdb-tt0829098] [tmdb-45791] -aAF.mp4"
2023/05/01 21:38:50 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/05/01 21:38:50 DEBUG : fs cache: adding new entry for parent of "/mnt/rclone-data/movies/group1/When Did You Last See Your Father (2007) [imdb-tt0829098] [tmdb-45791]/When Did You Last See Your Father (2007) [imdb-tt0829098] [tmdb-45791] -aAF.mp4", "/mnt/rclone-data/movies/group1/When Did You Last See Your Father (2007) [imdb-tt0829098] [tmdb-45791]"
2023/05/01 21:38:50 DEBUG : Creating backend with remote "/root/"
2023/05/01 21:38:50 DEBUG : fs cache: renaming cache item "/root/" to be canonical "/root"
2023-05-01 21:38:50 DEBUG : When Did You Last See Your Father (2007) [imdb-tt0829098] [tmdb-45791] -aAF.mp4: Need to transfer - File not found at Destination
2023-05-01 21:40:16 DEBUG : When Did You Last See Your Father (2007) [imdb-tt0829098] [tmdb-45791] -aAF.mp4: md5 = f2df294d0a2d9b487654327aa0a1d130 OK
2023-05-01 21:40:16 INFO  : When Did You Last See Your Father (2007) [imdb-tt0829098] [tmdb-45791] -aAF.mp4: Copied (new) to: testfile2.test
Transferred:        8.150 GiB / 8.150 GiB, 100%, 93.852 MiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:      1m25.7s
2023/05/01 21:40:16 INFO  : 
Transferred:        8.150 GiB / 8.150 GiB, 100%, 93.852 MiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:      1m25.7s

So ideally here we can make it so that without the caching it is as fast as possible to start the file transfer.
I have tested this process both with VFS cache mode set to off and full and it makes no difference.

4

I have exactly the same setup as you.

I have a glusterfs backend, and use rclone vfs cache on the load balancers on front of it for disk caching.

This is the settings I have used for 1 gbps load balancers with hdds

ExecStart=/usr/bin/rclone mount
--config=/home/xtreamcodes/.config/rclone/rclone.conf
--allow-other
--vfs-cache-max-age=8766h
--vfs-cache-max-size=14.1T
--vfs-cache-mode=full
--cache-dir=/mnt/cache
--vfs-cache-poll-interval=5m
--vfs-read-chunk-size=1M
--vfs-read-chunk-size-limit=8M
--buffer-size=1M
--no-modtime
--no-checksum
--dir-cache-time=8766h
--attr-timeout=1h
--timeout=10m
--umask=022
--low-level-retries=2
--retries=0
--log-level=INFO
--log-file=/opt/rclone.log
--async-read=true
--rc
--rc-addr=localhost:5572 \

And yes, if you don't do a rclone rc vfs/refresh the mount is unusable till the cache is built

Now I have upgraded to 10 gbps load balancers and SSDs I have increased the settings a little, but still not sure if optimal now I'm doing

--vfs-read-chunk-size-limit=512M but kept --vfs-read-chunk-size=10M for low latency and fast start of streaming of the file, so that it start fast and then it starts downloading more and saving to disk and --buffer-size=10M to avoid hammering my backend.

Seems to be performing better but I still don't know if that is even optimal. I have been tried to talk to @ncw to help me up setup rclone at scale for a long time but he is elusive :frowning:

But nice to know that more people are starting to use glusterfs + rclone !

5

I think the main issue for me would be dealing with cache invalidation. I would likely have to build my own system to refresh the cache on certain sub directories when changes are made (file upgrades, changing release groups, quality versions etc) which is doable, but I'd prefer to use a system that already exists as it will minimise work and will likely be a better implementation.

6

It would be nice, yes. I'd like a cross platform implementation though and unfortunately none of the 3 different notification types on Linux support a recursive watch last time I looked unlike macOS and Windows. There is more than one issue about this already!

I really need to see the log from the rclone mount to work out why it is slow. Or maybe even a strace to see what system calls it is doing.

I suspect it is because stat is slow on glusterfs and rclone doesn't parallelize them yet - that would be my guess.

Take a look at this issue and you'll see what has been discussed

7

Interesting, does a SMB/CIFS remote happen to support cache invalidation via polling instead?

8

Not yet no. Polling was one of the ideas to make a change mechanism for the local backend.

9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.