Is vfs_cache always file based?

Help and Support
1

What is the problem you are having with rclone?

High latency on some kinds of file access.

This shell command accesses only the file header, not the entire file (AFAIK)

mkvpropedit --edit track:s2 --set name=italiano Madam.Secretary.S01E17.Face.the.Nation.1080p.WEB-DL.DD5.1.H.264-BS.mkv

Right after giving the command above, in another shell I check the file /home/admin/.cache/rclone/vfs/gdrive/TV/Madam.Secretary.S01/Madam.Secretary.S01E17.Face.the.Nation.1080p.WEB-DL.DD5.1.H.264-BS.mkv and I see that, while the apparent size is 1860740091 right away, its disk usage grows from zero to 1.7 GiB in about 10 seconds (while the command is already finished). Sometimes, however, the command takes even more than 10 seconds to complete.

The measured bandwidth between my host and the Google Drive server seems fairly constant at about 240 Mbps, therefore it seems that perhaps the vfs cache is the culprit here.

I read somewhere that the vfs cache was chunk based but now is file based.

Run the command 'rclone version' and share the full output of the command.

rclone v1.68.2

  • os/version: ubuntu 22.04 (64 bit)
  • os/kernel: 5.15.0-84-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.23.3
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

This is the relevant line from /etc/fstab:

gdrive: /home/admin/downloads/gdrive rclone noauto,rw,nosuid,nodev,uid=1000,gid=1002,allow_other,syslog,config=/home/admin/.config/rclone/rclone.conf,vfs_cache_mode=writes,cache_dir=/home/admin/.cache/rclone,vfs-cache-max-size=4G,vfs-cache-min-free-space=1G,vfs-cache-max-age=60s 0 0

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[gdrive]
type = drive
client_id = XXX
client_secret = XXX
scope = drive
token = XXX
team_drive =
acknowledge_abuse = true
2

the rclone mount is file based. the rclone cache, in certain situations, can be chunk based.

if you edit a single byte of a file in the cloud, rclone has to download the entire file into the cache, change that single byte and then upload the entire file.

3

Does this mean that Google Drive API does not have seek and write operations?

Will try mode full, even if it seems to make disk usage grow more often.

4

in full mode, depending on the file system, the files are sparse and enables rclone to use chunk reading.
often, using file managers and command line, the fie size are is displayed as the full size, not the real/apparent size
tools like ncdu can show the apparent size.

rclone mount does not support random access writes of cloud files. rclone can only upload the entire file

chunker remote.

5

When accessing Google Drive through other means (like https://drive.google.com/), will I see files named BIG_FILE_NAME.rclone_chunk.ddd instead of the regular file BIG_FILE_NAME?

6

that is correct.

7

I have got Premium 2 TB plan for Google Drive.
Could it be that Google does some API calls rate limiting?

8

sure, but as you are a paid user, not sure what rate limits are.

fwiw, a debug log will show you exactly what rclone is doing.

9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.