Rclone mount behavior about large amount of files

Help and Support
1

I have some questions about rclone mount. If I use the following command to mount:

/usr/local/bin/rclone mount test:/ /test --allow-other --vfs-cache-mode writes --allow-non-empty

When copying many large files to /test, will rclone clear the files before upload completion, causing upload missing?

In the documentation about --vfs-cache, I notice that if a file is open, it won't be cleared even if it exceeds the maximum time limit. However, due to operating system limitations and performance considerations, rclone probably doesn't keep all pending upload files open simultaneously(not sure about this). In this case, how does rclone handle files that exceed the time limit but aren't open or uploaded yet?

From observing actual upload behavior, these files seem to be stored in the .cache directory, but I'd like to have a clear understanding of this behavior (preferably from official documentation or discussions).

Thanks.

2

Hi,

Here is my best presumption. Someone please correct me if Chat got it wrong.

When using rclone—especially with commands like sync, copy, or move—and there's a time limit set (like using --max-duration or hitting a system-imposed timeout), here's how it generally handles files that exceed the time limit but aren't open or uploaded yet:

  1. Incomplete Files Are Skipped:
    If the time limit is reached before a file starts transferring, that file will simply be skipped. rclone doesn’t start a new file transfer once the time limit is up.

  2. Transfers in Progress Are Usually Allowed to Finish:
    If a file transfer has already started before the time limit is hit, rclone will usually allow it to complete rather than cutting it off mid-transfer (depending on the specific backend and version of rclone). This is to avoid data corruption or partial uploads.

  3. Files Not Yet Opened Are Ignored:
    Files that haven't been opened for reading or uploading when the timer expires are ignored. No new transfers begin once the time limit is exceeded.

  4. Logging Behavior:
    You can often see in verbose logs (-v or -vv) that files are skipped or that the transfer ends due to the duration cap.

3

welcome to the forum,

no. the mount command maintains the vfs cache, so rclone knows which files:

  • need to be uploaded
  • currently in the process of uploading
  • are locked/in-use by the OS
  • have been uploaded
  • can be purged

fwiw, can see all that for yourself by using a rclone debug log.

check out my summary of the two vfs caches

1 Like