I'm checking if my understanding of --vfs-cache-mode writes (or full) is correct in Ubuntu 22.04. What I currently see is as follows:
I run below command
rclone -v mount koofr-remote: /home/somebody/KOOFR_DIR --vfs-cache-mode writes
I copy some files to the directory /home/somebody/KOOFR_DIR
Transfer is super-fast as for SSD Drive it runs on. Ubuntu Nautilus reports all files copied in no-time.
Cache directory ~/.cache/rclone/ grows by the size of copied files.
In logs I see that rclone mount begins the slow process of uploading to remote (Koofr storage) and after some time gradual cleaning (removing) files from cache dir
Eventually cache is empty and files are copied to destination.
Is above behaviour as expected?
My conern is that above behaviour is misleading (confirmation from File Explorer that upload completed) and there is no indication what is the current transfer speed and ETA of real completion when files will be finally on the remote storage.
What I'm currently doing is just checking in logs below lines:
2023/05/27 16:31:34 INFO : vfs cache: cleaned: objects 20 (was 20) in use 19, to upload 15, uploading 4, total size 1.929Gi (was 1.929Gi)
or below when all is finished and cleared
2023/05/27 16:29:19 INFO : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
It is very strange and not convenient
Run the command 'rclone version' and share the full output of the command.
rclone v1.62.2
os/version: ubuntu 22.04 (64 bit)
os/kernel: 5.15.0-72-generic (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.20.2
go/linking: static
go/tags: none
Which cloud storage system are you using? (eg Google Drive)
Koofr
The command you were trying to run (eg rclone copy /tmp remote:tmp)
rclone -v mount koofr-remote: /home/somebody/KOOFR_DIR --vfs-cache-mode writes
This is exactly how it works and the whole purpose of write cache.
It does not confirm anything about upload - it confirms that files were copied to the cache. If you turn cache off --vfs-cache-mode off your file explorer will wait until actual upload finishes if this is what you want.
{
// Status of the disk cache - only present if --vfs-cache-mode > off
"diskCache": {
"bytesUsed": 0,
"erroredFiles": 0,
"files": 0,
"hashType": 1,
"outOfSpace": false,
"path": "/home/user/.cache/rclone/vfs/local/mnt/a",
"pathMeta": "/home/user/.cache/rclone/vfsMeta/local/mnt/a",
"uploadsInProgress": 0,
"uploadsQueued": 0
},
"fs": "/mnt/a",
"inUse": 1,
// Status of the in memory metadata cache
"metadataCache": {
"dirs": 1,
"files": 0
},
// Options as returned by options/get
"opt": {
"CacheMaxAge": 3600000000000,
// ...
"WriteWait": 1000000000
}
}
This is what is available today. How you use it is up to you.
Now I think that in perfect world you would love something like this:
This is macOS Finder showing progress of uploading files to iCloud.
Could something like this be done with rclone and vfs-cache on your system? Yes for sure. But it would require your File Manager get closely integrated with rclone (could use e.g. vfs-stats). Somebody has to do it though:) For sure it is outside of rclone project scope.
Hmm, I see that rc mode involves running some http server to allow such queries... I will check it, thanks.
Pity there is just no flag like --show-cached-file-progress that will report how many % of cached file is already sent (rclone must know it and it's just one print ) I'm sure not only me has the same issue, how to know if cache will upload in 1 or 5h
By one chance, do you know what is exactly the purpose of "writes" or "full" cache mode? I assume it's to make it fault-tolerant (ie in "off" mode any network or cloud error may derail and stop whole process of sending eg 1000 files) and with cached "writes"/"full" we will have still attemp #2, #3 and so on.
I agree that it would be nice to have --show-cached-file-progres among many other wishes. Maybe good opportunity to get into programming and implement what is your top 1?
For write operations they do the same. BTW write cache is persistent - you can turn your computer off and then on after one week and it will continue what is not finished.
Difference is that full will also do full read caching. E.g. in your case files wont be removed from cache after upload. You read them next time you will read from your fast SSD.
I've already studied that documentation, there are still some basic gaps for me, I presume those could be covered only by checking the source code...
I was wondering if my understanding of
--vfs-cache-max-age duration
in "writes" mode is correct (default 1h).
Is this the time how long the cache will hold that file AFTER it was succesfully uploaded? Or maybe the minimal amount of time it will be stored in cache in case the upload was succesfull just let's say after 10 minutes and we still have 50 minutes to keep the file in cache "just in case"?
If this is basic question, sorry. I just can't find it in that rclone mount doc
Oh, interesting. From original version of the file you posted:
# This limits the age in the cache if the size is reached and it removes the oldest files first
--vfs-cache-max-age 9999h
Maybe my understanding of that option is completely wrong
Just as side note, I've experimented with --vfs-cache-max-size as well.
Some time ago I had the idea to set it as for example 200MB and maybe this will "throttle" the upload and make "writes" mode more as normal transfer when only 200MB chunks are sent at a time (and explorer properly reports progress).
But no, maybe it's a feature or a bug, but my rclone ignored that setting and "nicely" filled the cache will all files (about 50GB) and completely ignored that 200MB
) I'm sure not only me has the same issue, how to know if cache will upload in 1 or 5h 