Rclone v1.51
OS: Debian 10 x86 64bit
remote: Google Drive
So this is basically about using cache on machines with very limited free space i.e. 2-5GB. I have almost no experience with cache / different settings though have read through the Docs, so thanks for your patience!
A) Pushing to encrypted Gdrive when triggered by inotify. Talking about couple GBs per few hours and this will be the only method that writes to the specific remote.
This could be made with an rclone copy local remote: or by copying to the cached mounted remote mainly due to offline upload which seems the most resilient uploading method?
If the second choice is the better what would be the configuration options I should use? Does specifying just --cache-tmp-upload-path=/somewhere --cache-chunk-total-size=2G --cache-info-age=24h --dir-cache-time=12h
be enough with the rest as default? Is cache-tmp-upload-path taken into account in --cache-chunk-total-size or does that take extra space?
B) In this one it will only be reading from encrypted remote. Same configuration as the above minus the --cache-tmp-upload-path, half the values for info-age & 1-2 hour dir-cache? Obviously the size of the cache is too small so what other options would I need to make the files from the remote be available before they are fully downloaded (small chunks I guess etc).
Extra - Having a big remote with say 1 hour dir-cache / info-age (~1k dirs / 20k files) must be causing quite the load on APIs limits right?
Rclone remotes have been configured as:
[g1]
type = drive
[g1_cache]
type = cache
remote = g1:top_dir
[g1_crypt]
type = crypt
remote = g1_cache:
filename_encryption = standard
directory_name_encryption = true
I went straight ahead to cache, due to finding in the docs that the most resilient upload method is by using offline upload which is part of cache.
Would it be much to ask for some basic configuration with vfs for the above? From a quick reading I got:
( Keep in mind that RAM is also limited ~256M usable by rclone )
A) uploading (though considering my other thread it might be more error prone but might be better off with just a rclone copy and rclone copy --drive-server-side-across-configs?)
--vfs-read-chunk-* is not needed since we are only uploading
--dir-cache-time 24h (does this affect only dirs being removed or also newly added dirs?)
--buffer-size 16M
Is it required to have --vfs-cache-mode writes to be able to retry failed uploads? If needed then I guess +
--vfs-cache-max-size 2G
B) reading - here it's needed to be able to start accessing the file before it's completely downloaded from the remote
--vfs-read-chunk-size 32M (does this translate into memory used?)
--vfs-read-chunk-size-limit 0
--dir-cache-time 1h
--buffer-size 16M
--vfs-cache-mode minimal
--vfs-cache-max-size 2G
Will try to "play" with the above see if I understood the settings correctly
@ncw I tried some stuff for B with or without the vfs-cache it works fine regarding accessing the file before it's fully downloaded.
But with --vfs-cache-mode minimal I was expecting that when I read or write a file that it will get cached, so some entry should appear under .cache/rclone/vfs/. Instead there is nothing below that dir, the free space doesn't get reduced and when downloading the same file from the mount (/mnt/) I can see in iftop that it regrabs it from Gdrive.
I tried unmounting, rm -r .cache/rclone, remounting - the dirs didn't get recreated.
In the logs it's always visible : 2020/02/19 22:37:28 DEBUG : vfs cache root is "/root/.cache/rclone/vfs/remote"
if I do mkdir -p /root/.cache/rclone/vfs/; touch /root/.cache/rclone/vfs/remote I get "Failed to create vfs cache - disabling: failed to create cache remote: is a file not a directory" which is expected
if I do mkdir /root/.cache/rclone/vfs/remote I get
2020/02/19 22:43:51 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2020/02/19 22:43:52 DEBUG : Encrypted drive 'remote:': Mounting on "/mnt"
2020/02/19 22:42:24 DEBUG : vfs cache root is "/root/.cache/rclone/vfs/remote"
2020/02/19 22:42:24 DEBUG : Adding path "vfs/forget" to remote control registry
2020/02/19 22:42:24 DEBUG : Adding path "vfs/refresh" to remote control registry
2020/02/19 22:42:24 DEBUG : Adding path "vfs/poll-interval" to remote control registry
020/02/19 22:42:24 DEBUG : : Root:
2020/02/19 22:42:24 DEBUG : : Removed empty directory
2020/02/19 22:42:24 INFO : Cleaned the cache: objects 0 (was 0), total size 0 (was 0)
2020/02/19 22:42:24 DEBUG : : >Root: node=/, err=<nil>
2020/02/19 22:43:17 DEBUG : cache cleaner exiting
which just removes the remote dir and if download / upload something nothing changes under vfs dir
Ah I completely misunderstood that, it seems quite not clear to me, since the options reference cache a lot, thank you for clarifying it to me.
--vfs-cache-max-size int Max total size of objects in the cache
The cache has 4 different modes selected by --vfs-cache-mode
Basically here is where it mentions it has to be BOTH read + write, though without you pointing it out I missed the meaning behind the capital and as it means both actions This is very similar to “off” except that files opened for read AND write will be buffered to disks
Anyway for my "minimal" usage the 0.5-3sec it takes to start transferring the files is fine (seems to be affected by whether the file structure of the dir has been cached?)
Do you know by any chance the answer to these two questions:
--dir-cache-time a directory being removed from outside the mount won't be reflected till cache-time expires, is it the same for directories being added?
--vfs-read-chunk-size 32M (does this translate into memory used and how does it work with --buffer-size)
--dir-cache-time shouldn't effect directory additions since mine seem to show up properly as per the configured poll interval.
--vfs-read-chunk-size 32M defines the chunk-size for the initial chunk download. After the first request, it will keep doubling the size up to the value of --vfs-read-chunk-size-limit and add the data to the buffer. Effectively, only the value of --buffer-size will influence the memory usage of rclone.