I had a look in the /mnt/studio folder after the mount and all the folders small, which is what I would expect before anything is read. I added a file to /mnt/gmedia-cache and it went into /mnt/studio/rclone_tmp_upload , all good so far and as I suspected it should be there for 15 mins before getting uploaded. I came back later to have a look and the file was in my Google Drive and the /mnt/studio/rclone_tmp_upload folder was empty, however the /mnt/studio/rclone_chunk_cache was also empty, and to access the file it started to download it from the internet, this is not what I was expecting.
Are my expectations correct? I want files I’ve recently copied over to be at the top of the chunk cache as these are likely to be the ones I need access to in the near future. Have I missed a setting to make this work?
If you are using the cace-tmp-upload, you can turn off cache-writes.
You want the temporary area to be something on your local physical disks. It’s stored temporarily on the local volumes and once the age expires, it uploads.
Yes that’s what I saw, but it didn’t put the uploaded file in the chunk cache, which is what I was expecting with --cache-writes.
So after 15 mins it’s uploaded the file and if I want to access it again at 30 mins it has to download it before it goes back in the chunk cache, seems inefficient considering it was there a min ago.
The chunk cache is used for download files from your GD.
It doesn’t have any direct relation to the cache upload path.
My cache upload tmp is here:
[felix@gemini data]$ cd rclone_upload/
[felix@gemini rclone_upload]$ ls
smu5ej34ujbdoip1cm3mlk92q4 tnvepu36qiohcun8v84ddhsam0
[felix@gemini rclone_upload]$ pwd
/data/rclone_upload
[felix@gemini rclone_upload]$ ls -alR
.:
total 146
drwxrwxrwx 4 felix felix 4 Apr 4 10:36 .
drwxrwxrwx 23 felix felix 25 Apr 4 15:41 ..
drwxr-xr-x 185 felix felix 185 Apr 9 23:44 smu5ej34ujbdoip1cm3mlk92q4
drwxr-xr-x 117 felix felix 117 Apr 10 04:03 tnvepu36qiohcun8v84ddhsam0
./smu5ej34ujbdoip1cm3mlk92q4:
total 4541
drwxr-xr-x 185 felix felix 185 Apr 9 23:44 .
drwxrwxrwx 4 felix felix 4 Apr 4 10:36 ..
drwxr-xr-x 2 felix felix 2 Apr 9 17:58 014vftu6p0l7rhsputnfgc03c0
drwxr-xr-x 2 felix felix 2 Apr 3 00:26 088t1lndejatplu3lqrcngkml75nv1eplrvp0re01r48hnvae75g
You can see in your logs if it’s working as expected as you’ll definitely see files there and uploads happen.
Thanks, yes and I’ve just found some other posts too.
It was my misunderstanding of how it works, it was all working correctly I just wasn’t expecting it to have to download something that had just been uploaded.
Seems that --cache-writes and --cache-tmp-upload are mutually exclusive and work differently.
Maybe I can put a feature request in so that there is an option for --cache-tmp-upload to also get copied to the chunk cache on upload, that way when something gets put in the mount point it can be backed up offsite quickly and also available stored locally to work on, that would be more like how traditional tiered storage worked.
Once it’s copied to the cache_tmp area, it’s available once the local copy is complete. It handles moving and adjusting it all on the fly. It would only get copied (currently) to the chunk area if someone tried to read the file so copying before it’s accessed would be just overhead.
Overhead that might be useful in some circumstances which is why having it as an additional option would be handy for some people.
E.g. I’m editing my GoPro footage and think I’m finished so I store the 40GB of raw footage (extreme example) and project files on my rclone mount point, it get’s uploaded to Google Drive (or other provider), the next day I realise I forgot to splice in some footage so I need to open the project up again and add in another file from an extra sd card I missed off and re-render the output. I then have to wait for 40GB of original raw footage to download which might be an hour for me. If I had an option to save uploads to the chunk cache it would be local and ready to go straight away.
Can you see how this would be useful for me? Most of the data I access is data I’ve recently created.
Another example is using rclone mount with cache to surface a Google Team drive via an SMB share, users who add data to that would probably then be sending links for someone else to access fairly quickly, they don’t want to clog the internet connection with uploading and downloading the same file twice.
I realise I can adjust this --cache-tmp-wait-time to 24 hours or a week, but then it’s not backed up or accessible via my drive app on my phone or the web.
Hey Matt - I am looking to do something similar to this… IE have data being created, this being uploaded to the cloud and also the local cache simultaneously. then following cache expiration rules - time, size etc…
I've also been really confused on --cache-writes
From observing how files move it seems like writes do get cached in the chunk cache, but then as soon as they have been confirmed transferred they get removed. That seems like a very odd thing for a write-cache to do..
I mean, I can see how this may still be useful to make uploads more resistant to errors requiring retries, but it would be far more useful if it just didn't remove the written files until the cache hit it's limit. Just like the OP in this thread I often end up re-accessing recently written data.
So far I haven't been able to get either the VFS or the cache backend to do write-caching. They both seem to throw the writes out of cache once they have transferred. Either I'm missing something very important, or this seems like a big oversight and missed opportunity.
I made an issue on this to try to get to the bottom of it and change how this works.
