Uploads through cache with "writes = true" not persisted

Suspected Bug
1

Hello, I'm having an issue where uploaded files are immediately purged from the rclone cache. Watching ~/.cache/rclone, I see the file being written to the cache as it's uploaded, but it's immediately purged once the upload is complete.

With writes=true in the cache config, I was expecting the written file to hang around in the cache until it needs to be evicted to reclaim space. I even tried deleting the entire cache, but no difference (file.bin is only 1GB with a 50GB cache size in this case).

Config:
[cache]
type = cache
remote = drive:
chunk_total_size = 50G
chunk_size = 32M
info_age = 1M
workers = 8
writes = true

Log:
2020/01/10 02:13:11 DEBUG : &{file.bin (w)}: >Write: written=131072, err=
2020/01/10 02:13:11 DEBUG : &{file.bin (w)}: Write: len=131072, offset=1448214528
2020/01/10 02:13:11 DEBUG : &{file.bin (w)}: >Write: written=131072, err=
2020/01/10 02:13:11 DEBUG : &{file.bin (w)}: Write: len=51760, offset=1448345600
2020/01/10 02:13:11 DEBUG : &{file.bin (w)}: >Write: written=51760, err=
2020/01/10 02:13:11 DEBUG : &{file.bin (w)}: Flush:
2020/01/10 02:13:15 DEBUG : file.bin: put: uploaded to remote fs and saved in cache
2020/01/10 02:13:16 DEBUG : file.bin: put: added to cache
2020/01/10 02:13:16 DEBUG : : cache: expired Test
2020/01/10 02:13:16 INFO : : put: cache expired
2020/01/10 02:13:16 DEBUG : file.bin: Size and modification time the same (differ by -187.664µs, within tolerance 1ms)
2020/01/10 02:13:16 DEBUG : &{file.bin (w)}: >Flush: err=
2020/01/10 02:13:16 DEBUG : &{file.bin (w)}: Release:
2020/01/10 02:13:16 DEBUG : file.bin: WriteFileHandle.Release nothing to do
2020/01/10 02:13:16 DEBUG : &{file.bin (w)}: >Release: err=
2020/01/10 02:13:36 DEBUG : Cache remote cache:Test: starting cleanup
2020/01/10 02:13:36 INFO : cache-cleanup: chunks 38, est. size: 848.675M
2020/01/10 02:13:36 DEBUG : Google drive root 'Encrypted/614fbdi6pldkh0frctmgl3ces8': Checking for changes on remote
2020/01/10 02:13:36 INFO : file.bin: received cache expiry notification
2020/01/10 02:13:36 DEBUG : : changeNotify: relativePath="file.bin", type=3
2020/01/10 02:13:36 DEBUG : : invalidating directory cache
2020/01/10 02:13:36 DEBUG : : >changeNotify:
2020/01/10 02:13:36 DEBUG : : cache: expired Test
2020/01/10 02:13:36 DEBUG : file.bin: notify: expired ''

Using rclone v1.50.2-101-gb9fb313f-beta on Linux.

2

Once something is uploaded, it's removed from the cache tmp area as that's how it works.

Chunks are stored based on the time set in the config assuming you are using the cache-tmp feature.

3

Then I must be misunderstanding --cache-writes. If it doesn't keep written objects in the cache, then what does it do?

4

It makes it available while uploading:

https://rclone.org/cache/#write-support

5

I'm sorry, I still don't understand.

That link says it makes "it available from the cache store immediately once the upload is finished."

This is exactly the behavior I want, but not the behavior I'm seeing. How can it be available after the upload is finished if it's deleted after the upload is finished?

6

You would read it from the remote, hence available on the remote.

7

Err, "available from the cache store" means available on the remote?

8

If you aren't using offline uploading here:

https://rclone.org/cache/#write-support

and just write to the mount, it uploads it and is available once the upload is finished.

You can trust the data as that's what you are seeing too.

9

I understand what you're saying, and my confusion is around the docs explicitly stating the upload will be "available from the cache store". After an upload, you're saying it's available from the remote, not from the cache store. Right?

Regardless, it sounds like this isn't the flag I'm looking for. Is there any way to have rclone persist written files in the cache like it does with files that are opened for reading?

10

The cache store is the remote.

It isn't really possible unfortunately with the cache remote.

11

If I'm interpreting that log correctly, --cache-writes isn't purging your test file when the upload is completed, but rather thirty seconds later when the recent changes are polled from the google: remote.

Can you double check to see if the chunks are actually removed from the filesystem? You can set --cache-chunk-path and see if the chunks for your test file are still there. Perhaps this is just clearing out the metadata cache?

If rclone is actually clearing out the chunks, --poll-interval 0 might work around it for now (providing you aren't expecting legitimate changes from other locations).

12

--cache-writes has nothing to do with the cache backend.

Setting polling 0 would break anything uploaded so you'd never see it.

13

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.