How to use the RCLONE Cache - or NOT?

I mounted a SFTP remote like this:

rclone mount SS_WEBDAV_ALL: /Volumes/UNENCRYPTED/SS_WEBDAV_ALL --vfs-cache-mode full --vfs-write-back 20s --vfs-cache-max-age 24h --vfs-cache-max-size 50G --vfs-read-chunk-size 128M --vfs-read-chunk-size-limit off --poll-interval 0 --buffer-size 128M --cache-dir /Volumes/S990PRO/CACHES/RCLONE/SS_WEBDAV_ALL --dir-cache-time 5m --no-modtime --timeout 2m --transfers 4 --checkers 8 --log-file /Users/tja/.rclone_SS_WEBDAV_ALL.log --log-level INFO --daemon --no-console

SS_WEBDAV_ALL: is a NextCloud WebDAV server:

type = webdav
vendor = nextcloud

Then, I proceeded to upload about 120 GB to this remote.

And as it seems, FIRST rclone copies all of this to the CACHE dir:

tja@studio:/Volumes/S990PRO/CACHES/RCLONE$ du -hs SS_WEBDAV_ALL
51G SS_WEBDAV_ALL

...

tja@studio:/Volumes/S990PRO/CACHES/RCLONE$ du -hs SS_WEBDAV_ALL
94G SS_WEBDAV_ALL

It is still growing.

That is, of course, totally unneeded and I don't understand why rclone is doing this.
I mean, the original data is there - why copy it?!?

I understand caching for changed data or incomplete data that get's downloaded, but for an upload?!?

Can I change this behavior?

Have a look here:

In general you can have your cache off, writes only or for all operations.

1 Like

I interpret this subtle hint as directive to disable caching for such usage?

I thought that the cache would only be used for "useful" things and not for "everything" ... I did not come to the idea that I should disable it.

But then, are you referring to disabling it, or to something different, please?

To elaborate on this:

I never saw a cache that first copies unchanged data for some operation.

I would have suspected that it will mostly be used for downloading files - so that they are complete before writing them back to the target location or offering them in the mounted file system. So that they can then ne stably accessed.

If disabling is the solution here, I would need to partition any work into "pure upload" and "regular access" to that the cache will only be used for the latter.

But this is really unexpected behaviour, to be honest.

With rclone you can NOT have cache only for reads.

If you do write to your remote I would strongly recommend to use VFS full caching mode. It will be not only snappier behaviour but also greater compatibility with your other programs. Without write cache your mount won't behave like normal filesystem and some programs simply will refuse to work (in case you write to files in your mount).

BTW. You are the first person asking for cache only for read operations. I still can not fully understand why:) It is not really logical. As I do not see what benefits would be.

1 Like

Yes, I was reading the different options now in detail.
It seems, the best option would be "writes" ... that seems to be what I thought to be the "default" behaviour of a cache.

Thanks a bunch!

But I did not ask for a "read cache" - I am just accessing a remote that is basically empty and I am in the process of uploading stuff. There would no caching be needed, which can be done with the 'writes" option, if I understood that correctly.

Thanks again! :slight_smile:

1 Like

For initial upload I would use rclone copy/move/sync and not mount. Not only faster than mount but also gives you some visibility what is going on and stats.

1 Like

Oh, great idea!

But now over 120 GB are uploading .... can I stop that somehow?
Just umount ... and remove the cache?

Other tricks? :smiley:

yeah. I would do it exactly like this. Whatever is already uploaded wont be uploaded again - does not matter if you use rclone sync

1 Like

Going to try this!

In my log, I see those GOAWAY message, because of which I reduced the number of transfers back to 4, but that does not fix the problem.

I read that disabling HTTP2 may be the solution instead.
Going to try that too!

I tried, but it seems I can only use this for the next upload:

$ umount /Volumes/UNENCRYPTED/SS_WEBDAV_ALL
umount(/Volumes/UNENCRYPTED/SS_WEBDAV_ALL): Resource busy -- try 'diskutil unmount'

$ diskutil unmount /Volumes/UNENCRYPTED/SS_WEBDAV_ALL
Unmount failed for /Volumes/UNENCRYPTED/SS_WEBDAV_ALL

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.