Cache mode "writes" acts like "full" randomly

What is the problem you are having with rclone?

--vfs-cache-mode writes
I'm trying to play some video files and Rclone is trying to completely download it (in the cache folder) before letting it play.
I'm currently on Linux Mint 20.3, I've never had this issue on windows and the option worked normally (caching only files that were being written)
The issue seems to happen randomly, some files would stream normally, and some rclone would download completely before playing.
I installed rclone from the default software manager in linux mint.

Run the command 'rclone version' and share the full output of the command.

rclone v1.50.2

  • os/arch: linux/amd64
  • go version: go1.13.6

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount "Unlimited Drive:" /home/truck/unlimited --vfs-cache-mode writes --cache-dir "/home/truck/Rclone Cache" 

The rclone config contents with secrets removed.

It's encrypted

A log from the command with the -vv flag

2022/02/18 06:51:36 DEBUG : rclone: Version "v1.50.2" starting with parameters ["rclone" "mount" "Unlimited Drive:" "/home/truck/unlimited" "--vfs-cache-mode" "writes" "--cache-dir" "/home/truck/Rclone Cache" "-vv"]

hello and welcome to the forum,

this is not a rclone bug

--- update to latest stable rclone v1.57.0
the only way to get the latest rclone is
https://rclone.org/downloads/#script-download-and-install
--- change
--vfs-cache-mode writes
to
--vfs-cache-mode full

would be helpful to post the redacted rclone config

I do not wish to use full cache. I'm trying to cache only writes but it's caching everything (that too randomly). How is that not a bug?
And my config file is encrypted so its not really possible for me to post it.

well, so far there is zero evidence of a rclone bug.
-- redacted config file not posted
--- debug log, not posted
-- using a very old version of rclone, before v1.53.0.

I have upgraded to the latest version and its still caching fully instead of only caching "writes"

truck@linux-mint:~$ rclone version
rclone v1.57.0

  • os/version: linuxmint 20.3 (64 bit)
  • os/kernel: 5.4.0-100-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.17.2
  • go/linking: static
  • go/tags: none

Okay help me out here, the config file is encrypted, how do I decrypt it for my viewing in order to post it here.
And I did post the debug log as I was asked to while making the thread

that debug log was for the many years old version of rclone, so nothing of use there.

if you are going to call this a rclone bug, then need a full debug log documenting that.
so far, not providing any actionable info??

as per the docs, for --vfs-cache-mode writes
"read/write files are buffered to disk first"

Not really trying to "post" time, genuinely facing an issue and made a thread so I could perhaps get help on it.

if you are going to call this a rclone bug, then need a full debug log documenting that.

Not really calling it anything, i faced an issue and I thought the forums would be a good place to get help.

Fwiw, you could be a bit less toxic

truck@linux-mint:~$ rclone mount "Unlimited Drive:" /home/truck/unlimited --vfs-cache-mode writes --cache-dir "/home/truck/Rclone Cache" -vv
2022/02/18 07:28:04 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "mount" "Unlimited Drive:" "/home/truck/unlimited" "--vfs-cache-mode" "writes" "--cache-dir" "/home/truck/Rclone Cache" "-vv"]
2022/02/18 07:28:04 DEBUG : Creating backend with remote "Unlimited Drive:"

The file in question is a video file, read only, I'm not writing anything to it.

[Backup]
type = drive
client_id = redacted
client_secret = redacted
scope = drive
token = {"access_token":"redacted","expiry":"2022-02-18T07:56:08.14166618+05:30"}
team_drive = 
root_folder_id = redacted

[Unlimited Drive]
type = crypt
remote = Backup:Private
password = redacted
password2 = redacted

[bx1]
type = box
token = {"access_token":"redacted","token_type":"bearer","refresh_token":"redacted","expiry":"2021-09-13T07:36:02.2051576+05:30"}

[Box 1]
type = crypt
remote = bx1:Private
password = redacted
password2 = redacted
filename_encryption = standard
directory_name_encryption = true

[rd1]
type = drive
client_id = redacted
client_secret = redacted
scope = drive
token = {"access_token":"redacted"}
team_drive = 

[rd1e]
type = crypt
remote = rd1:Private
password = redacted
password2 = redacted

[rd2]
type = drive
client_id = redacted
client_secret = redacted
scope = drive
token = {"access_token":"redacted"}
team_drive = 

[rd2e]
type = crypt
remote = rd2:Private
password = redacted
password2 = redacted

[PUnion]
type = union
upstreams = rd1e: rd2e:

Config file with all sensitive information removed

Heres the full log

as you can see its saving the whole file even though i didnt set it to full.

It's hard to tell what you've done as your cache-dir is all jacked up probably from the different versions.

Your debug:

2022/02/18 08:42:01 DEBUG : Test folder/17.mp4: vfs cache: removed metadata from cache as cache file doesn't exist

But then:

2022/02/18 08:42:01 DEBUG : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item Test folder/17.mp4 not removed, freed 0 bytes

The read you are asking about comes from that local file that's already there so this line tells me it wasn't put there natively through rclone while the cache was there:

2022/02/18 08:42:01 INFO  : vfs cache: cleaned: objects 9 (was 9) in use 0, to upload 0, uploading 0, total size 0 (was 0)

Since there is nothing upload or your whole metadir is messed up.

  1. Stop rclone
  2. Delete/Move the cache-dir (delete if you have no data that's out value)
  3. Start rclone
  4. Repeat your process
  5. Share a new complete debug log

I have repeated the process with a new cache directory.


As you can see, even though the file is being opened for read-only, rclone is saving it in disk.

Test 2 with different files and a clean cache directory

This time it did not save the files.

sorry if that came across but imho, i think i was most helpful.

  1. i wrote "hello and welcome to the forum,"

  2. let you know this is not a rclone bug.

  3. let you know that you were using a very old version of rclone,
    prior to v1.53.0 which had a major update to the vfs file cache.

  4. tried twice to get you to post a full debug log, as that would show what the problem is.

  5. the forum moderator overloads will take down topics missing basic info such as config file and debug log, was trying to prevent that.

  6. made no personal comments and did not use any adjectives about you.

The issue whatever you app your using is opening it read/write:

2022/02/18 19:11:06 DEBUG : Test folder/18.mp4: Open: flags=OpenReadWrite

and not read only so it's being saved.

A normal read open like this:

2022/02/18 09:41:21 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: Attr:
2022/02/18 09:41:21 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: >Attr: a=valid=1s ino=0 size=404291584 mode=-rw-rw-r--, err=<nil>
2022/02/18 09:41:22 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: Open: flags=OpenReadOnly
2022/02/18 09:41:22 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: Open: flags=O_RDONLY

Would not write the file to disk so it's doing exactly what you've asked it to.

In checking the second log, you can see it has no writes, but only read:

grep -i write mrQbx33m.txt
truck@linux-mint:~$ rclone mount "Unlimited Drive:" /home/truck/unlimited --vfs-cache-mode writes --cache-dir "/home/truck/Rclone Cache 3" -vv
2022/02/18 19:21:20 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "mount" "Unlimited Drive:" "/home/truck/unlimited" "--vfs-cache-mode" "writes" "--cache-dir" "/home/truck/Rclone Cache 3" "-vv"]
2022/02/18 19:23:53 DEBUG : rclone: Version "v1.57.0" finishing with parameters ["rclone" "mount" "Unlimited Drive:" "/home/truck/unlimited" "--vfs-cache-mode" "writes" "--cache-dir" "/home/truck/Rclone Cache 3" "-vv"]

and

grep -i ONLY mrQbx33m.txt
2022/02/18 19:21:56 DEBUG : Anime/The Tatami Galaxy S01 1080p BDRip 10 bits DD x265-EMBER/S01E01-Tennis Club [91826C00].mkv: Open: flags=OpenReadOnly+OpenNonblock
2022/02/18 19:21:56 DEBUG : Anime/The Tatami Galaxy S01 1080p BDRip 10 bits DD x265-EMBER/S01E01-Tennis Club [91826C00].mkv: Open: flags=O_RDONLY|0x800
2022/02/18 19:21:57 DEBUG : Anime/The Tatami Galaxy S01 1080p BDRip 10 bits DD x265-EMBER/S01E01-Tennis Club [91826C00].mkv: Open: flags=OpenReadOnly+OpenNonblock
2022/02/18 19:21:57 DEBUG : Anime/The Tatami Galaxy S01 1080p BDRip 10 bits DD x265-EMBER/S01E01-Tennis Club [91826C00].mkv: Open: flags=O_RDONLY|0x800
2022/02/18 19:21:59 DEBUG : Anime/The Tatami Galaxy S01 1080p BDRip 10 bits DD x265-EMBER/S01E01-Tennis Club [91826C00].mkv: Open: flags=OpenReadOnly+OpenNonblock
2022/02/18 19:21:59 DEBUG : Anime/The Tatami Galaxy S01 1080p BDRip 10 bits DD x265-EMBER/S01E01-Tennis Club [91826C00].mkv: Open: flags=O_RDONLY|0x800
2022/02/18 19:22:00 DEBUG : Anime/The Tatami Galaxy S01 1080p BDRip 10 bits DD x265-EMBER/S01E01-Tennis Club [91826C00].mkv: Open: flags=OpenReadOnly+OpenNonblock
2022/02/18 19:22:00 DEBUG : Anime/The Tatami Galaxy S01 1080p BDRip 10 bits DD x265-EMBER/S01E01-Tennis Club [91826C00].mkv: Open: flags=O_RDONLY|0x800
2022/02/18 19:22:01 DEBUG : Anime/The Tatami Galaxy S01 1080p BDRip 10 bits DD x265-EMBER/S01E01-Tennis Club [91826C00].mkv: Open: flags=OpenReadOnly+OpenNonblock
2022/02/18 19:22:01 DEBUG : Anime/The Tatami Galaxy S01 1080p BDRip 10 bits DD x265-EMBER/S01E01-Tennis Club [91826C00].mkv: Open: flags=O_RDONLY|0x800
2022/02/18 19:22:03 DEBUG : Anime/The Tatami Galaxy S01 1080p BDRip 10 bits DD x265-EMBER/S01E01-Tennis Club [91826C00].mkv: Open: flags=OpenReadOnly+OpenNonblock
2022/02/18 19:22:03 DEBUG : Anime/The Tatami Galaxy S01 1080p BDRip 10 bits DD x265-EMBER/S01E01-Tennis Club [91826C00].mkv: Open: flags=O_RDONLY|0x800
2022/02/18 19:22:27 DEBUG : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item Anime/The Tatami Galaxy S01 1080p BDRip 10 bits DD x265-EMBER/S01E01-Tennis Club [91826C00].mkv not removed, freed 0 bytes
2022/02/18 19:22:44 DEBUG : Anime/The Tatami Galaxy S01 1080p BDRip 10 bits DD x265-EMBER/S01E02-Movie Club [77805C71].mkv: Open: flags=OpenReadOnly+OpenNonblock
2022/02/18 19:22:44 DEBUG : Anime/The Tatami Galaxy S01 1080p BDRip 10 bits DD x265-EMBER/S01E02-Movie Club [77805C71].mkv: Open: flags=O_RDONLY|0x800
2022/02/18 19:22:46 DEBUG : Anime/The Tatami Galaxy S01 1080p BDRip 10 bits DD x265-EMBER/S01E02-Movie Club [77805C71].mkv: Open: flags=OpenReadOnly+OpenNonblock
2022/02/18 19:22:46 DEBUG : Anime/The Tatami Galaxy S01 1080p BDRip 10 bits DD x265-EMBER/S01E02-Movie Club [77805C71].mkv: Open: flags=O_RDONLY|0x800
2022/02/18 19:22:48 DEBUG : Anime/The Tatami Galaxy S01 1080p BDRip 10 bits DD x265-EMBER/S01E02-Movie Club [77805C71].mkv: Open: flags=OpenReadOnly+OpenNonblock
2022/02/18 19:22:48 DEBUG : Anime/The Tatami Galaxy S01 1080p BDRip 10 bits DD x265-EMBER/S01E02-Movie Club [77805C71].mkv: Open: flags=O_RDONLY|0x800
2022/02/18 19:22:49 DEBUG : Anime/The Tatami Galaxy S01 1080p BDRip 10 bits DD x265-EMBER/S01E02-Movie Club [77805C71].mkv: Open: flags=OpenReadOnly+OpenNonblock
2022/02/18 19:22:49 DEBUG : Anime/The Tatami Galaxy S01 1080p BDRip 10 bits DD x265-EMBER/S01E02-Movie Club [77805C71].mkv: Open: flags=O_RDONLY|0x800
2022/02/18 19:22:50 DEBUG : Anime/The Tatami Galaxy S01 1080p BDRip 10 bits DD x265-EMBER/S01E02-Movie Club [77805C71].mkv: Open: flags=OpenReadOnly+OpenNonblock
2022/02/18 19:22:50 DEBUG : Anime/The Tatami Galaxy S01 1080p BDRip 10 bits DD x265-EMBER/S01E02-Movie Club [77805C71].mkv: Open: flags=O_RDONLY|0x800
2022/02/18 19:22:53 DEBUG : Anime/The Tatami Galaxy S01 1080p BDRip 10 bits DD x265-EMBER/S01E02-Movie Club [77805C71].mkv: Open: flags=OpenReadOnly+OpenNonblock
2022/02/18 19:22:53 DEBUG : Anime/The Tatami Galaxy S01 1080p BDRip 10 bits DD x265-EMBER/S01E02-Movie Club [77805C71].mkv: Open: flags=O_RDONLY|0x800
2022/02/18 19:23:27 DEBUG : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item Anime/The Tatami Galaxy S01 1080p BDRip 10 bits DD x265-EMBER/S01E01-Tennis Club [91826C00].mkv not removed, freed 0 bytes
2022/02/18 19:23:27 DEBUG : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item Anime/The Tatami Galaxy S01 1080p BDRip 10 bits DD x265-EMBER/S01E02-Movie Club [77805C71].mkv not removed, freed 0 bytes

So it's only read only and that means no items saved to the cache area.

Not an issue with rclone then. I was playing the video with VLC, I guess vlc is opening it in read/write . Any idea how I can prevent that?

You can't.

It's however the app is written and how they decide to open it. The change would be in the application.