What is the problem you are having with rclone?
I'm mounting my directory for the use with JRiver Media Center (similar to Plex). I noticed today that my cache drive is totally full.
rclone mount --rc --rc-addr=localhost:5571 --vfs-cache-max-size 512G --vfs-cache-mode full --cache-dir "D:\Rclone Cache" --buffer-size 128M --vfs-cache-max-age 1000000h --vfs-read-chunk-size 8M "DFS - Music:/" Y: --network-mode
What is your rclone version (output from
rclone v1.56.0 (just updated from 1.55.1)
- os/version: Microsoft Windows 10 Pro 2009 (64 bit)
- os/kernel: 10.0.19043.1165 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.16.5
- go/linking: dynamic
- go/tags: cmount
Which cloud storage system are you using? (eg Google Drive)
The command you were trying to run (eg
rclone copy /tmp remote:tmp)
The rclone config contents with secrets removed.
[DFS - Music]
type = drive
client_id = x
client_secret = x
scope = drive
token = x
team_drive = x
A log from the command with the
2021/08/31 13:30:52 DEBUG : vfs cache RemoveNotInUse (maxAge=3600000000000000000, emptyOnly=false): item file1.flac not removed, freed 0 bytes
2021/08/31 13:30:52 DEBUG : vfs cache RemoveNotInUse (maxAge=3600000000000000000, emptyOnly=false): item file2.flac not removed, freed 0 bytes
2021/08/31 13:30:52 INFO : vfs cache: cleaned: objects 16144 (was 16144) in use 0, to upload 0, uploading 0, total size 368.726G (was 368.726G)
That seems to indicate that the cache size is 368 GB, not 691, but Explorer is showing 691. Is something wrong in my config that's making the cache larger than it should be? I suspect that the max-age might be the issue, but these files will rarely change and if they get cached locally, I want to keep them local until I hit the size of my cache, then they should get removed. Is that not how this is set?
Please use the help and support template.
Animosity022 knows more about the vfs settings than i do
he will let us know if i am incorrect about anything
rclone uses sparse files, which can be very confusing.
"So if an application only reads the starts of each file, then rclone will only buffer the start of the file. These files will appear to be their full size in the cache, but they will be sparse files with only the data that has been downloaded present in them."
for example, there is media file that i just started to watch and paused after a few seconds.
rclone mount to be compatible with the local file system, rclone will create a sparse file of size 699MB but the size on disk will be 0B, zero bytes.
as rclone downloads chunks that size on disk will increase over time from 0B to 699MB
vlc expects a media file to have fixed size that will not change, but rclone wants to just download the chunks, on the fly.
the sparse file makes peace with these two opposing needs.
--vfs-cache-max-size 512G is the
size on disk, not the
Volume in drive D is EN08DDRIVE
Volume Serial Number is 3ED2-ACE8
Directory of d:\cache\vfs\mmmcrypt\m\media\movies\The.darwin.awards
04/29/2021 07:23 PM 733,360,128 The.darwin.awards.avi
1 File(s) 733,360,128 bytes
On my computer, that appears to be the case for the vfsMeta folder, but the vfs folder is pretty close to the same (note, since posting, I changed the max-age to at least get something to purge from the cache so my drive had enough free space to grab new files to listen to so it now shows the size of 586, not the 691 I had in my first post).
Not sure what's happening, but my cache size just keeps growing (well over the vfs-cache-max-size). Thanks!
It uses sparse files so while it might 'look' like a full file. It isn't.
If you share a debug log, you can see the actual log size.
My media program stopped downloading new files until I reduced the cache time. That seems like the drive was actually full, no? When the drive showed 23 mb free, it would not play files that were larger than 23 mb until I changed the cache age settings, which freed up 100 gb.
Without a log file, I really can't offer much else.
I'm happy to provide a log, the problem is it's not a singular issue. I've had logging to the screen turned on for a few days and see loads of this:
2021/09/03 19:42:38 DEBUG : file.flac: vfs cache: checking remote fingerprint "2322766,2021-09-04 02:20:10 +0000 UTC,973bd18f771bde19f9f5a337e550aede" against cached fingerprint "2322766,2021-09-04 02:20:10 +0000 UTC,973bd18f771bde19f9f5a337e550aede"
2021/09/03 19:42:38 ERROR : file.flac: vfs cache: truncate: failed to set as a sparse file: DeviceIoControl FSCTL_SET_SPARSE: Incorrect function.
2021/09/03 19:42:38 DEBUG : file.flac: vfs cache: truncate to size=2322766
2021/09/03 19:42:38 ERROR : file.flac: vfs cache: failed to set as a sparse file: DeviceIoControl FSCTL_SET_SPARSE: Incorrect function.
That means you have your file system as something that doesn't support sparse files.
That's only a snippet of a log
So we can't see what's being cleaned up or the size.
That's why I keep asking for a full log.
What file system is the cache?
It's an exFat disc. The only purpose of this disc is for caching this data, so I'll change it if need be. I'd be happy to provide a full log, it just happens to be huge. It's 182 mb, after running just for 10 minutes (before my last post I changed it to output to a file, not the screen.)
ExFat doesn't support sparse files as it would need to be NTFS.