Cache-tmp-upload-path not working in my config?

misteriks · April 18, 2020, 3:26pm

What is the problem you are having with rclone?

I have a cache-tmp-upload-path set in my rclone.conf file, but when I copy a file to the mounted drive I don't see any tmp files in the configured tmp upload location folder.
Also the cache-db-path and cache-chunk-path remain empty

I wonder is this is setup correctly to make it work as expected.

[Gdrive]
type = drive
scope = drive
token = {"access_token":"xxxxxxxxxxxxx","expiry":"2020-04-18T16:11:41.9370251+02:00"}
client_id = xxxxxxxxxxxxxxx.apps.googleusercontent.com
client_secret = xxxxxxxxxxxxxxxx
root_folder_id = xxxxxxxxxxxxxxxx

[GdriveCached]
type = cache
cache-remote = gdrive:media
cache-tmp-wait-time = 1m
cache-chunk-size = 15M
cache-info-age = 1y
cache-workers = 5
cache-plex-url = http://127.0.0.1:32400
cache-plex-username = xxxxxxxxxx@gmail.com
cache-plex-password = xxxxxxxxxxxxxxxxxxxxxx
cache-tmp-upload-path = D:\RcloneCache\TmpUpload
cache-db-path = D:\RcloneCache\CacheDBPath
cache-chunk-path = D:\RcloneCache\CacheChunkPath

What is your rclone version (output from `rclone version`)

v1.51.0

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows 10 x64

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

mount Gdrive: P: --config D:\RcloneCache\rclone.conf --cache-dir D:\RcloneCache\CacheDir --log-file D:\RcloneCache\Rclone.log --log-level INFO --buffer-size 0M --vfs-cache-mode writes --user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36"

A log from the command with the `-vv` flag (eg output from `rclone -vv copy /tmp remote:tmp`)

Animosity022 · April 18, 2020, 5:06pm

This setting is what is capturing the writes.

If you share the debug log, you can see the issue being written there as well.

misteriks · April 18, 2020, 6:54pm

I enabled debug logging and copied a 971 MB MKV file to the mounted drive (P:\media\TV Shows#test)

Debug log is attached debug.log (592.6 KB)

Animosity022 · April 18, 2020, 7:00pm

You still have vfs-cache-mode writes on though so it's not going to write the cache-tmp area.

misteriks · April 18, 2020, 7:03pm

Ah check, that wasn't clear to me. So it is either vfs-cache-mode or cache-tmp-upload-path ?

misteriks · April 18, 2020, 7:09pm

Attached a debug with vfs-cache-mode removed

debug2.log (756.1 KB)

darthShadow · April 18, 2020, 7:55pm

I don't believe the problem is with both vfs-cache-mode and cache-tmp-upload-path being set. If both are set, it will simply write to both locations before finally uploading to the remote.

The problem here is that you are not mounting the correct remote. If you want writes to show up in cache-tmp-upload-path too and basically use all the cache features, you need to mount the cache remote i.e. GdriveCached instead of its backing remote i.e. Gdrive

misteriks · April 18, 2020, 8:01pm

I changed the command line from

mount Gdrive: P: ......
to
mount GdriveCached: P: .....

after starting the service the content of my P: drive is the location of where rclone.exe is located

darthShadow · April 18, 2020, 8:09pm

The cache-remote in your config has the wrong name, it should match the existing remote name i.e. Gdrive instead of gdrive. Also, remove the media folder from it if it doesn't exist.

misteriks · April 18, 2020, 8:16pm

Yeah, I noticed that as well so fiddled around with it a bit.
These are the current settings, but the mounted drive still shows my program files directory of Rclone

mount GdriveCached: P: --config D:\RcloneCache\rclone.conf --vfs-read-chunk-size 128 --vfs-cache-mode writes --cache-dir D:\RcloneCache\CacheDir --log-file D:\RcloneCache\Rclone.log --log-level INFO --buffer-size 0M --vfs-cache-mode writes --user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36"

[Gdrive]
type = drive
scope = drive
token = {"access_token":"XXXXXX","token_type":"Bearer","refresh_token":"xxxx","expiry":"2020-04-18T22:12:19.0218241+02:00"}
client_id = XXXX.apps.googleusercontent.com
client_secret = XXXXX
root_folder_id = XXXX

[GdriveCached]
type = cache
cache-remote = Gdrive
cache-tmp-wait-time = 1m
cache-chunk-size = 15M
cache-info-age = 1y
cache-workers = 5
cache-plex-url = http://127.0.0.1:32400
cache-plex-username = XXXX@gmail.com
cache-plex-password = XXXXXX
cache-tmp-upload-path = D:\RcloneCache\TmpUpload
cache-db-path = D:\RcloneCache\CacheDBPath
cache-chunk-path = D:\RcloneCache\CacheChunkPath

darthShadow · April 18, 2020, 8:17pm

Colon is mandatory, I think. So, Gdrive:, not Gdrive

misteriks · April 18, 2020, 8:23pm

Ok, made that change as well, but still no joy

2020/04/18 22:21:24 DEBUG : rclone: Version "v1.51.0" starting with parameters ["C:\\Program Files\\RClone\\rclone.exe" "mount" "GdriveCached:" "P:" "--config" "D:\\RcloneCache\\rclone.conf" "--vfs-read-chunk-size" "128" "--vfs-cache-mode" "writes" "--cache-dir" "D:\\RcloneCache\\CacheDir" "--log-file" "D:\\RcloneCache\\Rclone.log" "--log-level" "DEBUG" "--buffer-size" "0M" "--vfs-cache-mode" "writes" "--user-agent" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36"]
2020/04/18 22:21:24 DEBUG : Using config file from "D:\\RcloneCache\\rclone.conf"
2020/04/18 22:21:24 DEBUG : GdriveCached: wrapped local://?/C:/Program Files/RClone at root 
2020/04/18 22:21:24 INFO  : GdriveCached: Cache DB path: C:\Windows\system32\config\systemprofile\AppData\Local\rclone\cache-backend\GdriveCached.db
2020/04/18 22:21:24 INFO  : GdriveCached: Cache chunk path: C:\Windows\system32\config\systemprofile\AppData\Local\rclone\cache-backend\GdriveCached
2020/04/18 22:21:24 INFO  : GdriveCached: Chunk Memory: true
2020/04/18 22:21:24 INFO  : GdriveCached: Chunk Size: 5M
2020/04/18 22:21:24 INFO  : GdriveCached: Chunk Total Size: 10G
2020/04/18 22:21:24 INFO  : GdriveCached: Chunk Clean Interval: 1m0s
2020/04/18 22:21:24 INFO  : GdriveCached: Workers: 4
2020/04/18 22:21:24 INFO  : GdriveCached: File Age: 6h0m0s
2020/04/18 22:21:24 DEBUG : Adding path "cache/expire" to remote control registry
2020/04/18 22:21:24 DEBUG : Adding path "cache/stats" to remote control registry
2020/04/18 22:21:24 DEBUG : Adding path "cache/fetch" to remote control registry
2020/04/18 22:21:24 DEBUG : Cache remote GdriveCached:: Mounting on "P:"
2020/04/18 22:21:24 INFO  : Cache remote GdriveCached:: poll-interval is not supported by this remote
2020/04/18 22:21:24 DEBUG : vfs cache root is "D:\\RcloneCache\\CacheDir\\vfs\\GdriveCached"
2020/04/18 22:21:24 DEBUG : Adding path "vfs/forget" to remote control registry
2020/04/18 22:21:24 DEBUG : Adding path "vfs/refresh" to remote control registry
2020/04/18 22:21:24 DEBUG : Adding path "vfs/poll-interval" to remote control registry
2020/04/18 22:21:24 DEBUG : Cache remote GdriveCached:: Mounting with options: ["-o" "fsname=GdriveCached:" "-o" "subtype=rclone" "-o" "max_readahead=131072" "-o" "attr_timeout=1" "-o" "atomic_o_trunc" "-o" "uid=-1" "-o" "gid=-1" "--FileSystemName=rclone" "-o" "volname=GdriveCached"]
2020/04/18 22:21:24 DEBUG : Cache remote GdriveCached:: Init: 
2020/04/18 22:21:24 DEBUG : Cache remote GdriveCached:: >Init:

darthShadow · April 18, 2020, 8:28pm

Did you create the config file manually or interactively via the rclone config ?

All the config options seem incorrect. None of them should have the cache- prefix in their names.

Please create the cache remote again from rclone config

darthShadow · April 18, 2020, 8:29pm

Cache Options: https://rclone.org/cache/#standard-options

Note the Config value for each option, that should be the key present in the config file.

misteriks · April 18, 2020, 8:37pm

It was created via the rclone config long time ago and then I manually added a few things

The cache- prefixes are also mentioned in that standard options doc, so that is why they are in my config.

I'm gonna create the mount from scratch like you suggested to see what that does

darthShadow · April 18, 2020, 8:39pm

You are confusing command-line arguments vs. config file options. For example, if you need to set the cache-remote, you would do it via --cache-remote on the command-line and remote in the config file.

misteriks · April 18, 2020, 8:55pm

That indeed got me confused

I've rebuild the config via the command line and it is now mounting the cached drive.

Thx for your help, really appreciated!

Animosity022 · April 18, 2020, 8:58pm

It only writes to one of them and not both. Most of the time based on writes, it writes to vfs-cache-mode area.

misteriks · April 18, 2020, 9:23pm

I now have both vfs-cache-mode and cache-tmp-upload-path and did a test upload and looks like it is in both locations

btw if I add this to my rclone.conf it doesn't get picked up when rclone is started.
tmp-upload-path = D:\RcloneCache\TmpUpload

Only when I add --cache-tmp-upload-path D:\RcloneCache\TmpUpload that location is used

darthShadow · April 18, 2020, 9:29pm

Again, as per the docs, it is tmp_upload_path, not tmp-upload-path