Cache and crypt

What is the problem you are having with rclone?

I noticed this with my mounts fist, however working backwards to try and find the root cause, it appears as-if my performance suffers greatly as soon as I use the cache remote.

This is a Gdrive encrypted remote, and reading the docs we know:

During testing, I experienced a lot of bans with the remotes in this order. I suspect it might be related to how crypt opens files on the cloud provider which makes it think we're downloading the full file instead of small chunks. Organizing the remotes in this order yields better results: cloud remote -> cache -> *crypt

I assumed I am doing it right but something appears odd when I look at the timings to perform a ls -alR on a mount. I see the same using 'lsl' shown below.

What is your rclone version (output from rclone version)

V1.52 & v1.52.0-038-g2571da3b-vfs-beta however also earlier versions

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Linux 64, using rclone/rclone Docker container

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Configuration:

[gdrive2]
type = drive
client_id = id_here
client_secret = secret here
token = token_here
root_folder_id = 0BEhS1smLMffMUk2AZA

[gdrive2-data]
type = crypt
remote = gdrive2:data
filename_encryption = standard
password = password1
password2 = password2

[gdrive2-data-cache]                                                                                        
type = cache                                                                                                
remote = gdrive2:data                                                                                       
plex_url =                                                                                                  
plex_username =                                                                                             
plex_password =                                                                                             
chunk_size = 8M                                                                                             
info_age = 1h                                                                                               
chunk_total_size = 10G                                                                                      
                                                                                                            
[gdrive2-data-cache-crypt]                                                                                  
type = crypt                                                                                                
remote = gdrive2-data-cache:                                                                                
filename_encryption = standard                                                                              
password = password1
password2 = password2

So, the closest of doing a 'ls -alR' appears to be 'lsl' and it takes nearly no time for lots of files:

# time rclone -vvv --config /config/rclone.conf lsl gdrive2-data:download/mystuff
real    2m 12.03s
user    0m 12.03s
sys     0m 1.95s

Mounting the same remote and doing it by hand takes 30minutes but that is also great compared to what we see later on:

# mkdir /mnt/test
# rclone -vvv --config /config/rclone.conf mount gdrive2-data:download/mystuff /mnt/test >/dev/null  &
# time ls -alR /mnt/test > /dev/null
..
real	30m 13.94s
user	0m 1.55s
sys	0m 3.35s

However, if I touch the crypted or not-crypted cache remotes take ~48h for the tree to be traversed.

Trying with remote gdrive2-data-cache-crypt, which is crypt -> cache -> gdrive you will see it takes ~11 minutes before files begin listening:

# time rclone -vvv --config /config/rclone.conf lsl gdrive2-data-cache-crypt:download/mystuff
2020/06/05 12:30:10 DEBUG : rclone: Version "v1.52.0-038-g2571da3b-vfs-beta" starting with parameters ["rclo
ne" "-vvv" "--config" "/config/rclone.conf" "lsl" "gdrive2-data-cache-crypt:download/mystuff"]
2020/06/05 12:30:10 DEBUG : Using config file from "/config/rclone.conf"
2020/06/05 12:30:19 DEBUG : gdrive2-data-cache: wrapped gdrive2:data/2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee
5s41wjv4mfknap at root 2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap
2020/06/05 12:30:19 INFO  : gdrive2-data-cache: Cache DB path: /root/.cache/rclone/cache-backend/gdrive2-dat
a-cache.db
2020/06/05 12:30:19 INFO  : gdrive2-data-cache: Cache chunk path: /root/.cache/rclone/cache-backend/gdrive2-
data-cache
2020/06/05 12:30:23 INFO  : gdrive2-data-cache: Chunk Memory: true
2020/06/05 12:30:23 INFO  : gdrive2-data-cache: Chunk Size: 8M
2020/06/05 12:30:23 INFO  : gdrive2-data-cache: Chunk Total Size: 10G
2020/06/05 12:30:23 INFO  : gdrive2-data-cache: Chunk Clean Interval: 1m0s
2020/06/05 12:30:23 INFO  : gdrive2-data-cache: Workers: 4
2020/06/05 12:30:23 INFO  : gdrive2-data-cache: File Age: 1h0m0s
2020/06/05 12:30:23 DEBUG : Adding path "cache/expire" to remote control registry
2020/06/05 12:30:23 DEBUG : Adding path "cache/stats" to remote control registry
2020/06/05 12:30:23 DEBUG : Adding path "cache/fetch" to remote control registry
2020/06/05 12:30:24 DEBUG : gdrive2-data-cache: wrapped gdrive2:data/2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap at root 2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap
2020/06/05 12:30:24 INFO  : gdrive2-data-cache: Cache DB path: /root/.cache/rclone/cache-backend/gdrive2-data-cache.db
2020/06/05 12:30:24 INFO  : gdrive2-data-cache: Cache chunk path: /root/.cache/rclone/cache-backend/gdrive2-data-cache
2020/06/05 12:30:24 INFO  : gdrive2-data-cache: Chunk Memory: true
2020/06/05 12:30:24 INFO  : gdrive2-data-cache: Chunk Size: 8M
2020/06/05 12:30:24 INFO  : gdrive2-data-cache: Chunk Total Size: 10G
2020/06/05 12:30:24 INFO  : gdrive2-data-cache: Chunk Clean Interval: 1m0s
2020/06/05 12:30:24 INFO  : gdrive2-data-cache: Workers: 4
2020/06/05 12:30:24 INFO  : gdrive2-data-cache: File Age: 1h0m0s
2020/06/05 12:30:24 DEBUG : Adding path "cache/expire" to remote control registry
2020/06/05 12:30:24 DEBUG : Adding path "cache/stats" to remote control registry
2020/06/05 12:30:24 DEBUG : Adding path "cache/fetch" to remote control registry
2020/06/05 12:30:24 DEBUG : Cache remote gdrive2-data-cache:2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap: list recursively from ''
2020/06/05 12:31:23 DEBUG : Cache remote gdrive2-data-cache:2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap: starting cleanup
2020/06/05 12:31:24 DEBUG : Cache remote gdrive2-data-cache:2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap: starting cleanup
2020/06/05 12:31:24 DEBUG : Google drive root 'data/2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap': Checking for changes on remote
2020/06/05 12:31:24 DEBUG : Google drive root 'data/2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap': Checking for changes on remote
2020/06/05 12:32:24 DEBUG : Google drive root 'data/2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap': Checking for changes on remote
2020/06/05 12:32:24 DEBUG : Google drive root 'data/2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap': Checking for changes on remote
2020/06/05 12:32:26 DEBUG : Cache remote gdrive2-data-cache:2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap: starting cleanup
2020/06/05 12:32:27 DEBUG : Cache remote gdrive2-data-cache:2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap: starting cleanup
2020/06/05 12:33:24 DEBUG : Google drive root 'data/2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap': Checking for changes on remote
2020/06/05 12:33:24 DEBUG : Google drive root 'data/2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap': Checking for changes on remote
2020/06/05 12:33:34 DEBUG : Cache remote gdrive2-data-cache:2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap: starting cleanup
2020/06/05 12:33:37 DEBUG : Cache remote gdrive2-data-cache:2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap: starting cleanup
2020/06/05 12:34:24 DEBUG : Google drive root 'data/2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap': Checking for changes on remote
2020/06/05 12:34:24 DEBUG : Google drive root 'data/2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap': Checking for changes on remote
2020/06/05 12:34:45 DEBUG : Cache remote gdrive2-data-cache:2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap: starting cleanup
2020/06/05 12:34:48 DEBUG : Cache remote gdrive2-data-cache:2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap: starting cleanup
2020/06/05 12:35:24 DEBUG : Google drive root 'data/2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap': Checking for changes on remote
2020/06/05 12:32:24 DEBUG : Google drive root 'data/2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap': Checking for changes on remote
2020/06/05 12:32:26 DEBUG : Cache remote gdrive2-data-cache:2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap: starting cleanup
2020/06/05 12:32:27 DEBUG : Cache remote gdrive2-data-cache:2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap: starting cleanup
2020/06/05 12:33:24 DEBUG : Google drive root 'data/2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap': Checking for changes on remote
2020/06/05 12:33:24 DEBUG : Google drive root 'data/2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap': Checking for changes on remote
2020/06/05 12:33:34 DEBUG : Cache remote gdrive2-data-cache:2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap: starting cleanup
2020/06/05 12:33:37 DEBUG : Cache remote gdrive2-data-cache:2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap: starting cleanup
2020/06/05 12:34:24 DEBUG : Google drive root 'data/2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap': Checking for changes on remote
2020/06/05 12:34:24 DEBUG : Google drive root 'data/2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap': Checking for changes on remote
2020/06/05 12:34:45 DEBUG : Cache remote gdrive2-data-cache:2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap: starting cleanup
2020/06/05 12:34:48 DEBUG : Cache remote gdrive2-data-cache:2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap: starting cleanup
2020/06/05 12:35:24 DEBUG : Google drive root 'data/2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap': Checking for changes on remote
2020/06/05 12:38:31 DEBUG : Cache remote gdrive2-data-cache:2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap: starting cleanup
2020/06/05 12:39:24 DEBUG : Google drive root 'data/2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap': Checking for changes on remote
2020/06/05 12:39:24 DEBUG : Google drive root 'data/2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap': Checking for changes on remote
2020/06/05 12:39:28 DEBUG : Cache remote gdrive2-data-cache:2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap: starting cleanup
2020/06/05 12:39:46 DEBUG : Cache remote gdrive2-data-cache:2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap: starting cleanup
2020/06/05 12:40:24 DEBUG : Google drive root 'data/2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap': Checking for changes on remote
2020/06/05 12:40:24 DEBUG : Google drive root 'data/2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap': Checking for changes on remote
2020/06/05 12:40:29 DEBUG : Cache remote gdrive2-data-cache:2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap: starting cleanup
2020/06/05 12:40:54 DEBUG : Cache remote gdrive2-data-cache:2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap: starting cleanup
2020/06/05 12:41:24 DEBUG : Google drive root 'data/2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap': Checking for changes on remote
2020/06/05 12:41:24 DEBUG : Google drive root 'data/2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap': Checking for changes on remote
2020/06/05 12:41:33 DEBUG : Cache remote gdrive2-data-cache:2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap: starting cleanup
2020/06/05 12:41:55 DEBUG : Cache remote gdrive2-data-cache:2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap: starting cleanup
2894447174 2020-06-04 11:24:47.000000000 My.Folder/Season.01/My.File.-.s01e04.WEBDL-1080p.mkv
3005171835 2020-06-04 11:17:44.000000000 My.Folder/Season.01/My.File.-.s01e06.WEBDL-1080p.mkv
3347682910 2020-06-04 11:15:35.000000000 My.Folder/Season.01/My.File.-.s01e03.WEBDL-1080p.mkv
3112497731 2020-06-04 10:28:04.000000000 My.Folder/Season.01/My.File.-.s01e02.WEBDL-1080p.mkv
2857634337 2020-06-04 10:21:56.000000000 My.Folder/Season.01/My.File.-.s01e05.WEBDL-1080p.mkv
...and so it goes on...

This takes ~48h to complete

The same slowness is there when using remote gdrive2-data-cache, which is cache -> gdrive only.

First I get the path name since this is non-crypted remote,, then the lsl:

# rclone --config /config/rclone.conf cryptdecode --reverse gdrive2-data: download/mystuff/
download/mystuff/        2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap/

# rm -rf /root/.cache/rclone
# time rclone -vvv --config /config/rclone.conf lsl gdrive2-data-cache:2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap/
2020/06/05 14:56:12 DEBUG : rclone: Version "v1.52.0-038-g2571da3b-vfs-beta" starting with parameters ["rclon
e" "-vvv" "--config" "/config/rclone.conf" "lsl" "gdrive2-data-cache:2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5
s41wjv4mfknap/"]
2020/06/05 14:56:12 DEBUG : Using config file from "/config/rclone.conf"
2020/06/05 14:56:14 DEBUG : gdrive2-data-cache: wrapped gdrive2:data/2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap at root 2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap
2020/06/05 14:56:14 INFO  : gdrive2-data-cache: Cache DB path: /root/.cache/rclone/cache-backend/gdrive2-data-cache.db
2020/06/05 14:56:14 INFO  : gdrive2-data-cache: Cache chunk path: /root/.cache/rclone/cache-backend/gdrive2-data-cache
2020/06/05 14:56:15 INFO  : gdrive2-data-cache: Chunk Memory: true
2020/06/05 14:56:15 INFO  : gdrive2-data-cache: Chunk Size: 8M
2020/06/05 14:56:15 INFO  : gdrive2-data-cache: Chunk Total Size: 10G
2020/06/05 14:56:15 INFO  : gdrive2-data-cache: Chunk Clean Interval: 1m0s
2020/06/05 14:56:15 INFO  : gdrive2-data-cache: Workers: 4
2020/06/05 14:56:15 INFO  : gdrive2-data-cache: File Age: 1h0m0s
2020/06/05 14:56:15 DEBUG : Adding path "cache/expire" to remote control registry
2020/06/05 14:56:15 DEBUG : Adding path "cache/stats" to remote control registry
2020/06/05 14:56:15 DEBUG : Adding path "cache/fetch" to remote control registry
2020/06/05 14:56:15 DEBUG : fs cache: renaming cache item "gdrive2-data-cache:2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap/" to be canonical "gdrive2-data-cache:2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap"
2020/06/05 14:56:15 DEBUG : Cache remote gdrive2-data-cache:2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap: list recursively from ''
2020/06/05 14:56:15 DEBUG : Cache remote gdrive2-data-cache:2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap: list recursively from ''
2020/06/05 14:57:15 DEBUG : Cache remote gdrive2-data-cache:2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap: starting cleanup
2020/06/05 14:57:16 DEBUG : Google drive root 'data/2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap': Checking for changes on remote
2020/06/05 14:58:16 DEBUG : Google drive root 'data/2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap': Checking for changes on remote
2020/06/05 14:58:19 DEBUG : Cache remote gdrive2-data-cache:2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap: starting cleanup
2020/06/05 14:59:16 DEBUG : Google drive root 'data/2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap': Checking for changes on remote
2020/06/05 14:59:23 DEBUG : Cache remote gdrive2-data-cache:2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap: starting cleanup
2020/06/05 15:00:16 DEBUG : Google drive root 'data/2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap': Checking for changes on remote
2020/06/05 15:00:25 DEBUG : Cache remote gdrive2-data-cache:2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap: starting cleanup
2020/06/05 15:01:16 DEBUG : Google drive root 'data/2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap': Checking for changes on remote
2020/06/05 15:01:39 DEBUG : Cache remote gdrive2-data-cache:2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap: starting cleanup
2020/06/05 15:02:16 DEBUG : Google drive root 'data/2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap': Checking for changes on remote
2020/06/05 15:02:16 DEBUG : gdrive2: Loaded invalid token from config file - ignoring
2020/06/05 15:02:26 DEBUG : gdrive2: Saved new token in config file
2020/06/05 15:02:48 DEBUG : Cache remote gdrive2-data-cache:2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap: starting cleanup
2020/06/05 15:03:16 DEBUG : Google drive root 'data/2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap': Checking for changes on remote
2020/06/05 15:03:50 DEBUG : Cache remote gdrive2-data-cache:2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap: starting cleanup
2020/06/05 15:04:16 DEBUG : Google drive root 'data/2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap': Checking for changes on remote
2020/06/05 15:04:54 DEBUG : Cache remote gdrive2-data-cache:2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap: starting cleanup
2020/06/05 15:05:16 DEBUG : Google drive root 'data/2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap': Checking for changes on remote
2020/06/05 15:05:58 DEBUG : Cache remote gdrive2-data-cache:2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap: starting cleanup
2020/06/05 15:06:16 DEBUG : Google drive root 'data/2r2rqgh1ghufu7243pse9v51cf/nnwagqghb2ee5s41wjv4mfknap': Checking for changes on remote
5646219503 2020-05-21 10:41:59.000000000 n4ap2nf45prqsi4rdr2bammdi8/mkoshdh4gcm20nqiud9f9qvbu4/46fk088d9tap39jcgbr1oosv5r34n97r3tg56vaqq8phhlqkh1vnjrnogq13q4vdojn9v2hv83hvu
4955809795 2020-05-21 10:41:59.000000000 n4ap2nf45prqsi4rdr2bammdi8/mkoshdh4gcm20nqiud9f9qvbu4/ec21k1lhm2g21h0ikakqcig2iean4k2td6gulao5lo7cm9oal10p9u8i4uovghcradl3kqm9mqhqq
5448590993 2020-05-21 10:41:59.000000000 n4ap2nf45prqsi4rdr2bammdi8/mkoshdh4gcm20nqiud9f9qvbu4/9pii1u7p8at22754ukcc97v79o7olkvggpao5j5gnaeq6p22dg0n4b7dp9f33tdf7gq6t609ail0a
...

This takes ~48h to complete

Any ideas on what I'm doing wrong? I seem to be hitting some kind of tarr-pit as soon as the cache remote is involved.

Thanks!

Cache has no maintainer and will removed at some point.

Why are you using the cache at all? Just use the regular crypt remote and use --fast-list and you should be fine.

hello,

i know this is a basic question, but is there a specific reason you use the cache remote?

https://rclone.org/cache/#status

Thanks both.

Seems like I left things too long! I've not realised how far along VFS has come.

There was no reason to specifically use the cache remote other than lowering API hits, as @Animosity022 points out. I've peaked at his homescripts and things are looking better of course. I'll focus on VFS, thanks!

Thanks!

1 Like

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.