Lsl commands sometimes nearly instantly, and sometimes they're super slow (cache remote)

Still not sure if i'm doing something wrong, or if this just doesn't work, but using my cache remote, lsl commands are sometimes nearly instantly (clearly using my cache information) and sometimes they're super slow (because it's gigantic remote). The difference is between a few seconds and a few days. So I'd like to understand it more.

This is what happens on -v

2020/05/04 17:26:43 INFO : cleancache: Cache DB path: C:\Users\MYNAME\AppData
Local\rclone\cache-backend\cleancache.db
2020/05/04 17:26:43 INFO : cleancache: Cache chunk path: C:\Users\MYNAME\AppDa
ta\Local\rclone\cache-backend\cleancache
2020/05/04 17:26:43 INFO : cleancache: Chunk Memory: true
2020/05/04 17:26:43 INFO : cleancache: Chunk Size: 5M
2020/05/04 17:26:43 INFO : cleancache: Chunk Total Size: 500M
2020/05/04 17:26:43 INFO : cleancache: Chunk Clean Interval: 1m0s
2020/05/04 17:26:43 INFO : cleancache: Workers: 4
2020/05/04 17:26:43 INFO : cleancache: File Age: 8.333333333333334M
2020/05/04 17:26:44 INFO : cleancache: Cache DB path: C:\Users\MYNAME\AppData
Local\rclone\cache-backend\cleancache.db
2020/05/04 17:26:44 INFO : cleancache: Cache chunk path: C:\Users\MYNAME\AppDa
ta\Local\rclone\cache-backend\cleancache
2020/05/04 17:26:44 INFO : cleancache: Chunk Memory: true
2020/05/04 17:26:44 INFO : cleancache: Chunk Size: 5M
2020/05/04 17:26:44 INFO : cleancache: Chunk Total Size: 500M
2020/05/04 17:26:44 INFO : cleancache: Chunk Clean Interval: 1m0s
2020/05/04 17:26:44 INFO : cleancache: Workers: 4
2020/05/04 17:26:44 INFO : cleancache: File Age: 8.333333333333334M

Do I need a larger chunk size? Did I setup my cache wrong? Is this just working as intended?

My cleancache.db file is 2gb and my cloud is about 30tb.

On -vv I get a lot of this:

2020/05/04 17:41:06 DEBUG : Adding path "cache/expire" to remote control registr
y
2020/05/04 17:41:06 DEBUG : Adding path "cache/stats" to remote control registry

2020/05/04 17:41:06 DEBUG : Adding path "cache/fetch" to remote control registry

2020/05/04 17:41:06 DEBUG : Cache remote cleancache:FILEPATH: list recursively from ''
2020/05/04 17:41:07 DEBUG : pacer: low level retry 1/10 (error googleapi: Error
403: User Rate Limit Exceeded. Rate of requests for user exceed configured proje
ct quota. You may consider re-evaluating expected per-user traffic to the API an
d adjust project quota limits accordingly. You may monitor aggregate quota usage
and adjust limits in the API Console: https://console.developers.google.com/api
s/api/drive.googleapis.com/quotas?project=202264815644, userRateLimitExceeded)
2020/05/04 17:41:07 DEBUG : pacer: Rate limited, increasing sleep to 1.991278884
s
2020/05/04 17:41:07 DEBUG : pacer: Reducing sleep to 0s

2020/05/04 17:42:06 DEBUG : Cache remote cleancache:FILEPATH: starting cleanup
2020/05/04 17:42:06 DEBUG : Google drive root 'cleancrypt/FILEPATH': Checking for changes on remote
2020/05/04 17:42:06 DEBUG : Cache remote cleancache:FILEPATH: starting cleanup
2020/05/04 17:42:06 DEBUG : Google drive root 'cleancrypt/FILEPATH': Checking for changes on remote

the cache backend is no longer maintained and has known bugs.

is there a reason you use it?

what is your rclone command?
what version of rclone are you using?

"User Rate Limit Exceeded"
have you tried to use flags to slow down the number of api calls?

https://rclone.org/drive/#limitations
"Drive has quite a lot of rate limiting"

rclone lsl -v --fast-list "cachecrypt:filepaths" is the command I would use, or size or lsd.

The main reason I use cache is purely to speed up lsl/lsd/size/etc. Is there a better way to speed those up than cache?

Should I just turn my cache off? My crypt is nested inside my cache so I can just use the crypt naked without cache.

i find no need for cache remote.

but @Animosity022 and @thestigma know all about google storage.

What version of rclone are you running?
What version of winfsp?

What does your rclone.conf look like without the keys/passwords?

I do not know what winfsp even is. I'm not mounting this as a filesystem.

you can ignore winfsp the comment, as you are not running a mount.
but the other info requested is needed...

I've got a lot of information in the conf that's useless that I'm editing out. hopefully I left the key information in:

[googledrive]
type = drive
service_account_file =
token = {"access_token":"","token_type":"Bearer","refresh_token":"","expiry":"2020-05-04T18:26:41.0854241-04:00"}
team_drive =
root_folder_id =

[cleancrypt]
type = crypt
remote = googledrive:cleancrypt
filename_encryption = obfuscate
directory_name_encryption = true
password =
password2 =

[cleancache]
type = cache
remote = googledrive:cleancrypt
plex_url =
plex_username =
plex_password =
chunk_size = 5M
info_age = 8.333333333333334M
chunk_total_size = 500M

[cleancachecrypt]
type = crypt
remote = cleancache:
filename_encryption = obfuscate
directory_name_encryption = true
password =
password2 =

What version of rclone?

Hmm, usually 1.51, but I just realized that recently I accidentally used rclone 1.47.

I've got all the rclone folders I ever used sitting next to each other: 1.25 1.26 1.27 1.29 1.32 1.33 1.35 1.39 1.40 1.41 1.42 1.43 1.44 1.45 1.46 1.47 1.51.

I ran it again in 1.51, and it works just the same (not well). This "feature" has never "worked" for me though, ever, in any version.

I'm not even sure if it's possible for it to work. I'm not even sure if the cache tries or wants or is designed to store lsl data. I've just always hoped it would.

You cache time is set for 8.3 minutes? You probably want to set that up to be a bit higher.

Mixing versions with the cache backend is going to produce some odd results as you have versions mixed in there that it didn't exist.

The cache backend was primarily built for Plex and to be used with a mount.

What are you trying to accomplish? Keep certain files on disk? Keep more of the directory/file content on disk? Are you changing your remote elsewhere as well ?

The goal is to "Keep more of the directory/file content on disk?" (Purely to speed up lsl commands). I've found that sometimes if I run a bunch of lsl commands back to back they vastly speed up after the first one (working). However sometimes it doesn't speed up at all ever (not-working).

And I thought it was 8.3 MONTHS! If M is minutes not Months that would entirely explain my confusion.

Also yeah, I expect it to break whenever I change versions, but usually what I do is open one windows cmd window and then run all the commands through there without changing directory. So, unless I make a mistake I shouldn't be changing versions more than once every time I reboot which is every 2-3months.

I guess maybe I should switch to using mount, but trying to mount a filesystem seems excessive when all I want to do is speed up the second lsl command when I run two in a row.

You'd have to run a command with -vv and you can see what it is doing.

If it takes long, it's most likely recaching the items, which should get fixed by extending the info age.

Offhand, I don't know how the cache would know a change was made as normally it's a mount and polling tells it a change happened to invalidate the cache.

With it not being a mount, I'm not sure how that would work.

If you capture a few times and have a 'slow' one with a log with -vv, we can see why.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.