Rclone Cache with GDrive not working properly

Help and Support
1

I've tried setting up the Rclone cache but it does not provide any performance improvement using rclone sync or rclonebrowser in comparison to without cache

My intend is to make rclonebrowser faster. Rclone cache provides metadata cache so i feel it should provide the desired result but it does not

I don't care about chunk cache. GDrive is a backup location for me so most of my data is uploaded, not downloaded

The metadata cache cacheGDrive.db is sized at 256MB of which ~60% is filled with data

Goals:

  • Make rclonebrowser faster when traversing directories

Added benefit

  • Take some of the weight of the GDrive API so that i do not trigger the rate limit so soon

Any input as to what i'm doing wrong would be appreciated

===============================
[remoteGDrive]
type = drive
client_id =
client_secret =
scope = drive
root_folder_id =
service_account_file =
token = ****

[cacheGDrive]
type = cache
remote = remoteGDrive:Crypt
info_age = 1w
chunk_clean_interval = 1m
db_path = E:\rClone-Cache\cache-backend
chunk_path = E:\rClone-Cache\cache-backend
db_purge = false
workers = 1
rps = 5
writes = false
db_wait_time = 1m

[cryptGDcache]
type = crypt
remote = cacheGDrive:
filename_encryption = standard
directory_name_encryption = true
password = ******
password2 = ******
===============================

rclone-v1.47.0-070-g8c550363-beta-windows-amd64

C:\rclone\rclone.exe sync --bwlimit 3M --tpslimit 1 --checkers 4 --fast-list --delete-after --delete-excluded --exclude-from "C:/rclone/Configs/Exclusions.txt" --progress --log-file "C:\Users\User\Desktop\MirrorD Log\Rclone Mirror_D.log" --log-level INFO D:/ cryptGDcache:D

Rclonebrowser uses the following to obtain its directory information
https://rclone.org/commands/rclone_lsd/
https://rclone.org/commands/rclone_lsl/

2

You basically gimped your mount with tpslimit 1 as you should remove that.

checkers does nothing on a mount.

don't think exclude or the deletes do anything a mount as those are for copy/move/sync.

The bwlimit would also gimp it quite a bit so I'd remove that too.

Is there a reason you are using the beta ?

3

it's not a mount.

Only rclone browser and rclone sync
tpslimit 1 only applies to rclone sync, not to rclonebrowser

I'm using the beta because i had an issue with Google drive file stream and rclone that i needed to report

4

For sync, tpslimit gimps you command and makes it slow along with the bwlimit of only 3M.

Run the same command with -vv and share the log.

5

I will get those logs
bwlimit is because i only have 5M upload. So cannot have it eating up all of my upload bandwidth
tpslimit is because of the google drive rate limiting. I felt that it helped

But can we get back to the main focus of this issue which is rclonebrowser or at least the underlying commands
Why doesn't the cache make any difference in its performance (the earlier things mentioned by you do not apply to rclonebrowser)?

6

Run the sync with -vv and share the output of the debug log. The cache definitely makes a difference after it's warmed up.

As I noted though with the 1 tps limit, you are seriously gimping your performance as a whole.

7

Here are the logs.

The logs for the 2 mentioned rclonebrowser commands
Rclone TEST-D-LSD 2019-05-23 22208.log
Rclone TEST-D-LSL 2019-05-23 22208.log
Rclone TEST-LSD 2019-05-23 22208.log
Rclone TEST-LSL 2019-05-23 22208.log
Rclone TEST-NAS-LSD 2019-05-23 22208.log
Rclone TEST-NAS-LSL 2019-05-23 22208.log
Rclone TEST-TEST-LSD 2019-05-23 22208.log
Rclone TEST-TEST-LSL 2019-05-23 22208.log
So it seems the issue is rate limiting related.
These 8 were run in sequence after each other
Rclonebrowser normally starts 2 paralel to each other for every opened directory. Which means it could be worse then in sequence

As for the sync command with and without tps made a 2 minute difference in favor of with.
Rclone Mirror_D-WITH TPS 2019-05-23 03515.log
Rclone Mirror_D-WITHOUT TPS 2019-05-23 03515.log
44 minutes for with tps 1
46 minutes for without tps
The reason is most likely a severe case of rate limiting for without tps
For these tests there were no changes to be synced. So only checks were performed.
The logs were 54 mb in size so i shrunk them down a bit

So all in all i think the cache is working just fine. It's just that there are still commands pushed to Gdrive causing rate limiting when using rclonebrowser.

8

You need to setup your own API key/client ID.

https://rclone.org/drive/#making-your-own-client-id

That's the cause for the rate limiting as the default rclone key is shared and a bit over subscribed.

9

That helps a lot. Thank you

Can't believe i overlooked that when setting everything up

10

I am noticing based on the below logs that empty dirs are not cached. Is that on purpose?

Rclone TEST-TEST-LSD 2019-05-23 22208.log
Rclone TEST-TEST-LSL 2019-05-23 22208.log

11

It's a bug:

12

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.