--no-traverse too slow with lot of files

Help and Support
21

hi, have you tested with --fast-list?

22

didn't help....

23

yes. nothing changed.

24

--drive-pacer-min-sleep=0ms
that is for gdrive, not GCS.
so makes sense that the flag did not make a difference.

1 Like
25

I guess I should explain why I wanted to try --drive-pacer-min-sleep=0ms:

The GCS backend uses the Google Drive (gdrive) pacer with a minimum sleep of 10ms:

This limits rclone to a maximum of 100 requests per second (a request is e.g. a check to see if an object exists as needed by --no-traverse).

The above Quotas and limits for GCS shows that it can handle 5,000 object reads (incl. metadata) per second and then scale as needed.

The current rclone pacer settings for GCS therefore seems to reduce the object checking speed by a factor 50.

I think next step is for @ncw to confirm my observation. If correct, then he can probably make a beta where minSleep is changed to 0ms or introduce --gcs-pacer-min-sleep to allow configuration.

The OneDrive log and pacing seems fine, that is relatively slow as expected. --no-traverse will be the slowest approach unless you are checking (and possibly copying) a small number of source files to a destination with a (very) high number of files/folders.

26

I think the pacer has gone wrong... Notice how the sleep isn't reducing its getting bigger, and there are no low level retries.

I've had a look at the code and this is because the pacer is misconfigured - the burst size is wrong by default. I fixed that then I realised that using the S3 pacer would be better which doesn't have any delay except in error conditions which is what you want when you pay for transactions.

So can you give this a go?

v1.59.0-beta.6041.1b1ed9969.fix-gcs-pacer on branch fix-gcs-pacer (uploaded in 15-30 mins)

1 Like
27

now...
with --no-traverse we get 18.752 MiB/s

and no more DEBUG : pacer: Reducing sleep to in the log.

thats really great!

1 Like
28

Thanks for testing :slight_smile:

I've merged this to master now which means it will be in the latest beta in 15-30 minutes and released in v1.59

I'll put it in a v1.58.1 at some point too

1 Like
29

And here is the stable branch build

v1.58.1-beta.6040.5074bd0d5.v1.58-stable on branch v1.58-stable (uploaded in 15-30 mins)

1 Like
30

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.