Best way to speed up gdrive indexing?

Setup:

Google Drive TeamDrive as Source: 20,000-30,000 medium sized files (2-10 gb) in a single directory

Local Array as Destination

rclone command (using service accounts and a crypt remote):

rclone copy/sync src_crypt: dst: -v --tpslimit 6 --transfers 5 --drive-chunk-size 512M --max-transfer=550G --cutoff-mode=soft --log-file=rclone.log

I use the following command to copy files from Google Drive to my local remote, and most of the time (hours) is spent indexing and comparing files between src:dest, with a few minutes spent on actual file transfers. The result is files appearing like 8 hours later, than when they were first available.

What is the best way to get near instant file syncs between the two remotes?

I see there is

--checkers=N

--tpslimit float

Would increasing these two help? What is the maximum I can increase it to without hitting TeamdriveAPI limits?

Is there maybe a more elegant way?

Like:

  1. Get rclone to print all files uploaded to gdrive
  2. Export list to local array
  3. Execute rclone to download only files on this list, rather than running copy/sync on the whole directory, avoiding the whole indexing/checksum process

hi,
when you posted, there was a template of quetions, none of which got answered.
a bit hard to give good advice.

take a full read of this topic
https://forum.rclone.org/t/understanding-pacer-behavior-with-google-drive/29774/7

should not take eight hours. nothing close to that.
a debug log would show what is going on.

--- make sure to create a client id for gdrive.
--- should test using --fasl-list

Hey, I've read up on a few of the options and that thread you linked. So far --fast-list has made a major difference in cutting down runs to an hour. The other options didn't seem to make much of a difference, so thank you already.

The reason for all those questions is I guess I'm just confused what each option applies to even after reading the documentation.

--drive-pacer applies to the gdrive API
--checkers applies to the src/dest comparison, so transferring from local to local would also use this
--tpslimit float I am unsure what this is for. The description makes it sound like the same thing as drive-pacer

--- make sure to create a cliend id for gdrive.

I am using service accounts, and I am unsure what the API/transaction/pacer limits are for this type of account. Standard?

good,

each use-case is different with many variable affecting overall performance.
if you plan to run this sync on a regular basis, i would run a set of tests, tweaking the flags values.
then you will have the optimized command

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.