I use the following command to copy files from Google Drive to my local remote, and most of the time (hours) is spent indexing and comparing files between src:dest, with a few minutes spent on actual file transfers. The result is files appearing like 8 hours later, than when they were first available.
What is the best way to get near instant file syncs between the two remotes?
I see there is
--checkers=N
--tpslimit float
Would increasing these two help? What is the maximum I can increase it to without hitting TeamdriveAPI limits?
Is there maybe a more elegant way?
Like:
Get rclone to print all files uploaded to gdrive
Export list to local array
Execute rclone to download only files on this list, rather than running copy/sync on the whole directory, avoiding the whole indexing/checksum process
Hey, I've read up on a few of the options and that thread you linked. So far --fast-list has made a major difference in cutting down runs to an hour. The other options didn't seem to make much of a difference, so thank you already.
The reason for all those questions is I guess I'm just confused what each option applies to even after reading the documentation.
--drive-pacer applies to the gdrive API --checkers applies to the src/dest comparison, so transferring from local to local would also use this --tpslimit float I am unsure what this is for. The description makes it sound like the same thing as drive-pacer
--- make sure to create a cliend id for gdrive.
I am using service accounts, and I am unsure what the API/transaction/pacer limits are for this type of account. Standard?
each use-case is different with many variable affecting overall performance.
if you plan to run this sync on a regular basis, i would run a set of tests, tweaking the flags values.
then you will have the optimized command