I can't run this command
rclone copy drive: cf: --transfers 25 -vP --stats 15s --fast-list --checkers 35 --size-only --multi-thread-streams 0 --no-traverse
Because it disables --fast-list thinking there is a bug because the directories are empty, this causes google drive to rate limit it so much that it takes ~20min for this folder.
There is ~5k subfolders, and they are empty because I only use google drive now for new files, but I need to have the same folder layout as the other remote.
Please provide all details as requested in Help and Support template.
And what exactly the issue is? That your command is slow? What makes you think that there is a bug?
I think OP is referring to this known issue:
Specifically:
This can potentially have a false positive if the user has multiple
empty directories which are queried at once. The consequence of this
will be that ListR is disabled for a while until the directories are
found to be actually empty in which case it will be re-enabled.
I have sometimes encountered it too, and I generally avoid using --fast-list because of it.
Thanks for that. If you are right than solution is simple:) or rather there is no issue at all - it is just gdive bug.
and BTW it is still not fixed after 3 years...
Yes this is the issue. Is there any way to force fast list regardless of the results ?
Nope. And why would you like to do this? We know that then --fast-list will not provide complete directories listing.
Solution is to fix it on google API side - which seem they are not interested to do.
Because it will fix the issue for me, and I don't care if gets bad results or not as I'm running copy and know what i'm doing
then you can compile your own custom rclone version reverting this commit
I can find out how to build rclone but how to revert the commit without causing bugs or issues?
rclone copy 1: cf: --transfers 25 -vvvP --stats 15 --checkers 35 --size-only --multi-thread-streams 0 --no-traverse --fast-list
2023/08/05 05:37:48 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "copy" "1:TV" "cf:TV" "--transfers" "25" "-vvvP" "--stats" "15" "--checkers" "35" "--size-only" "--multi-thread-streams" "0" "--no-traverse" "--fast-list"]
2023/08/05 05:37:48 DEBUG : Creating backend with remote "1:"
2023/08/05 05:37:48 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/08/05 05:37:49 DEBUG : Creating backend with remote "cf:"
2023/08/05 05:37:49 DEBUG : Creating backend with remote "cfcrypt:XXXXXXXXX"
2023/08/05 05:37:49 DEBUG : Resolving service "s3" region "us-east-1"
2023-08-05 05:37:51 DEBUG : Google drive root 'TV': Disabling ListR to work around bug in drive as multi listing (10) returned no entries
2023-08-05 05:37:51 DEBUG : Google drive root 'TV': Recycled 10 entries
2023-08-05 05:37:51 DEBUG : pacer: Reducing sleep to 42.408249ms
2023-08-05 05:37:51 DEBUG : pacer: Reducing sleep to 141.404421ms
2023-08-05 05:37:52 DEBUG : pacer: Reducing sleep to 212.870098ms
2023-08-05 05:37:52 DEBUG : pacer: Reducing sleep to 309.01462ms
2023-08-05 05:37:52 DEBUG : pacer: Reducing sleep to 399.007261ms
2023-08-05 05:37:52 DEBUG : pacer: Reducing sleep to 478.182935ms
2023-08-05 05:37:52 DEBUG : pacer: Reducing sleep to 572.460928ms
2023-08-05 05:37:52 DEBUG : pacer: Reducing sleep to 663.268551ms
2023-08-05 05:37:52 DEBUG : pacer: Reducing sleep to 760.035909ms
2023-08-05 05:37:52 DEBUG : pacer: Reducing sleep to 856.47616ms
2023-08-05 05:37:52 DEBUG : pacer: Reducing sleep to 946.864751ms
2023-08-05 05:37:52 DEBUG : pacer: Reducing sleep to 1.04658393s
2023-08-05 05:37:52 DEBUG : pacer: Reducing sleep to 1.140214089s
2023-08-05 05:37:52 DEBUG : pacer: Reducing sleep to 1.240013789s
2023-08-05 05:37:52 DEBUG : pacer: Reducing sleep to 1.33973885s
2023-08-05 05:37:52 DEBUG : pacer: Reducing sleep to 1.424995929s
2023-08-05 05:37:52 DEBUG : pacer: Reducing sleep to 1.513287975s
2023-08-05 05:37:52 DEBUG : pacer: Reducing sleep to 1.606534017s
2023-08-05 05:37:52 DEBUG : pacer: Reducing sleep to 1.682777818s
2023-08-05 05:37:52 DEBUG : pacer: Reducing sleep to 1.778558281s
2023-08-05 05:37:52 DEBUG : pacer: Reducing sleep to 1.862817544s
2023-08-05 05:37:52 DEBUG : pacer: Reducing sleep to 1.959489822s
2023-08-05 05:37:52 DEBUG : pacer: Reducing sleep to 2.0368001s
2023-08-05 05:37:52 DEBUG : pacer: Reducing sleep to 2.130739343s
2023-08-05 05:37:52 DEBUG : pacer: Reducing sleep to 2.2219836s
2023-08-05 05:37:52 DEBUG : pacer: Reducing sleep to 2.321895094s
2023-08-05 05:37:52 DEBUG : pacer: Reducing sleep to 2.421842213s
2023-08-05 05:37:52 DEBUG : pacer: Reducing sleep to 2.521239474s
2023-08-05 05:37:52 DEBUG : pacer: Reducing sleep to 2.601401392s
2023-08-05 05:37:52 DEBUG : pacer: Reducing sleep to 2.693039654s
2023-08-05 05:37:52 DEBUG : pacer: Reducing sleep to 2.785497506s
2023-08-05 05:37:52 DEBUG : pacer: Reducing sleep to 2.856753577s
2023-08-05 05:37:52 DEBUG : pacer: Reducing sleep to 2.923226315s
2023-08-05 05:37:52 DEBUG : pacer: Reducing sleep to 3.023004467s
2023-08-05 05:37:52 DEBUG : pacer: Reducing sleep to 3.121970011s
2023-08-05 05:37:52 DEBUG : pacer: Reducing sleep to 3.163584166s
2023-08-05 05:37:52 DEBUG : pacer: Reducing sleep to 3.26308832s
2023-08-05 05:37:52 DEBUG : pacer: Reducing sleep to 3.061942571s
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Elapsed time: 4.1s^C
Yeah it is very old commit so rebasing would be potentially hell a lot of work.
Probably easier to analyse its source code and just disable switching fast-list off.
Now these are extreme measures and maybe it would be easier to focus on throttling your command. When it does not hit google API limits it should be faster overall.
try to add --tpslimit=10 and run with default checkers/transfers - might be faster
What I would try is to remove condition which is causing problem - empty directories. Put temporarily some placeholder file in every directory.
rclone lsd gdrive: --recursive | cut -c44- > dirsList.txt
will give me list of all directories
Now simple bash script to create placeholder file in every dir (it can be a bit slow but needs to be run only once):
#!/bin/bash
input="dirsList.txt"
while IFS= read -r line
do
echo "delete later" | rclone rcat "gdrive:$line/placeholder"
done < "$input"
Now --fast-list will fly.
later:
use similar way to delete all placeholder files - create list of files and rclone delete --files-from
Worth noting: despite the name, --fast-list isn't always faster. From the docs:
- It will use fewer transactions (important if you pay for them)
- It will use more memory. Rclone has to load the whole listing into memory.
- It may be faster because it uses fewer transactions
- It may be slower because it can't be parallelized
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.