The first one downloas at 80 Mbytes/s the second one downloads at 50 Mbytes/s
The second download API rate limited which is why there is a slower speed.
You can see it all those lines.
Why is this happening? its the same configurarion and same google drive remote
Since your rclone.conf is missing details, I don’t know if you made your own client ID or not.
Regardless, if you think of it like this as you only have x transactions per second with Google.
If you use one drive, let’s say you may use 10 since you multi stream the download.
If you union and split that into multiple downloads and stream it more times since it’s split over the union, you are now hitting say x times more per second so you get throttled as seen in the logs.
More isn’t always better as you are using big number over defaults and basically choking yourself forcing the slowdown.
Test with defaults and increase after things are working properly.
rclone copy respaldo:/encriptado/qbuksgtaandhbra3mv61ntsqeg prueba: --drive-chunk-size 1024M --progress --multi-thread-streams=10ç
rclone copy respaldo:/encriptado/qbuksgtaandhbra3mv61ntsqeg prueba --drive-chunk-size 1024M --progress --multi-thread-streams=10
[respaldo]
type = drive
client_id = 815032apps.googleusercontent.com
scope = drive
token = {"access_token":"ya29.a0ARrdaM-KWYRHhQsDfIfDZBFPMChuJizenn9RZGwy8-FlK7SnbzYkqt9LQQX6YoKCpi7L8dmlOP5bLm_LsZ3all23NEPcoMMNM950onCP8nprlXdcs_-oHYUI_p8JFg","token_type":"Bearer","refresh_token":"1//0hzIihQJgpE4a-vzOCR6s6CKMyNGi6CfAZSGlFHPiiqDrfRSQs","expiry":"2021-11-20T20:20:16.970659-03:00"}
client_secret = GOCSPX-Z7ViQ
[prueba]
type = union
upstreams = D:/prueba D:/prueba1
[local]
type = local
I tried with multiple thread disabled and its the same.
Do I have something wrong on the config?
I can't see what you tried so I have no idea since you didn't:
- share a command
- share a log
It isn't a config issue as you are hitting API limits as shown in the logs over and over.
I delete de command, I only use rclone copy respaldo:/encriptado/qbuksgtaandhbra3mv61ntsqeg prueba: --drive-chunk-size 1024M --progress
I dont get why I have api limits, so nobody could donwload at full speed to an union
Union or no union, you are hitting API limits. The union is just a red herring here and not the cause.
Again, you shared a command and no log so not sure what you'd like me to review.
No reason for this either as it's only for uploading to Google Drive.
2021/11/21 12:56:24 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "copy" "respaldo:/encriptado/qbuksgtaandhbra3mv61ntsqeg" "prueba:" "--progress" "-vv" "--log-file=mylogfile.txt"]
2021/11/21 12:56:24 DEBUG : Creating backend with remote "respaldo:/encriptado/qbuksgtaandhbra3mv61ntsqeg"
2021/11/21 12:56:24 DEBUG : Using config file from "D:\\googledrive\\rclone.conf"
2021/11/21 12:56:24 DEBUG : respaldo: Loaded invalid token from config file - ignoring
2021/11/21 12:56:24 DEBUG : Saving config "token" in section "respaldo" of the config file
2021/11/21 12:56:24 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2021/11/21 12:56:24 DEBUG : respaldo: Saved new token in config file
2021/11/21 12:56:24 DEBUG : Google drive root 'encriptado/qbuksgtaandhbra3mv61ntsqeg': 'root_folder_id = 0AIxxqvVOU8FUUk9PVA' - save this in the config to speed up startup
2021/11/21 12:56:25 DEBUG : fs cache: renaming cache item "respaldo:/encriptado/qbuksgtaandhbra3mv61ntsqeg" to be canonical "respaldo:encriptado/qbuksgtaandhbra3mv61ntsqeg"
2021/11/21 12:56:25 DEBUG : Creating backend with remote "prueba:"
2021/11/21 12:56:25 DEBUG : Creating backend with remote "D:/prueba1"
2021/11/21 12:56:25 DEBUG : Creating backend with remote "D:/prueba"
2021/11/21 12:56:25 DEBUG : fs cache: renaming cache item "D:/prueba1" to be canonical "//?/D:/prueba1"
2021/11/21 12:56:25 DEBUG : fs cache: switching user supplied name "D:/prueba1" for canonical name "//?/D:/prueba1"
2021/11/21 12:56:25 DEBUG : fs cache: renaming cache item "D:/prueba" to be canonical "//?/D:/prueba"
2021/11/21 12:56:25 DEBUG : fs cache: switching user supplied name "D:/prueba" for canonical name "//?/D:/prueba"
2021/11/21 12:56:25 DEBUG : union root '': actionPolicy = *policy.EpAll, createPolicy = *policy.EpMfs, searchPolicy = *policy.FF
2021/11/21 12:56:36 DEBUG : pacer: Reducing sleep to 16.08ms
2021/11/21 12:56:36 DEBUG : pacer: Reducing sleep to 62.7681ms
2021/11/21 12:56:36 DEBUG : pacer: Reducing sleep to 144.5868ms
2021/11/21 12:56:36 DEBUG : pacer: Reducing sleep to 180.0657ms
2021/11/21 12:56:36 DEBUG : pacer: Reducing sleep to 275.1074ms
2021/11/21 12:56:36 DEBUG : pacer: Reducing sleep to 363.0103ms
2021/11/21 12:56:36 DEBUG : pacer: Reducing sleep to 369.1163ms
2021/11/21 12:56:36 DEBUG : pacer: Reducing sleep to 378.312ms
2021/11/21 12:56:36 DEBUG : pacer: Reducing sleep to 469.5081ms
2021/11/21 12:56:36 DEBUG : pacer: Reducing sleep to 561.7128ms
2021/11/21 12:56:36 DEBUG : pacer: Reducing sleep to 610.4801ms
2021/11/21 12:56:36 DEBUG : pacer: Reducing sleep to 549.7838ms
2021/11/21 12:56:36 DEBUG : pacer: Reducing sleep to 238.1207ms
2021/11/21 12:56:37 DEBUG : pacer: Reducing sleep to 0s
2021/11/21 12:56:38 DEBUG : pacer: Reducing sleep to 74.7847ms
2021/11/21 12:56:38 DEBUG : pacer: Reducing sleep to 0s
2021/11/21 12:56:38 DEBUG : pacer: Reducing sleep to 88.8307ms
2021/11/21 12:56:38 DEBUG : pacer: Reducing sleep to 181.7041ms
2021/11/21 12:56:38 DEBUG : pacer: Reducing sleep to 274.8797ms
2021/11/21 12:56:38 DEBUG : pacer: Reducing sleep to 372.9256ms
2021/11/21 12:56:38 DEBUG : pacer: Reducing sleep to 468.7432ms
2021/11/21 12:56:38 DEBUG : pacer: Reducing sleep to 518.9063ms
2021/11/21 12:56:38 DEBUG : pacer: Reducing sleep to 428.3066ms
2021/11/21 12:56:38 DEBUG : pacer: Reducing sleep to 517.3673ms
2021/11/21 12:56:38 DEBUG : pacer: Reducing sleep to 608.2348ms
2021/11/21 12:56:38 DEBUG : pacer: Reducing sleep to 586.014ms
2021/11/21 12:56:39 DEBUG : pacer: Reducing sleep to 133.2234ms
2021/11/21 12:56:40 DEBUG : pacer: Reducing sleep to 0s
2021/11/21 12:56:40 DEBUG : pacer: Reducing sleep to 36.9435ms
2021/11/21 12:56:40 DEBUG : pacer: Reducing sleep to 112.8359ms
2021/11/21 12:56:40 DEBUG : pacer: Reducing sleep to 30.2926ms
2021/11/21 12:56:40 DEBUG : pacer: Reducing sleep to 123.5733ms
2021/11/21 12:56:40 DEBUG : pacer: Reducing sleep to 186.097ms
2021/11/21 12:56:40 DEBUG : pacer: Reducing sleep to 250.7034ms
2021/11/21 12:56:40 DEBUG : pacer: Reducing sleep to 349.7237ms
2021/11/21 12:56:40 DEBUG : pacer: Reducing sleep to 446.7942ms
2021/11/21 12:56:40 DEBUG : pacer: Reducing sleep to 534.1056ms
2021/11/21 12:56:40 DEBUG : pacer: Reducing sleep to 577.4126ms
2021/11/21 12:56:40 DEBUG : pacer: Reducing sleep to 562.9549ms
2021/11/21 12:56:40 DEBUG : pacer: Reducing sleep to 626.9644ms
2021/11/21 12:56:41 DEBUG : pacer: Reducing sleep to 535.9867ms
2021/11/21 12:56:41 DEBUG : pacer: Reducing sleep to 106.1626ms
2021/11/21 12:56:42 DEBUG : pacer: Reducing sleep to 0s
2021/11/21 12:56:42 DEBUG : pacer: Reducing sleep to 14.6633ms
2021/11/21 12:56:42 DEBUG : pacer: Reducing sleep to 110.2448ms
2021/11/21 12:56:42 DEBUG : pacer: Reducing sleep to 0s
2021/11/21 12:56:42 DEBUG : pacer: Reducing sleep to 73.3174ms
2021/11/21 12:56:42 DEBUG : pacer: Reducing sleep to 156.703ms
2021/11/21 12:56:42 DEBUG : pacer: Reducing sleep to 256.1658ms
2021/11/21 12:56:42 DEBUG : pacer: Reducing sleep to 356.1658ms
2021/11/21 12:56:42 DEBUG : pacer: Reducing sleep to 422.2672ms
2021/11/21 12:56:42 DEBUG : pacer: Reducing sleep to 500.9898ms
2021/11/21 12:56:42 DEBUG : pacer: Reducing sleep to 592.2176ms
2021/11/21 12:56:43 DEBUG : pacer: Reducing sleep to 467.4294ms
2021/11/21 12:56:43 DEBUG : pacer: Reducing sleep to 508.6469ms
rclone copy respaldo:/encriptado/qbuksgtaandhbra3mv61ntsqeg prueba: --progress -vv
And again, you are getting rate limited.
I can't figure out what you are trying to do as you seem to be copying from drive to a union which is the same drive?
What are you trying to accomplish?
I have around 20 TB on google drive, I want to download it to my disk, I have 5 disk with adds up to 24 TB, if I download it to an union of the disk it downloads at 48 Mbytes/s max, If I download to a only disk from the same google drive remote I get 82 Mbytes/s.
I dont get why do Im getting rate limited when I download to an union and not when I download to a single disk, even if the union its two folders on that single disk.
Its the same command, the same disk, only change its the union
There isn't an easy way to tell as you'd have to go some detective work on your API in your Quota console to see what's going on.
Are other people using it?
Those are both local disks mounted as folders?
And pretty please, share a full debug log as you keep sharing snippets so I only see part of the log as you stopped it before anything actually happened after the rate limits.
Those are two folders on a local drive, I mounted the two folders like an union to make a test.
I stopped it because it has too many characters and I cant post it here
Use pastebin/github/google drive link/any way you want to share a link to a file.
I have some hunches but I hate guessing and a full log file makes it so much easier.
Can you give me the command for a full log?
Add --log-file C:\rclone.log on the end or wherever you want to keep the log.
Let the whole thing finish and upload/share the complete log file.
It does not says anything else, just
2021/11/21 17:57:40 DEBUG : pacer: Reducing sleep to 582.5397ms
2021/11/21 17:57:40 DEBUG : pacer: Reducing sleep to 409.0654ms
2021/11/21 17:57:41 DEBUG : pacer: Reducing sleep to 342.7063ms
2021/11/21 17:57:41 DEBUG : pacer: Reducing sleep to 0s
2021/11/21 17:57:42 DEBUG : pacer: Reducing sleep to 336.1µs
2021/11/21 17:57:42 DEBUG : pacer: Reducing sleep to 90.3965ms
2021/11/21 17:57:42 DEBUG : pacer: Reducing sleep to 172.8298ms
2021/11/21 17:57:42 DEBUG : pacer: Reducing sleep to 48.9685ms
2021/11/21 17:57:42 DEBUG : pacer: Reducing sleep to 123.7927ms
2021/11/21 17:57:42 DEBUG : pacer: Reducing sleep to 223.7927ms
2021/11/21 17:57:42 DEBUG : pacer: Reducing sleep to 320.83ms
2021/11/21 17:57:42 DEBUG : pacer: Reducing sleep to 415.4855ms
2021/11/21 17:57:42 DEBUG : pacer: Reducing sleep to 499.029ms
2021/11/21 17:57:42 DEBUG : pacer: Reducing sleep to 547.1398ms
2021/11/21 17:57:42 DEBUG : pacer: Reducing sleep to 400.3672ms
2021/11/21 17:57:42 DEBUG : pacer: Reducing sleep to 452.8868ms
2021/11/21 17:57:43 DEBUG : pacer: Reducing sleep to 62.3767ms
2021/11/21 17:57:43 DEBUG : pacer: Reducing sleep to 0s
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
But they are 20 Tb