Need to copy 40k large files (3-4gb each) from Google Drive to OneDrive.
But the copy speed is ridiculously slow on 1 Gbps port, getting only 20-30 Mbps up and down.
rclone copy gdrive: onedrive: --fast-list --checkers=16 --transfers=12 --tpslimit=3 --onedrive-chunk-size=250M -v
What could be done here to fully utilize 1 Gbps?
From my limited testing with Onedrive, i found it to be kind of similar in performance as Gdrive. Slightly higher transfers quota and good bandwidth.
What happens if you run this with completely standard settings? 12 transfers is probably far too much to be effective Onedrive, and tpslimit 3 is very restrictive. Why set it so low?
EDIT: Also - have you set up your own clientID for Onedrive? This could certainly affect performance. I have no idea what the shared common rclone quota is for Onedrive is - ie. if it's ample or not. Would have to ask Nick about that.
EDIT2: I did a quick test for you and I can replicate your issue. I am having a hard time breaking 20MB/sec no matter the amount of transfers. I can't recall noting the bandwidth was that poor, but when I think more about it I think I tested connection-limits more than bandwidth, so I might just be mistaken.
Otherwise it is possible that this is something that has just changed - or it could concievably be high load due to Corona-related thing. I have heard that Google had some minor issues in some systems (including Gdrive) from a high spike in traffic recently. I guess a lot of people suddenly had a lot of time to sit on their computers and do some cleanup ...
So I am not sure about how to "fix" this - except that you probably want to keep an eye on it and see if it varies throughout the day-cycle and in the comming days (in case it is due to high load on the systems).
I am seeing some unusually poor results out of Gdrive as well right now... I suspect some of this may be related. I have made a thread in case this looks to be some persistent or escalating issue:
Yes. I have my own client for google drive and onedrive both.
I have large files so tpslimit=3 won't affect data transfers.
Downloading files from google drive to local is vert fast, I can fully utilize 1 Gbps. When uploading to onedrive from local I can also utilize full 1 Gbps. The problem occurs when directly copying google drive to onedrive.
Sure it does, as you are limiting to 3 transactions per second and you have 16 checkers and 12 transfers. You should remove the tpslimit and just work with the defaults. By upping those things, it is going to make things slower.
You'd need to include a full debug log to see what's going on.
Now I did:
rclone copy sdrive: od1:Sam --fast-list --checkers=10 --transfers=6 --tpslimit=8 --onedrive-chunk-size=250M -v
Speed is now 550 Mbps upload & download simultaneously, which is an improvement; but still not utilizing full 1 Gbps.
You'd want to try the defaults as you are still limiting it back to 8 instead of letting it default at 10. A full debug log would be helpful as noted above.
You mean -vv log.? If yes, there is nothing helpful.
I don't know if there anything helpful as I haven't seen it. If you don't want to post it, just say that as it's hard to debug things and see what's going on without logs.
./rclone-beta copy sdrive: od1: --config=rclone.conf --fast-list --checkers=12 --transfers=8 --onedrive-chunk-size=250M --stats=0 -vv
log
2020/03/26 18:37:44 DEBUG : A300FUXXU1CPL1_A300FUVFR1CQA2_VDF.zip: Uploading segment 1310720000/1735061696 size 262144000
2020/03/26 18:38:00 DEBUG : A300FUXXU1CPL1_A300FUVFR1CQA2_VDH.zip: Uploading segment 1310720000/1735061696 size 262144000
2020/03/26 18:38:10 DEBUG : A300FUXXU1CPL1_A300FUVFR1CQA2_VDI.zip: Uploading segment 1310720000/1735061696 size 262144000
2020/03/26 18:38:26 DEBUG : A300FUXXU1CPL1_A300FUVFR1CQA2_VD2.zip: Uploading segment 1310720000/1735061696 size 262144000
2020/03/26 18:38:48 DEBUG : A300FUXXU1CPL1_A300FUVFR1CQA2_VGR.zip: Uploading segment 262144000/1735061696 size 262144000
2020/03/26 18:39:12 DEBUG : Couldn't decode error response: invalid character '<' looking for beginning of value
2020/03/26 18:39:12 DEBUG : pacer: low level retry 1/10 (error 503 Service Unavailable: )
2020/03/26 18:39:12 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/03/26 18:39:17 DEBUG : pacer: Reducing sleep to 15ms
2020/03/26 18:39:17 DEBUG : A300FUXXU1CPL1_A300FUVFR1CQA2_OMN.zip: Uploading segment 1572864000/1735061696 size 162197696
2020/03/26 18:39:22 DEBUG : pacer: Reducing sleep to 11.25ms
2020/03/26 18:39:22 DEBUG : A300FUXXU1CPL1_A300FUVFR1CQA2_ATL.zip: Uploading segment 1572864000/1735061696 size 162197696
2020/03/26 18:39:23 DEBUG : Couldn't decode error response: invalid character '<' looking for beginning of value
2020/03/26 18:39:23 DEBUG : pacer: low level retry 1/10 (error 503 Service Unavailable: )
2020/03/26 18:39:23 DEBUG : pacer: Rate limited, increasing sleep to 22.5ms
2020/03/26 18:40:24 DEBUG : pacer: Reducing sleep to 16.875ms
2020/03/26 18:40:24 DEBUG : A300FUXXU1CPL1_A300FUVFR1CQA2_VDF.zip: Uploading segment 1572864000/1735061696 size 162197696
2020/03/26 18:40:47 DEBUG : pacer: Reducing sleep to 12.65625ms
2020/03/26 18:40:47 DEBUG : A300FUXXU1CPL1_A300FUVFR1CQA2_VDH.zip: Uploading segment 1572864000/1735061696 size 162197696
2020/03/26 18:40:56 DEBUG : pacer: Reducing sleep to 10ms
2020/03/26 18:40:56 DEBUG : A300FUXXU1CPL1_A300FUVFR1CQA2_VDI.zip: Uploading segment 1572864000/1735061696 size 162197696
2020/03/26 18:41:04 INFO : A300FUXXU1CPL1_A300FUVFR1CQA2_OMN.zip: Copied (new)
2020/03/26 18:41:04 DEBUG : A300FUXXU1CPL1_A300FUVFR1CQA2_VOD.zip: Starting multipart upload
2020/03/26 18:41:06 DEBUG : A300FUXXU1CPL1_A300FUVFR1CQA2_VOD.zip: Uploading segment 0/1735061696 size 262144000
2020/03/26 18:41:07 INFO : A300FUXXU1CPL1_A300FUVFR1CQA2_ATL.zip: Copied (new)
2020/03/26 18:41:08 DEBUG : A300FUXXU1CPL2_A300FUITV1CPI1_ITV.zip: Starting multipart upload
2020/03/26 18:41:09 DEBUG : A300FUXXU1CPL2_A300FUITV1CPI1_ITV.zip: Uploading segment 0/1588312207 size 262144000
2020/03/26 18:41:20 DEBUG : A300FUXXU1CPL1_A300FUVFR1CQA2_VD2.zip: Uploading segment 1572864000/1735061696 size 162197696
2020/03/26 18:41:55 DEBUG : A300FUXXU1CPL1_A300FUVFR1CQA2_TCL.zip: Uploading segment 1572864000/1735061696 size 162197696
2020/03/26 18:42:06 DEBUG : One drive root 'SamsungDrive': Token expired - 8 uploads in progress - refreshing
2020/03/26 18:42:06 DEBUG : od1: Loaded invalid token from config file - ignoring
2020/03/26 18:42:07 DEBUG : Keeping previous permissions for config file: -rw-rw-r--
2020/03/26 18:42:07 DEBUG : od1: Saved new token in config file
2020/03/26 18:42:08 DEBUG : One drive root 'SamsungDrive': Token refresh successful
2020/03/26 18:42:31 INFO : A300FUXXU1CPL1_A300FUVFR1CQA2_VDF.zip: Copied (new)
2020/03/26 18:42:32 DEBUG : A300FUXXU1CPL4_A300FUAMO1CPJ1_AMO.zip: Starting multipart upload
2020/03/26 18:42:33 DEBUG : A300FUXXU1CPL4_A300FUAMO1CPJ1_AMO.zip: Uploading segment 0/1533682313 size 262144000
2020/03/26 18:42:51 DEBUG : A300FUXXU1CPL1_A300FUVFR1CQA2_VGR.zip: Uploading segment 524288000/1735061696 size 262144000
2020/03/26 18:43:23 INFO : A300FUXXU1CPL1_A300FUVFR1CQA2_VDH.zip: Copied (new)
2020/03/26 18:43:24 DEBUG : A300FUXXU1CPL4_A300FUFTM1CPJ1_FTM.zip: Starting multipart upload
2020/03/26 18:43:25 DEBUG : A300FUXXU1CPL4_A300FUFTM1CPJ1_FTM.zip: Uploading segment 0/1703663803 size 262144000
2020/03/26 18:43:28 INFO : A300FUXXU1CPL1_A300FUVFR1CQA2_VDI.zip: Copied (new)
2020/03/26 18:43:29 DEBUG : A300FUXXU1CPL4_A300FUIDE1CPJ1_IDE.zip: Starting multipart upload
2020/03/26 18:43:30 DEBUG : A300FUXXU1CPL4_A300FUIDE1CPJ1_IDE.zip: Uploading segment 0/1533708642 size 262144000
2020/03/26 18:44:00 INFO : A300FUXXU1CPL1_A300FUVFR1CQA2_VD2.zip: Copied (new)
2020/03/26 18:44:01 DEBUG : A300FUXXU1CPL4_A300FUORO1CPJ1_ORO.zip: Starting multipart upload
2020/03/26 18:44:02 DEBUG : A300FUXXU1CPL4_A300FUORO1CPJ1_ORO.zip: Uploading segment 0/1555322506 size 262144000
I only get 100-150 Mbps on 1 Gbps port.
But when I do Google Drive to Dropbox I get full 1 Gbps. So, OneDrive is the culprit.
What could be done here?
hi,
i noticed that there is no space character between the od1: and --config=rclone.conf
while editing i missed it. the actual input was correct.
It's pretty relevant that you are seeing 503s from the provider causing you to slow down. You can always open a support call with them as well.
Okay, but that does not mean slow upload speed. does it?
2020/03/26 19:30:11 DEBUG : Couldn't decode error response: invalid character '<' looking for beginning of value
What does it mean?
That's from the 503 error you got:
2020/03/26 18:39:12 DEBUG : Couldn't decode error response: invalid character '<' looking for beginning of value
2020/03/26 18:39:12 DEBUG : pacer: low level retry 1/10 (error 503 Service Unavailable: )
2020/03/26 18:39:12 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/03/26 18:39:17 DEBUG : pacer: Reducing sleep to 15ms
Yes, it slows down because the service not available thus making upload speed slower since the service isn't responding..
That isn't a wildly helpful error is it! Rclone is trying to decode an HTML response as a JSON blob (which is what it is expecting).
Rclone falls back to giving you the HTTP status 503 Service Unavailable
It might be worth adding a --tpslimit to try slowing down the transactions a bit so you don't trip the 503 errors.
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.