What is the problem you are having with rclone?
I'm trying to move some content (like 10TB) from my personal OneDrive Business (25TB account) over to a SharePoint Online team site share (25TB) as to make space and keep growing. As you know Office 365 for business offers unlimited space, yet they limit you to 25TB and ask you to use SharePoint. I read that moving over 500MB (yes MB) is not possible via web, so thought on using server move via rclone. Yet seems I'm heavily throttled right away, transfers look like they do go through but take forever and it's making this impossible.
What is your rclone version (output from rclone version
)
rclone version
rclone v1.53.1
- os/arch: linux/amd64
- go version: go1.15
Which OS you are using and how many bits (eg Windows 7, 64 bit)
CentOS 7 64bit
Which cloud storage system are you using? (eg Google Drive)
OneDrive Business
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
rclone copy One:Folder1 One2:Folder1 --onedrive-server-side-across-configs --onedrive-no-versions -P
The rclone config contents with secrets removed.
[One]
type = onedrive
client_id = **REDACTED**
client_secret = **REDACTED**
token = **REDACTED**
drive_id = **REDACTED**
drive_type = business
[One2]
type = onedrive
client_id = **REDACTED**
client_secret = **REDACTED**
token = **REDACTED**
drive_id = **REDACTED**
drive_type = documentLibrary
A log from the command with the -vv
flag
# rclone copy One:Folder1 One2:Folder1 --onedrive-server-side-across-configs --onedrive-no-versions -vv
2020/10/01 17:16:40 DEBUG : rclone: Version "v1.53.1" starting with parameters ["rclone" "copy" "One:Folder1" "One2:Folder1" "--onedrive-server-side-across-configs" "--onedrive-no-versions" "-vv"]
2020/10/01 17:16:40 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2020/10/01 17:16:40 DEBUG : Creating backend with remote "One:Folder1"
2020/10/01 17:16:41 DEBUG : Creating backend with remote "One2:Folder1"
2020/10/01 17:16:41 DEBUG : One2: Loaded invalid token from config file - ignoring
2020/10/01 17:16:41 DEBUG : One2: Saved new token in config file
2020/10/01 17:16:42 DEBUG : Too many requests. Trying again in 120 seconds.
2020/10/01 17:16:42 DEBUG : pacer: low level retry 1/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/10/01 17:16:42 DEBUG : pacer: Rate limited, increasing sleep to 2m0s
2020/10/01 17:16:42 DEBUG : Too many requests. Trying again in 120 seconds.
2020/10/01 17:16:42 DEBUG : pacer: low level retry 1/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/10/01 17:16:42 DEBUG : pacer: Reducing sleep to 1m30s
2020/10/01 17:16:42 DEBUG : pacer: Reducing sleep to 1m7.5s
2020/10/01 17:16:42 DEBUG : pacer: Reducing sleep to 50.625s
2020/10/01 17:16:42 DEBUG : pacer: Reducing sleep to 37.96875s
2020/10/01 17:16:44 DEBUG : pacer: Reducing sleep to 28.4765625s
2020/10/01 17:17:41 INFO :
Transferred: 0 / 24.496 GBytes, 0%, 0 Bytes/s, ETA -
Transferred: 0 / 3, 0%
Elapsed time: 1m1.4s
Transferring:
* 157.jASiKREAO/60.sGD.j…clone_chunk.001_mmp64n: 0% /10G, 0/s, -
* 157.jASiKREAO/62.Ngu.O…clone_chunk.001_sbzt4g: 0% /10G, 0/s, -
* 107.iQOEyrEzAKO/173.Pr….k719.NP8.ZHGMaHGM.zxI: 0% /4.496G, 0/s, -
2020/10/01 17:18:41 INFO :
Transferred: 0 / 24.496 GBytes, 0%, 0 Bytes/s, ETA -
Transferred: 0 / 3, 0%
Elapsed time: 2m1.4s
Transferring:
* 157.jASiKREAO/60.sGD.j…clone_chunk.001_mmp64n: 0% /10G, 0/s, -
* 157.jASiKREAO/62.Ngu.O…clone_chunk.001_sbzt4g: 0% /10G, 0/s, -
* 107.iQOEyrEzAKO/173.Pr….k719.NP8.ZHGMaHGM.zxI: 0% /4.496G, 0/s, -
2020/10/01 17:18:42 DEBUG : pacer: Reducing sleep to 21.357421875s
2020/10/01 17:19:12 DEBUG : pacer: Reducing sleep to 16.018066406s
2020/10/01 17:19:34 DEBUG : pacer: Reducing sleep to 12.013549804s
2020/10/01 17:19:37 INFO : 224.dFsFCzz/80.jLyLIFF3.GJ6: removing version "1.0"
2020/10/01 17:19:37 DEBUG : 224.dFsFCzz/80.jLyLIFF3.GJ6: QuickXorHash = 316e4b650651b1921770874653c192c21a44c041 OK
2020/10/01 17:19:37 INFO : 224.dFsFCzz/80.jLyLIFF3.GJ6: Copied (server side copy)
2020/10/01 17:19:41 INFO :
Transferred: 1.317M / 25.055 GBytes, 0%, 7.521 kBytes/s, ETA 5w5d10h16m24s
Transferred: 1 / 5, 20%
Elapsed time: 3m1.4s
Transferring:
* 157.jASiKREAO/60.sGD.j…clone_chunk.001_mmp64n: 0% /10G, 0/s, -
* 157.jASiKREAO/62.Ngu.O…clone_chunk.001_sbzt4g: 0% /10G, 0/s, -
* 107.iQOEyrEzAKO/173.Pr….k719.NP8.ZHGMaHGM.zxI: 0% /4.496G, 0/s, -
* 123.hAEnEZBIvrF/55.nIG…By.guACw.lCHA.5335.uPC: 0% /570.628M, 0/s, -
2020/10/01 17:19:48 DEBUG : pacer: Reducing sleep to 9.010162353s
2020/10/01 17:20:00 DEBUG : pacer: Reducing sleep to 6.757621764s
2020/10/01 17:20:09 DEBUG : pacer: Reducing sleep to 5.068216323s
2020/10/01 17:20:16 DEBUG : pacer: Reducing sleep to 3.801162242s
2020/10/01 17:20:21 DEBUG : pacer: Reducing sleep to 2.850871681s
2020/10/01 17:20:25 DEBUG : pacer: Reducing sleep to 2.13815376s
2020/10/01 17:20:27 DEBUG : pacer: Reducing sleep to 1.60361532s
2020/10/01 17:20:30 DEBUG : pacer: Reducing sleep to 1.20271149s
2020/10/01 17:20:31 DEBUG : pacer: Reducing sleep to 902.033617ms
2020/10/01 17:20:33 DEBUG : pacer: Reducing sleep to 676.525212ms
2020/10/01 17:20:33 DEBUG : pacer: Reducing sleep to 507.393909ms
2020/10/01 17:20:34 DEBUG : pacer: Reducing sleep to 380.545431ms
2020/10/01 17:20:35 DEBUG : pacer: Reducing sleep to 285.409073ms
2020/10/01 17:20:35 DEBUG : pacer: Reducing sleep to 214.056804ms
2020/10/01 17:20:35 DEBUG : pacer: Reducing sleep to 160.542603ms
2020/10/01 17:20:35 DEBUG : pacer: Reducing sleep to 120.406952ms
2020/10/01 17:20:35 DEBUG : pacer: Reducing sleep to 90.305214ms
2020/10/01 17:20:36 DEBUG : pacer: Reducing sleep to 67.72891ms
2020/10/01 17:20:36 DEBUG : pacer: Reducing sleep to 50.796682ms
2020/10/01 17:20:36 DEBUG : pacer: Reducing sleep to 38.097511ms
2020/10/01 17:20:36 DEBUG : pacer: Reducing sleep to 28.573133ms
2020/10/01 17:20:36 DEBUG : pacer: Reducing sleep to 21.429849ms
2020/10/01 17:20:36 DEBUG : pacer: Reducing sleep to 16.072386ms
2020/10/01 17:20:36 DEBUG : pacer: Reducing sleep to 12.054289ms
2020/10/01 17:20:36 DEBUG : pacer: Reducing sleep to 10ms
2020/10/01 17:20:41 INFO :
Transferred: 1.317M / 901.116 GBytes, 0%, 5.635 kBytes/s, ETA 5y16w3d16h2m7s
Transferred: 1 / 3784, 0%
Elapsed time: 4m1.4s
Transferring:
* 157.jASiKREAO/60.sGD.j…clone_chunk.001_mmp64n: 0% /10G, 0/s, -
* 157.jASiKREAO/62.Ngu.O…clone_chunk.001_sbzt4g: 0% /10G, 0/s, -
* 107.iQOEyrEzAKO/173.Pr….k719.NP8.ZHGMaHGM.zxI: 0% /4.496G, 0/s, -
* 123.hAEnEZBIvrF/55.nIG…By.guACw.lCHA.5335.uPC: 0% /570.628M, 0/s, -
2020/10/01 17:20:47 DEBUG : pacer: low level retry 1/10 (error serviceNotAvailable: Service unavailable)
2020/10/01 17:20:47 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/10/01 17:20:47 DEBUG : pacer: Reducing sleep to 15ms
2020/10/01 17:20:47 DEBUG : pacer: Reducing sleep to 11.25ms
2020/10/01 17:20:47 DEBUG : pacer: Reducing sleep to 10ms
2020/10/01 17:20:48 DEBUG : pacer: low level retry 1/10 (error serviceNotAvailable: Service unavailable)
2020/10/01 17:20:48 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/10/01 17:20:48 DEBUG : pacer: Reducing sleep to 15ms
2020/10/01 17:20:48 DEBUG : pacer: Reducing sleep to 11.25ms
2020/10/01 17:20:48 DEBUG : pacer: Reducing sleep to 10ms
2020/10/01 17:20:48 DEBUG : pacer: low level retry 1/10 (error serviceNotAvailable: Service unavailable)
2020/10/01 17:20:48 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/10/01 17:20:48 DEBUG : pacer: Reducing sleep to 15ms
2020/10/01 17:20:48 DEBUG : pacer: Reducing sleep to 11.25ms
2020/10/01 17:20:48 DEBUG : pacer: Reducing sleep to 10ms
2020/10/01 17:20:48 DEBUG : pacer: low level retry 1/10 (error serviceNotAvailable: Service unavailable)
2020/10/01 17:20:48 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/10/01 17:20:49 DEBUG : pacer: Reducing sleep to 15ms
2020/10/01 17:20:49 DEBUG : pacer: Reducing sleep to 11.25ms
2020/10/01 17:20:49 DEBUG : pacer: Reducing sleep to 10ms
2020/10/01 17:20:49 DEBUG : pacer: low level retry 1/10 (error serviceNotAvailable: Service unavailable)
2020/10/01 17:20:49 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/10/01 17:20:49 DEBUG : pacer: Reducing sleep to 15ms
2020/10/01 17:20:49 DEBUG : pacer: Reducing sleep to 11.25ms
2020/10/01 17:20:49 DEBUG : pacer: Reducing sleep to 10ms
2020/10/01 17:20:49 DEBUG : pacer: low level retry 1/10 (error serviceNotAvailable: Service unavailable)
2020/10/01 17:20:49 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/10/01 17:20:49 DEBUG : pacer: Reducing sleep to 15ms
2020/10/01 17:20:49 DEBUG : pacer: Reducing sleep to 11.25ms
2020/10/01 17:20:49 DEBUG : pacer: Reducing sleep to 10ms
2020/10/01 17:20:49 DEBUG : pacer: low level retry 1/10 (error serviceNotAvailable: Service unavailable)
2020/10/01 17:20:49 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/10/01 17:20:49 DEBUG : pacer: Reducing sleep to 15ms
2020/10/01 17:20:49 DEBUG : pacer: Reducing sleep to 11.25ms
2020/10/01 17:20:49 DEBUG : pacer: Reducing sleep to 10ms
2020/10/01 17:20:50 DEBUG : pacer: low level retry 1/10 (error serviceNotAvailable: Service unavailable)
2020/10/01 17:20:50 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/10/01 17:20:50 DEBUG : pacer: Reducing sleep to 15ms
2020/10/01 17:20:50 DEBUG : pacer: Reducing sleep to 11.25ms
2020/10/01 17:20:50 DEBUG : pacer: Reducing sleep to 10ms
2020/10/01 17:20:50 DEBUG : pacer: low level retry 1/10 (error serviceNotAvailable: Service unavailable)
2020/10/01 17:20:50 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/10/01 17:20:50 DEBUG : pacer: Reducing sleep to 15ms
2020/10/01 17:20:50 DEBUG : pacer: Reducing sleep to 11.25ms
2020/10/01 17:20:50 DEBUG : pacer: Reducing sleep to 10ms
2020/10/01 17:20:50 DEBUG : pacer: low level retry 1/10 (error serviceNotAvailable: Service unavailable)
2020/10/01 17:20:50 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/10/01 17:20:50 DEBUG : pacer: Reducing sleep to 15ms
2020/10/01 17:20:50 DEBUG : pacer: Reducing sleep to 11.25ms
2020/10/01 17:20:50 DEBUG : pacer: Reducing sleep to 10ms
2020/10/01 17:20:50 DEBUG : pacer: low level retry 1/10 (error serviceNotAvailable: Service unavailable)
2020/10/01 17:20:50 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/10/01 17:20:50 DEBUG : pacer: Reducing sleep to 15ms
2020/10/01 17:20:50 DEBUG : pacer: Reducing sleep to 11.25ms
2020/10/01 17:20:50 DEBUG : pacer: Reducing sleep to 10ms
2020/10/01 17:20:51 DEBUG : pacer: low level retry 1/10 (error serviceNotAvailable: Service unavailable)
2020/10/01 17:20:51 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/10/01 17:20:51 DEBUG : pacer: Reducing sleep to 15ms
2020/10/01 17:20:51 DEBUG : pacer: Reducing sleep to 11.25ms
2020/10/01 17:20:51 DEBUG : pacer: Reducing sleep to 10ms
2020/10/01 17:20:51 DEBUG : pacer: low level retry 1/10 (error serviceNotAvailable: Service unavailable)
2020/10/01 17:20:51 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/10/01 17:20:51 DEBUG : pacer: Reducing sleep to 15ms
2020/10/01 17:20:51 DEBUG : pacer: Reducing sleep to 11.25ms
2020/10/01 17:20:51 DEBUG : pacer: Reducing sleep to 10ms