Move from OneDrive Business to Sharepoint team site

What is the problem you are having with rclone?

I'm trying to move some content (like 10TB) from my personal OneDrive Business (25TB account) over to a SharePoint Online team site share (25TB) as to make space and keep growing. As you know Office 365 for business offers unlimited space, yet they limit you to 25TB and ask you to use SharePoint. I read that moving over 500MB (yes MB) is not possible via web, so thought on using server move via rclone. Yet seems I'm heavily throttled right away, transfers look like they do go through but take forever and it's making this impossible.

What is your rclone version (output from rclone version)

rclone version

rclone v1.53.1

  • os/arch: linux/amd64
  • go version: go1.15

Which OS you are using and how many bits (eg Windows 7, 64 bit)

CentOS 7 64bit

Which cloud storage system are you using? (eg Google Drive)

OneDrive Business

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy One:Folder1 One2:Folder1 --onedrive-server-side-across-configs --onedrive-no-versions -P

The rclone config contents with secrets removed.

[One]
type = onedrive
client_id = **REDACTED**
client_secret = **REDACTED**
token = **REDACTED**
drive_id = **REDACTED**
drive_type = business

[One2]
type = onedrive
client_id = **REDACTED**
client_secret = **REDACTED**
token = **REDACTED**
drive_id = **REDACTED**
drive_type = documentLibrary

A log from the command with the -vv flag

# rclone copy One:Folder1 One2:Folder1 --onedrive-server-side-across-configs --onedrive-no-versions -vv
2020/10/01 17:16:40 DEBUG : rclone: Version "v1.53.1" starting with parameters ["rclone" "copy" "One:Folder1" "One2:Folder1" "--onedrive-server-side-across-configs" "--onedrive-no-versions" "-vv"]
2020/10/01 17:16:40 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2020/10/01 17:16:40 DEBUG : Creating backend with remote "One:Folder1"
2020/10/01 17:16:41 DEBUG : Creating backend with remote "One2:Folder1"
2020/10/01 17:16:41 DEBUG : One2: Loaded invalid token from config file - ignoring
2020/10/01 17:16:41 DEBUG : One2: Saved new token in config file
2020/10/01 17:16:42 DEBUG : Too many requests. Trying again in 120 seconds.
2020/10/01 17:16:42 DEBUG : pacer: low level retry 1/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/10/01 17:16:42 DEBUG : pacer: Rate limited, increasing sleep to 2m0s
2020/10/01 17:16:42 DEBUG : Too many requests. Trying again in 120 seconds.
2020/10/01 17:16:42 DEBUG : pacer: low level retry 1/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/10/01 17:16:42 DEBUG : pacer: Reducing sleep to 1m30s
2020/10/01 17:16:42 DEBUG : pacer: Reducing sleep to 1m7.5s
2020/10/01 17:16:42 DEBUG : pacer: Reducing sleep to 50.625s
2020/10/01 17:16:42 DEBUG : pacer: Reducing sleep to 37.96875s
2020/10/01 17:16:44 DEBUG : pacer: Reducing sleep to 28.4765625s
2020/10/01 17:17:41 INFO  :
Transferred:             0 / 24.496 GBytes, 0%, 0 Bytes/s, ETA -
Transferred:            0 / 3, 0%
Elapsed time:       1m1.4s
Transferring:
 * 157.jASiKREAO/60.sGD.j…clone_chunk.001_mmp64n:  0% /10G, 0/s, -
 * 157.jASiKREAO/62.Ngu.O…clone_chunk.001_sbzt4g:  0% /10G, 0/s, -
 * 107.iQOEyrEzAKO/173.Pr….k719.NP8.ZHGMaHGM.zxI:  0% /4.496G, 0/s, -

2020/10/01 17:18:41 INFO  :
Transferred:             0 / 24.496 GBytes, 0%, 0 Bytes/s, ETA -
Transferred:            0 / 3, 0%
Elapsed time:       2m1.4s
Transferring:
 * 157.jASiKREAO/60.sGD.j…clone_chunk.001_mmp64n:  0% /10G, 0/s, -
 * 157.jASiKREAO/62.Ngu.O…clone_chunk.001_sbzt4g:  0% /10G, 0/s, -
 * 107.iQOEyrEzAKO/173.Pr….k719.NP8.ZHGMaHGM.zxI:  0% /4.496G, 0/s, -

2020/10/01 17:18:42 DEBUG : pacer: Reducing sleep to 21.357421875s
2020/10/01 17:19:12 DEBUG : pacer: Reducing sleep to 16.018066406s
2020/10/01 17:19:34 DEBUG : pacer: Reducing sleep to 12.013549804s
2020/10/01 17:19:37 INFO  : 224.dFsFCzz/80.jLyLIFF3.GJ6: removing version "1.0"
2020/10/01 17:19:37 DEBUG : 224.dFsFCzz/80.jLyLIFF3.GJ6: QuickXorHash = 316e4b650651b1921770874653c192c21a44c041 OK
2020/10/01 17:19:37 INFO  : 224.dFsFCzz/80.jLyLIFF3.GJ6: Copied (server side copy)
2020/10/01 17:19:41 INFO  :
Transferred:        1.317M / 25.055 GBytes, 0%, 7.521 kBytes/s, ETA 5w5d10h16m24s
Transferred:            1 / 5, 20%
Elapsed time:       3m1.4s
Transferring:
 * 157.jASiKREAO/60.sGD.j…clone_chunk.001_mmp64n:  0% /10G, 0/s, -
 * 157.jASiKREAO/62.Ngu.O…clone_chunk.001_sbzt4g:  0% /10G, 0/s, -
 * 107.iQOEyrEzAKO/173.Pr….k719.NP8.ZHGMaHGM.zxI:  0% /4.496G, 0/s, -
 * 123.hAEnEZBIvrF/55.nIG…By.guACw.lCHA.5335.uPC:  0% /570.628M, 0/s, -

2020/10/01 17:19:48 DEBUG : pacer: Reducing sleep to 9.010162353s
2020/10/01 17:20:00 DEBUG : pacer: Reducing sleep to 6.757621764s
2020/10/01 17:20:09 DEBUG : pacer: Reducing sleep to 5.068216323s
2020/10/01 17:20:16 DEBUG : pacer: Reducing sleep to 3.801162242s
2020/10/01 17:20:21 DEBUG : pacer: Reducing sleep to 2.850871681s
2020/10/01 17:20:25 DEBUG : pacer: Reducing sleep to 2.13815376s
2020/10/01 17:20:27 DEBUG : pacer: Reducing sleep to 1.60361532s
2020/10/01 17:20:30 DEBUG : pacer: Reducing sleep to 1.20271149s
2020/10/01 17:20:31 DEBUG : pacer: Reducing sleep to 902.033617ms
2020/10/01 17:20:33 DEBUG : pacer: Reducing sleep to 676.525212ms
2020/10/01 17:20:33 DEBUG : pacer: Reducing sleep to 507.393909ms
2020/10/01 17:20:34 DEBUG : pacer: Reducing sleep to 380.545431ms
2020/10/01 17:20:35 DEBUG : pacer: Reducing sleep to 285.409073ms
2020/10/01 17:20:35 DEBUG : pacer: Reducing sleep to 214.056804ms
2020/10/01 17:20:35 DEBUG : pacer: Reducing sleep to 160.542603ms
2020/10/01 17:20:35 DEBUG : pacer: Reducing sleep to 120.406952ms
2020/10/01 17:20:35 DEBUG : pacer: Reducing sleep to 90.305214ms
2020/10/01 17:20:36 DEBUG : pacer: Reducing sleep to 67.72891ms
2020/10/01 17:20:36 DEBUG : pacer: Reducing sleep to 50.796682ms
2020/10/01 17:20:36 DEBUG : pacer: Reducing sleep to 38.097511ms
2020/10/01 17:20:36 DEBUG : pacer: Reducing sleep to 28.573133ms
2020/10/01 17:20:36 DEBUG : pacer: Reducing sleep to 21.429849ms
2020/10/01 17:20:36 DEBUG : pacer: Reducing sleep to 16.072386ms
2020/10/01 17:20:36 DEBUG : pacer: Reducing sleep to 12.054289ms
2020/10/01 17:20:36 DEBUG : pacer: Reducing sleep to 10ms
2020/10/01 17:20:41 INFO  :
Transferred:        1.317M / 901.116 GBytes, 0%, 5.635 kBytes/s, ETA 5y16w3d16h2m7s
Transferred:            1 / 3784, 0%
Elapsed time:       4m1.4s
Transferring:
 * 157.jASiKREAO/60.sGD.j…clone_chunk.001_mmp64n:  0% /10G, 0/s, -
 * 157.jASiKREAO/62.Ngu.O…clone_chunk.001_sbzt4g:  0% /10G, 0/s, -
 * 107.iQOEyrEzAKO/173.Pr….k719.NP8.ZHGMaHGM.zxI:  0% /4.496G, 0/s, -
 * 123.hAEnEZBIvrF/55.nIG…By.guACw.lCHA.5335.uPC:  0% /570.628M, 0/s, -

2020/10/01 17:20:47 DEBUG : pacer: low level retry 1/10 (error serviceNotAvailable: Service unavailable)
2020/10/01 17:20:47 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/10/01 17:20:47 DEBUG : pacer: Reducing sleep to 15ms
2020/10/01 17:20:47 DEBUG : pacer: Reducing sleep to 11.25ms
2020/10/01 17:20:47 DEBUG : pacer: Reducing sleep to 10ms
2020/10/01 17:20:48 DEBUG : pacer: low level retry 1/10 (error serviceNotAvailable: Service unavailable)
2020/10/01 17:20:48 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/10/01 17:20:48 DEBUG : pacer: Reducing sleep to 15ms
2020/10/01 17:20:48 DEBUG : pacer: Reducing sleep to 11.25ms
2020/10/01 17:20:48 DEBUG : pacer: Reducing sleep to 10ms
2020/10/01 17:20:48 DEBUG : pacer: low level retry 1/10 (error serviceNotAvailable: Service unavailable)
2020/10/01 17:20:48 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/10/01 17:20:48 DEBUG : pacer: Reducing sleep to 15ms
2020/10/01 17:20:48 DEBUG : pacer: Reducing sleep to 11.25ms
2020/10/01 17:20:48 DEBUG : pacer: Reducing sleep to 10ms
2020/10/01 17:20:48 DEBUG : pacer: low level retry 1/10 (error serviceNotAvailable: Service unavailable)
2020/10/01 17:20:48 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/10/01 17:20:49 DEBUG : pacer: Reducing sleep to 15ms
2020/10/01 17:20:49 DEBUG : pacer: Reducing sleep to 11.25ms
2020/10/01 17:20:49 DEBUG : pacer: Reducing sleep to 10ms
2020/10/01 17:20:49 DEBUG : pacer: low level retry 1/10 (error serviceNotAvailable: Service unavailable)
2020/10/01 17:20:49 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/10/01 17:20:49 DEBUG : pacer: Reducing sleep to 15ms
2020/10/01 17:20:49 DEBUG : pacer: Reducing sleep to 11.25ms
2020/10/01 17:20:49 DEBUG : pacer: Reducing sleep to 10ms
2020/10/01 17:20:49 DEBUG : pacer: low level retry 1/10 (error serviceNotAvailable: Service unavailable)
2020/10/01 17:20:49 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/10/01 17:20:49 DEBUG : pacer: Reducing sleep to 15ms
2020/10/01 17:20:49 DEBUG : pacer: Reducing sleep to 11.25ms
2020/10/01 17:20:49 DEBUG : pacer: Reducing sleep to 10ms
2020/10/01 17:20:49 DEBUG : pacer: low level retry 1/10 (error serviceNotAvailable: Service unavailable)
2020/10/01 17:20:49 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/10/01 17:20:49 DEBUG : pacer: Reducing sleep to 15ms
2020/10/01 17:20:49 DEBUG : pacer: Reducing sleep to 11.25ms
2020/10/01 17:20:49 DEBUG : pacer: Reducing sleep to 10ms
2020/10/01 17:20:50 DEBUG : pacer: low level retry 1/10 (error serviceNotAvailable: Service unavailable)
2020/10/01 17:20:50 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/10/01 17:20:50 DEBUG : pacer: Reducing sleep to 15ms
2020/10/01 17:20:50 DEBUG : pacer: Reducing sleep to 11.25ms
2020/10/01 17:20:50 DEBUG : pacer: Reducing sleep to 10ms
2020/10/01 17:20:50 DEBUG : pacer: low level retry 1/10 (error serviceNotAvailable: Service unavailable)
2020/10/01 17:20:50 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/10/01 17:20:50 DEBUG : pacer: Reducing sleep to 15ms
2020/10/01 17:20:50 DEBUG : pacer: Reducing sleep to 11.25ms
2020/10/01 17:20:50 DEBUG : pacer: Reducing sleep to 10ms
2020/10/01 17:20:50 DEBUG : pacer: low level retry 1/10 (error serviceNotAvailable: Service unavailable)
2020/10/01 17:20:50 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/10/01 17:20:50 DEBUG : pacer: Reducing sleep to 15ms
2020/10/01 17:20:50 DEBUG : pacer: Reducing sleep to 11.25ms
2020/10/01 17:20:50 DEBUG : pacer: Reducing sleep to 10ms
2020/10/01 17:20:50 DEBUG : pacer: low level retry 1/10 (error serviceNotAvailable: Service unavailable)
2020/10/01 17:20:50 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/10/01 17:20:50 DEBUG : pacer: Reducing sleep to 15ms
2020/10/01 17:20:50 DEBUG : pacer: Reducing sleep to 11.25ms
2020/10/01 17:20:50 DEBUG : pacer: Reducing sleep to 10ms
2020/10/01 17:20:51 DEBUG : pacer: low level retry 1/10 (error serviceNotAvailable: Service unavailable)
2020/10/01 17:20:51 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/10/01 17:20:51 DEBUG : pacer: Reducing sleep to 15ms
2020/10/01 17:20:51 DEBUG : pacer: Reducing sleep to 11.25ms
2020/10/01 17:20:51 DEBUG : pacer: Reducing sleep to 10ms
2020/10/01 17:20:51 DEBUG : pacer: low level retry 1/10 (error serviceNotAvailable: Service unavailable)
2020/10/01 17:20:51 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/10/01 17:20:51 DEBUG : pacer: Reducing sleep to 15ms
2020/10/01 17:20:51 DEBUG : pacer: Reducing sleep to 11.25ms
2020/10/01 17:20:51 DEBUG : pacer: Reducing sleep to 10ms

After I ran this command this is what is now in One2:

# rclone ls One2:
  1380899 Folder1/224.dFsFCzz/80.jLyLIFF3.GJ6
598346400 Folder1/123.hAEnEZBIvrF/55.nIG.UHx.dyLLS.nBy.guACw.lCHA.5335.uPC
        0 Folder1/157.jASiKREAO/60.sGD.jHMF.NE.rSzSDM.hRKzMC.1919.0979O.aKTqzX.cc+6.0.W153-Hes.LJU.rclone_chunk.001_mmp64n
        0 Folder1/157.jASiKREAO/62.Ngu.Okugtcdngu.4231.3202r.DnwTcA.FVU.z486-kHV.omx.rclone_chunk.001_sbzt4g
        0 Folder1/107.iQOEyrEzAKO/173.PryvAr.QvBA.N.arJ.QnL.YvIr.vA.YnF.irtnF.7552.6535C.OeevC.k719.NP8.ZHGMaHGM.zxI

micro$oft azure has been having some major problems today and other days this week.

Ha! This would make sense, of course when I decide to migrate, they decide to break everything... I'll try to wait until next week and try again and report. Hopefully it's fixed by then. I really do not want to download and upload again over 10TB of files :frowning:

So other than the outage, does the command look good? Any recommendations to make it better? Some places recommend using the user-agent flag to avoid throttling.

--user-agent "ISV|rclone|rclone/v1.53.1"

Any other ideas or suggestions? Has anybody attempted (and succeeded) doing this?

Thanks!

hi,
i do not use micro$oft cloud, so i cannot help you tweak the settings.

i would create a test folder the represents the mix of file sizes of your data.
and do some testing with the flags.

If it helps anybody seems server move is not supported. Even with that flag, I'm currently downloading and uploading all traffic. Guess my ISP will hate me :roll_eyes:

don't be a tease,
the linked i shared, do you thnk that was the problem. did the problem pass?

from the log
224.dFsFCzz/80.jLyLIFF3.GJ6: Copied (server side copy)

Ok found my problem, I was copying from my Chunker to the new share and kill two birds with one stone (unchunk now that OneDrive supports 100GB files and move the files) , guess this is not supported or there is some other issues... @ncw could you confirm?

The new command was:

# rclone copy COne:Folder1 One2:Folder1 --onedrive-server-side-across-configs --onedrive-no-versions -vv
2020/10/03 15:57:10 DEBUG : rclone: Version "v1.53.1" starting with parameters ["rclone" "copy" "COne:Folder1" "One2:Folder1" "--onedrive-server-side-across-configs" "--onedrive-no-versions" "-vv"]
2020/10/03 15:57:10 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2020/10/03 15:57:10 DEBUG : Creating backend with remote "COne:Folder1"
2020/10/03 15:57:10 DEBUG : Creating backend with remote "One:Folder1"
2020/10/03 15:57:11 DEBUG : Creating backend with remote "One2:Folder1"
2020/10/03 15:57:12 DEBUG : 224.dFsFCzz/80.jLyLIFF3.GJ6: Starting multipart upload
2020/10/03 15:57:13 DEBUG : 42.UlD/140.Xujfwx.Rzsxnq.Gjshmrfwp.0897.ZMI.MdGWNI.ItqgD.anxnts.MIW98+0948u.GqzWfD.WjrzC.YwzjMI.3.9-XU2P/118.axmizA.UCvAqt.Jmvkpuizs PLZ54    usD.usD: Starting multipart upload
2020/10/03 15:57:13 DEBUG : 42.UlD/140.Xujfwx.Rzsxnq.Gjshmrfwp.0897.ZMI.MdGWNI.ItqgD.anxnts.MIW98+0948u.GqzWfD.WjrzC.YwzjMI.3.9-XU2P/101.iFuqHI.cKDIyB.RuDsxCqHA.6.555st.C7JI: Starting multipart upload
2020/10/03 15:57:13 DEBUG : 42.UlD/140.Xujfwx.Rzsxnq.Gjshmrfwp.0897.ZMI.MdGWNI.ItqgD.anxnts.MIW98+0948u.GqzWfD.WjrzC.YwzjMI.3.9-XU2P/102.jGvrIJ.dLEJzC.SvEtyDrIB.8.666tu.D8KJ: Starting multipart upload

Where config is:

[One]
type = onedrive
client_id = **REDACTED**
client_secret = **REDACTED**
token = **REDACTED**
drive_id = **REDACTED**
drive_type = business

[One2]
type = onedrive
client_id = **REDACTED**
client_secret = **REDACTED**
token = **REDACTED**
drive_id = **REDACTED**
drive_type = documentLibrary

[COne]
type = chunker
remote = One:
chunk_size = 10G
hash_type = md5

Using original command is not sending error, just not moving anything:

# rclone copy One:Folder1 One2:Folder1 --onedrive-server-side-across-configs --onedrive-no-versions -vv
2020/10/03 17:10:30 DEBUG : rclone: Version "v1.53.1" starting with parameters ["rclone" "copy" "One:Folder1" "One2:Folder1" "--onedrive-server-side-across-configs" "--onedrive-no-versions" "-vv"]
2020/10/03 17:10:30 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2020/10/03 17:10:30 DEBUG : Creating backend with remote "One:Folder1"
2020/10/03 17:10:31 DEBUG : Creating backend with remote "One2:Folder1"
2020/10/03 17:10:33 ERROR : 157.jASiKREAO/203.UzsJDAwK.SFywDK.0897.9868H.TDMjsQ.P042-VjgfWk/~tmp9D_19.lqjAurnB.jwpnuB.6453.5424y.kuDAjH.G608-mAxwnB.A88.rclone_chunk.001_h32r11: Failed to copy: invalidRequest: Invalid request
2020/10/03 17:10:37 INFO  : 224.dFsFCzz/80.jLyLIFF3.GJ6: removing version "1.0"
2020/10/03 17:10:37 DEBUG : 224.dFsFCzz/80.jLyLIFF3.GJ6: QuickXorHash = 316e4b650651b1921770874653c192c21a44c041 OK
2020/10/03 17:10:37 INFO  : 224.dFsFCzz/80.jLyLIFF3.GJ6: Copied (server side copy)
2020/10/03 17:10:41 INFO  : 157.jASiKREAO/126.TuHHyuHu.dEI.UsHqDI.Tu.VKCuu.5353.ckbjy.4313F.mUR.X597-d6jVb4n/204.wxKKBxKx.GHL.xvKtGL.wx.yNFxx.1919.FNEMB.0979I.Pxu.A153-G2MyE0Q.K66.rclone_chunk.001_0ph4cv: removing version "1.0"
2020/10/03 17:10:42 DEBUG : 157.jASiKREAO/126.TuHHyuHu.dEI.UsHqDI.Tu.VKCuu.5353.ckbjy.4313F.mUR.X597-d6jVb4n/204.wxKKBxKx.GHL.xvKtGL.wx.yNFxx.1919.FNEMB.0979I.Pxu.A153-G2MyE0Q.K66.rclone_chunk.001_0ph4cv: QuickXorHash = 573de685e2379ded059d62adbeab1f01327df4dc OK
2020/10/03 17:10:42 INFO  : 157.jASiKREAO/126.TuHHyuHu.dEI.UsHqDI.Tu.VKCuu.5353.ckbjy.4313F.mUR.X597-d6jVb4n/204.wxKKBxKx.GHL.xvKtGL.wx.yNFxx.1919.FNEMB.0979I.Pxu.A153-G2MyE0Q.K66.rclone_chunk.001_0ph4cv: Copied (server side copy)
2020/10/03 17:11:31 INFO  :
Transferred:       49.012M / 2.011 TBytes, 0%, 841.556 kBytes/s, ETA 4w1d16h47m10s
Errors:                 1 (retrying may help)
Transferred:            2 / 10014, 0%
Elapsed time:       1m0.9s
Transferring:
 * 157.jASiKREAO/60.sGD.j…clone_chunk.001_mmp64n:  0% /10G, 0/s, -
 * 157.jASiKREAO/62.Ngu.O…clone_chunk.001_sbzt4g:  0% /10G, 0/s, -
 * 157.jASiKREAO/29.kHzNx…clone_chunk.001_mmp8gx:  0% /10G, 0/s, -
 * 42.UlD/1.cKIys/135.lzM… CDK lzQ 053O 5346.LJU:  0% /4.826G, 0/s, -

2020/10/03 17:12:31 INFO  :
Transferred:       49.012M / 2.011 TBytes, 0%, 419.504 kBytes/s, ETA 8w3d13h54m7s
Errors:                 1 (retrying may help)
Transferred:            2 / 10014, 0%
Elapsed time:       2m0.9s
Transferring:
 * 157.jASiKREAO/60.sGD.j…clone_chunk.001_mmp64n:  0% /10G, 0/s, -
 * 157.jASiKREAO/62.Ngu.O…clone_chunk.001_sbzt4g:  0% /10G, 0/s, -
 * 157.jASiKREAO/29.kHzNx…clone_chunk.001_mmp8gx:  0% /10G, 0/s, -
 * 42.UlD/1.cKIys/135.lzM… CDK lzQ 053O 5346.LJU:  0% /4.826G, 0/s, -

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.