Extermly slow speeds from Google Drive to OneDrive (with ClientID's)

What is the problem you are having with rclone?

Copying from Google Workspace to OneDrive business is extremely slow after the first attempt.

  • Client ID has been set up for Google Drive and OneDrive
  • Tried using the rclone copy command (what I'm calling simple command) and with various flags that others have used in the forum (test and tried a few of them, and ended up with what I'm calling the complex command)
  • The first attempt seemed reasonably fast (IIRC ~ 10mb), but now it is super slow (rating from 50 byes to 1-5kb).
  • Overall I am attempting to copy ~180gb of data from Google Drive to OneDrive
  • To be honest, I am not exactly sure how to check throttling

Run the command 'rclone version' and share the full output of the command.

rclone v1.60.1

  • os/version: ubuntu 22.04 (64 bit)
  • os/kernel: 5.15.0-53-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.19.3
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Source: Google Drive (Workspace)
Destination: OneDrive (M365 E3)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

simple command

rclone copy jf-gdrive: jf-onedrive:GoogleDriveJF --progress --log-file 'simple1' --log-level DEBUG

complex command

rclone copy jf-gdrive: jf-onedrive:GoogleDriveJF -vv --drive-pacer-min-sleep=10ms --checkers=16 --drive-skip-shortcuts --ignore-existing --drive-chunk-size 256M -vv

The rclone config contents with secrets removed.

[jf-gdrive]
type = drive
client_id = //removed
client_secret = //removed
scope = drive
token = //removed
team_drive =
root_folder_id = 0AO4SmDrVvmYiUk9PVA

[jf-onedrive]
type = onedrive
token = //removed
drive_id = //removed
drive_type = business
client_id = //removed
client_secret = //removed

A log from the command with the -vv flag

Simple Command Log gist

First few lines of log:

2022/11/26 11:46:14 DEBUG : rclone: Version "v1.60.1" starting with parameters ["rclone" "copy" "jf-gdrive:" "jf-onedrive:GoogleDriveJF" "--progress" "--log-file" "simple1" "--log-level" "DEBUG"]
2022/11/26 11:46:14 DEBUG : Creating backend with remote "jf-gdrive:"
2022/11/26 11:46:14 DEBUG : Using config file from "/home/jeffrey/.config/rclone/rclone.conf"
2022/11/26 11:46:14 DEBUG : Creating backend with remote "jf-onedrive:GoogleDriveJF"
2022/11/26 11:46:16 NOTICE: Dangling shortcut "Smart Insights" detected
2022/11/26 11:46:16 DEBUG : Contact Information: No export formats found for "application/vnd.google-apps.form"
2022/11/26 11:46:16 NOTICE: Creativus Design - IT Scope Document updated.docx: Duplicate object found in source - ignoring
2022/11/26 11:46:16 DEBUG : CASTANET - Jeffrey Fazal - My network.xlsx: Size and modification time the same (differ by -804ms, within tolerance 1s)
2022/11/26 11:46:16 DEBUG : CASTANET - Jeffrey Fazal - My network.xlsx: Unchanged skipping
2022/11/26 11:46:16 DEBUG : CASTANET - Jeffrey Fazal - NSW Australian Marketing Institue.xlsx: Size and modification time the same (differ by -179ms, within tolerance 1s)
2022/11/26 11:46:16 DEBUG : CASTANET - Jeffrey Fazal - NSW Australian Marketing Institue.xlsx: Unchanged skipping

Complex Log Gist Link

First few lines of completed list

2022/11/26 11:58:17 DEBUG : rclone: Version "v1.60.1" starting with parameters ["rclone" "copy" "jf-gdrive:" "jf-onedrive:GoogleDriveJF" "--progress" "--log-file" "complex1" "--log-level" "DEBUG" "--drive-pacer-min-sleep=10ms" "--checkers=16" "--drive-skip-shortcuts" "--ignore-existing" "--drive-chunk-size" "256M"]
2022/11/26 11:58:17 DEBUG : Creating backend with remote "jf-gdrive:"
2022/11/26 11:58:17 DEBUG : Using config file from "/home/jeffrey/.config/rclone/rclone.conf"
2022/11/26 11:58:17 DEBUG : jf-gdrive: detected overridden config - adding "{a0HWK}" suffix to name
2022/11/26 11:58:17 DEBUG : fs cache: renaming cache item "jf-gdrive:" to be canonical "jf-gdrive{a0HWK}:"
2022/11/26 11:58:17 DEBUG : Creating backend with remote "jf-onedrive:GoogleDriveJF"
2022/11/26 11:58:19 DEBUG : Contact Information: No export formats found for "application/vnd.google-apps.form"
2022/11/26 11:58:19 NOTICE: Creativus Design - IT Scope Document updated.docx: Duplicate object found in source - ignoring
2022/11/26 11:58:19 DEBUG : CASTANET - Jeffrey Fazal - My network.xlsx: Destination exists, skipping
2022/11/26 11:58:19 DEBUG : CASTANET - Jeffrey Fazal - NSW Australian Marketing Institue.xlsx: Destination exists, skipping
2022/11/26 11:58:19 DEBUG : CASTANET - Jeffrey Fazal - Resturant Owners ACT.xlsx: Destination exists, skipping

What's your internet provider speed? Moving from one cloud provider to another means you need to download and reupload every single file.

Onedrive has multitude of weird and undocumented throttles, so you might also be a victim of those. Nothing to do other than keep trying until they raise the quota of your particular account.

Also, my understanding is Onedrive is limited to 25TB, having to ask for a personal sharepoint drive for additional storage limited to 25TB as well, not sure if that's still the case.

Checking your logs, seems you are moving lot of small files, this usually takes longer as it takes a couple of seconds to load each transfer, you might have seen better speeds/performance while transferring bigger files. I see you also hace some pacer, but they look pretty mild and nothing too crazy I would worry about, seems it's only slow due to the sheer amount of small files being transferred and not much you can do about that.

My internet speed is 100/40.

I wouldn't be suprised if it was some sort of throttle, since everything was working fine initally.

OneDrive is now 1TB, though there should be a way to change it. In anycase I didn't consider disk quota simply because my OneDrive has plenty of diskspace.

Yes there is a lot of small file since it's mainly documents and smaller images.

I guess I'll keep trying. I read in another thread that somone tried running rclone on another server and that seemed to do the trick. Is it worth going down that route?

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.