RClone extremely slow download speeds - SharePoint

What is the problem you are having with rclone?
Extremely slow download speeds for multiple clients. Had this issue for a good few months now and finally need to make a forum request to help me solve it. These systems, no matter what state they are in operationally, are brand new Ubuntu 20.04 systems.

Someone told me there is a user agent parameter I can do? I don't know how to do this.

The configuration is:

  1. Rclone configured using remote option 31) OneDrive
  2. Configuring it with headless, all options default.
  3. Then going to my PC, running rclone authorize onedrive. Put token in console of headless dedi.
  4. Finish setup.
  5. Upload 100GB file - Normal, about 10MB a second, not hugely concerning, and I don't really care. It's for backups so not bothered by it.
  6. Downloading the file - Oh boy - 1/2MB a second? Seriously? How do I solve this. This is urgent and a client can't get their backup unless they wait 10 hours+.

Run the command 'rclone version' and share the full output of the command.

root@S01:~# rclone version
rclone v1.62.2
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.4.0-146-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.20.2
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)
OneDrive/SharePoint for business.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy -vv backups:S01/servers/c693131a-d0fa-49ad-9e37-a2ed69e33969/c693131a-d0fa-49ad-9e37-a2ed69e33969_12-04-23.tgz .

The rclone config contents with secrets removed.

[backups]
type = onedrive
token = {"access_token":"xxxxxx","expiry":"2023-04-12T12:52:43.131815796-04:00"}
drive_id = xxxx
drive_type = documentLibrary

A log from the command with the -vv flag

root@S01:/var/lib/pterodactyl/volumes/c693131a-d0fa-49ad-9e37-a2ed69e33969# rclone copy -vv backups:S01/servers/c693131a-d0fa-49ad-9e37-a2ed69e33969/c693131a-d0fa-49ad-9e37-a2ed69e33969_12-04-23.tgz .
2023/04/12 11:29:35 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "copy" "-vv" "backups:S01/servers/c693131a-d0fa-49ad-9e37-a2ed69e33969/c693131a-d0fa-49ad-9e37-a2ed69e33969_12-04-23.tgz" "."]
2023/04/12 11:29:35 DEBUG : Creating backend with remote "backups:S01/servers/c693131a-d0fa-49ad-9e37-a2ed69e33969/c693131a-d0fa-49ad-9e37-a2ed69e33969_12-04-23.tgz"
2023/04/12 11:29:35 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/04/12 11:29:37 DEBUG : fs cache: adding new entry for parent of "backups:S01/servers/c693131a-d0fa-49ad-9e37-a2ed69e33969/c693131a-d0fa-49ad-9e37-a2ed69e33969_12-04-23.tgz", "backups:S01/servers/c693131a-d0fa-49ad-9e37-a2ed69e33969"
2023/04/12 11:29:37 DEBUG : Creating backend with remote "."
2023/04/12 11:29:37 DEBUG : fs cache: renaming cache item "." to be canonical "/var/lib/pterodactyl/volumes/c693131a-d0fa-49ad-9e37-a2ed69e33969"
2023/04/12 11:29:38 DEBUG : c693131a-d0fa-49ad-9e37-a2ed69e33969_12-04-23.tgz: Need to transfer - File not found at Destination
2023/04/12 11:29:38 DEBUG : c693131a-d0fa-49ad-9e37-a2ed69e33969_12-04-23.tgz: Starting multi-thread copy with 4 parts of size 26.255Gi
2023/04/12 11:29:38 DEBUG : c693131a-d0fa-49ad-9e37-a2ed69e33969_12-04-23.tgz: multi-thread copy: stream 4/4 (84572700672-112763543739) size 26.255Gi starting
2023/04/12 11:29:38 DEBUG : c693131a-d0fa-49ad-9e37-a2ed69e33969_12-04-23.tgz: multi-thread copy: stream 3/4 (56381800448-84572700672) size 26.255Gi starting
2023/04/12 11:29:38 DEBUG : c693131a-d0fa-49ad-9e37-a2ed69e33969_12-04-23.tgz: multi-thread copy: stream 2/4 (28190900224-56381800448) size 26.255Gi starting
2023/04/12 11:29:38 DEBUG : c693131a-d0fa-49ad-9e37-a2ed69e33969_12-04-23.tgz: multi-thread copy: stream 1/4 (0-28190900224) size 26.255Gi starting
2023/04/12 11:30:37 INFO  :
Transferred:      188.406 MiB / 105.019 GiB, 0%, 2.998 MiB/s, ETA 9h56m51s
Transferred:            0 / 1, 0%
Elapsed time:       1m2.5s
Transferring:
 * c693131a-d0fa-49ad-9e3…d69e33969_12-04-23.tgz:  0% /105.019Gi, 2.998Mi/s, 9h56m51s

2023/04/12 11:31:37 INFO  :
Transferred:      366.672 MiB / 105.019 GiB, 0%, 2.938 MiB/s, ETA 10h7m57s
Transferred:            0 / 1, 0%
Elapsed time:       2m2.5s
Transferring:
 * c693131a-d0fa-49ad-9e3…d69e33969_12-04-23.tgz:  0% /105.019Gi, 2.938Mi/s, 10h7m57s

2023/04/12 11:32:37 INFO  :
Transferred:      549.016 MiB / 105.019 GiB, 1%, 2.935 MiB/s, ETA 10h7m34s
Transferred:            0 / 1, 0%
Elapsed time:       3m2.5s
Transferring:
 * c693131a-d0fa-49ad-9e3…d69e33969_12-04-23.tgz:  0% /105.019Gi, 2.935Mi/s, 10h7m34s

2023/04/12 11:33:37 INFO  :
Transferred:      725.500 MiB / 105.019 GiB, 1%, 2.927 MiB/s, ETA 10h8m10s
Transferred:            0 / 1, 0%
Elapsed time:       4m2.5s
Transferring:
 * c693131a-d0fa-49ad-9e3…d69e33969_12-04-23.tgz:  0% /105.019Gi, 2.927Mi/s, 10h8m10s

2023/04/12 11:34:37 INFO  :
Transferred:      904.203 MiB / 105.019 GiB, 1%, 2.945 MiB/s, ETA 10h3m29s
Transferred:            0 / 1, 0%
Elapsed time:       5m2.5s
Transferring:
 * c693131a-d0fa-49ad-9e3…d69e33969_12-04-23.tgz:  0% /105.019Gi, 2.945Mi/s, 10h3m29s

2023/04/12 11:35:37 INFO  :
Transferred:        1.057 GiB / 105.019 GiB, 1%, 2.926 MiB/s, ETA 10h6m28s
Transferred:            0 / 1, 0%
Elapsed time:       6m2.5s
Transferring:
 * c693131a-d0fa-49ad-9e3…d69e33969_12-04-23.tgz:  1% /105.019Gi, 2.926Mi/s, 10h6m28s

2023/04/12 11:36:37 INFO  :
Transferred:        1.234 GiB / 105.019 GiB, 1%, 2.997 MiB/s, ETA 9h51m5s
Transferred:            0 / 1, 0%
Elapsed time:       7m2.5s
Transferring:
 * c693131a-d0fa-49ad-9e3…d69e33969_12-04-23.tgz:  1% /105.019Gi, 2.997Mi/s, 9h51m5s

2023/04/12 11:37:37 INFO  :
Transferred:        1.413 GiB / 105.019 GiB, 1%, 3.118 MiB/s, ETA 9h27m6s
Transferred:            0 / 1, 0%
Elapsed time:       8m2.5s
Transferring:
 * c693131a-d0fa-49ad-9e3…d69e33969_12-04-23.tgz:  1% /105.019Gi, 3.118Mi/s, 9h27m6s
ETC, 10 hours or so.

onedrive/sharepoint is well known for being slow, not designed for large files such as backup files.
there is no --magic solution.

based on your config, this was not done?
You may choose to create and use your own Client ID, in case the default one does not work well for you. For example, you might see throttling.

might tweak
https://rclone.org/onedrive/#onedrive-chunk-size
https://rclone.org/docs/#multi-thread-streams-n

Used my own client ID - Didn't solve it. Not sure if I configured it correctly though. In the documentation, it states:

Now the application is complete. Run rclone config to create or edit a OneDrive remote. Supply the app ID and password as Client ID and Secret, respectively. rclone will walk you through the remaining steps.

So I put in my Application (client) ID and the Value into the fields of rclone config called "client_id" and "client_secret". This worked and I am still connected to OD, but the problem still persists.

root@S01:/var/lib/pterodactyl/volumes/c693131a-d0fa-49ad-9e37-a2ed69e33969# rclone copy -vv backups:S01/servers/c693131a-d0fa-49ad-9e37-a2ed69e33969/c693131a-d0fa-49ad-9e37-a2ed69e33969_12-04-23.tgz .
2023/04/12 12:27:03 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "copy" "-vv" "backups:S01/servers/c693131a-d0fa-49ad-9e37-a2ed69e33969/c693131a-d0fa-49ad-9e37-a2ed69e33969_12-04-23.tgz" "."]
2023/04/12 12:27:03 DEBUG : Creating backend with remote "backups:S01/servers/c693131a-d0fa-49ad-9e37-a2ed69e33969/c693131a-d0fa-49ad-9e37-a2ed69e33969_12-04-23.tgz"
2023/04/12 12:27:03 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/04/12 12:27:06 DEBUG : fs cache: adding new entry for parent of "backups:S01/servers/c693131a-d0fa-49ad-9e37-a2ed69e33969/c693131a-d0fa-49ad-9e37-a2ed69e33969_12-04-23.tgz", "backups:S01/servers/c693131a-d0fa-49ad-9e37-a2ed69e33969"
2023/04/12 12:27:06 DEBUG : Creating backend with remote "."
2023/04/12 12:27:06 DEBUG : fs cache: renaming cache item "." to be canonical "/var/lib/pterodactyl/volumes/c693131a-d0fa-49ad-9e37-a2ed69e33969"
2023/04/12 12:27:06 DEBUG : c693131a-d0fa-49ad-9e37-a2ed69e33969_12-04-23.tgz: Need to transfer - File not found at Destination
2023/04/12 12:27:06 DEBUG : c693131a-d0fa-49ad-9e37-a2ed69e33969_12-04-23.tgz: Starting multi-thread copy with 4 parts of size 26.255Gi
2023/04/12 12:27:06 DEBUG : c693131a-d0fa-49ad-9e37-a2ed69e33969_12-04-23.tgz: multi-thread copy: stream 4/4 (84572700672-112763543739) size 26.255Gi starting
2023/04/12 12:27:06 DEBUG : c693131a-d0fa-49ad-9e37-a2ed69e33969_12-04-23.tgz: multi-thread copy: stream 3/4 (56381800448-84572700672) size 26.255Gi starting
2023/04/12 12:27:06 DEBUG : c693131a-d0fa-49ad-9e37-a2ed69e33969_12-04-23.tgz: multi-thread copy: stream 1/4 (0-28190900224) size 26.255Gi starting
2023/04/12 12:27:06 DEBUG : c693131a-d0fa-49ad-9e37-a2ed69e33969_12-04-23.tgz: multi-thread copy: stream 2/4 (28190900224-56381800448) size 26.255Gi starting
2023/04/12 12:28:06 INFO  :
Transferred:      194.344 MiB / 105.019 GiB, 0%, 3.105 MiB/s, ETA 9h36m11s
Transferred:            0 / 1, 0%
Elapsed time:       1m2.9s
Transferring:
 * c693131a-d0fa-49ad-9e3…d69e33969_12-04-23.tgz:  0% /105.019Gi, 3.105Mi/s, 9h36m11s

2023/04/12 12:29:06 INFO  :
Transferred:      373.656 MiB / 105.019 GiB, 0%, 3.007 MiB/s, ETA 9h54m
Transferred:            0 / 1, 0%
Elapsed time:       2m2.9s
Transferring:
 * c693131a-d0fa-49ad-9e3…d69e33969_12-04-23.tgz:  0% /105.019Gi, 3.007Mi/s, 9h54m0s

2023/04/12 12:30:06 INFO  :
Transferred:      553.406 MiB / 105.019 GiB, 1%, 2.969 MiB/s, ETA 10h33s
Transferred:            0 / 1, 0%
Elapsed time:       3m2.9s
Transferring:
 * c693131a-d0fa-49ad-9e3…d69e33969_12-04-23.tgz:  0% /105.019Gi, 2.969Mi/s, 10h0m33s

2023/04/12 12:31:06 INFO  :
Transferred:      729.625 MiB / 105.019 GiB, 1%, 3.006 MiB/s, ETA 9h52m10s
Transferred:            0 / 1, 0%
Elapsed time:       4m2.9s
Transferring:
 * c693131a-d0fa-49ad-9e3…d69e33969_12-04-23.tgz:  0% /105.019Gi, 3.006Mi/s, 9h52m10s

2023/04/12 12:32:06 INFO  :
Transferred:      909.828 MiB / 105.019 GiB, 1%, 2.981 MiB/s, ETA 9h56m10s
Transferred:            0 / 1, 0%
Elapsed time:       5m2.9s
Transferring:
 * c693131a-d0fa-49ad-9e3…d69e33969_12-04-23.tgz:  0% /105.019Gi, 2.981Mi/s, 9h56m10s

2023/04/12 12:33:06 INFO  :
Transferred:        1.060 GiB / 105.019 GiB, 1%, 2.962 MiB/s, ETA 9h59m2s
Transferred:            0 / 1, 0%
Elapsed time:       6m2.9s
Transferring:
 * c693131a-d0fa-49ad-9e3…d69e33969_12-04-23.tgz:  1% /105.019Gi, 2.962Mi/s, 9h59m2s

2023/04/12 12:34:06 INFO  :
Transferred:        1.238 GiB / 105.019 GiB, 1%, 3.032 MiB/s, ETA 9h44m12s
Transferred:            0 / 1, 0%
Elapsed time:       7m2.9s
Transferring:
 * c693131a-d0fa-49ad-9e3…d69e33969_12-04-23.tgz:  1% /105.019Gi, 3.032Mi/s, 9h44m12s

2023/04/12 12:35:06 INFO  :
Transferred:        1.413 GiB / 105.019 GiB, 1%, 3.148 MiB/s, ETA 9h21m42s
Transferred:            0 / 1, 0%
Elapsed time:       8m2.9s
Transferring:
 * c693131a-d0fa-49ad-9e3…d69e33969_12-04-23.tgz:  1% /105.019Gi, 3.148Mi/s, 9h21m42s

2023/04/12 12:36:06 INFO  :
Transferred:        1.584 GiB / 105.019 GiB, 2%, 2.892 MiB/s, ETA 10h10m22s
Transferred:            0 / 1, 0%
Elapsed time:       9m2.9s
Transferring:
 * c693131a-d0fa-49ad-9e3…d69e33969_12-04-23.tgz:  1% /105.019Gi, 2.892Mi/s, 10h10m22s

2023/04/12 12:37:06 INFO  :
Transferred:        1.762 GiB / 105.019 GiB, 2%, 2.841 MiB/s, ETA 10h20m22s
Transferred:            0 / 1, 0%
Elapsed time:      10m2.9s
Transferring:
 * c693131a-d0fa-49ad-9e3…d69e33969_12-04-23.tgz:  1% /105.019Gi, 2.841Mi/s, 10h20m22s

2023/04/12 12:38:06 INFO  :
Transferred:        1.939 GiB / 105.019 GiB, 2%, 3.063 MiB/s, ETA 9h34m17s
Transferred:            0 / 1, 0%
Elapsed time:      11m2.9s
Transferring:
 * c693131a-d0fa-49ad-9e3…d69e33969_12-04-23.tgz:  1% /105.019Gi, 3.063Mi/s, 9h34m17s

2023/04/12 12:39:06 INFO  :
Transferred:        2.110 GiB / 105.019 GiB, 2%, 2.877 MiB/s, ETA 10h10m25s
Transferred:            0 / 1, 0%
Elapsed time:      12m2.9s
Transferring:
 * c693131a-d0fa-49ad-9e3…d69e33969_12-04-23.tgz:  2% /105.019Gi, 2.877Mi/s, 10h10m25s

2023/04/12 12:40:06 INFO  :
Transferred:        2.285 GiB / 105.019 GiB, 2%, 2.958 MiB/s, ETA 9h52m38s
Transferred:            0 / 1, 0%
Elapsed time:      13m2.9s
Transferring:
 * c693131a-d0fa-49ad-9e3…d69e33969_12-04-23.tgz:  2% /105.019Gi, 2.958Mi/s, 9h52m38s

Regarding your chunk size, I did half of 250M, so 125M - Still same issue.

root@S01:/var/lib/pterodactyl/volumes/c693131a-d0fa-49ad-9e37-a2ed69e33969# rclone copy -vv --onedrive-chunk-size 131072000 backups:S01/servers/c693131a-d0fa-49ad-9e37-a2ed69e33969/c693131a-d0fa-49ad-9e37-a2ed69e33969_12-04-23.tgz .
2023/04/12 12:42:20 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "copy" "-vv" "--onedrive-chunk-size" "131072000" "backups:S01/servers/c693131a-d0fa-49ad-9e37-a2ed69e33969/c693131a-d0fa-49ad-9e37-a2ed69e33969_12-04-23.tgz" "."]
2023/04/12 12:42:20 DEBUG : Creating backend with remote "backups:S01/servers/c693131a-d0fa-49ad-9e37-a2ed69e33969/c693131a-d0fa-49ad-9e37-a2ed69e33969_12-04-23.tgz"
2023/04/12 12:42:20 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/04/12 12:42:20 DEBUG : backups: detected overridden config - adding "{QLdLX}" suffix to name
2023/04/12 12:42:22 DEBUG : fs cache: adding new entry for parent of "backups:S01/servers/c693131a-d0fa-49ad-9e37-a2ed69e33969/c693131a-d0fa-49ad-9e37-a2ed69e33969_12-04-23.tgz", "backups{QLdLX}:S01/servers/c693131a-d0fa-49ad-9e37-a2ed69e33969"
2023/04/12 12:42:22 DEBUG : Creating backend with remote "."
2023/04/12 12:42:22 DEBUG : fs cache: renaming cache item "." to be canonical "/var/lib/pterodactyl/volumes/c693131a-d0fa-49ad-9e37-a2ed69e33969"
2023/04/12 12:42:23 DEBUG : c693131a-d0fa-49ad-9e37-a2ed69e33969_12-04-23.tgz: Need to transfer - File not found at Destination
2023/04/12 12:42:23 DEBUG : c693131a-d0fa-49ad-9e37-a2ed69e33969_12-04-23.tgz: Starting multi-thread copy with 4 parts of size 26.255Gi
2023/04/12 12:42:23 DEBUG : c693131a-d0fa-49ad-9e37-a2ed69e33969_12-04-23.tgz: multi-thread copy: stream 4/4 (84572700672-112763543739) size 26.255Gi starting
2023/04/12 12:42:23 DEBUG : c693131a-d0fa-49ad-9e37-a2ed69e33969_12-04-23.tgz: multi-thread copy: stream 2/4 (28190900224-56381800448) size 26.255Gi starting
2023/04/12 12:42:23 DEBUG : c693131a-d0fa-49ad-9e37-a2ed69e33969_12-04-23.tgz: multi-thread copy: stream 3/4 (56381800448-84572700672) size 26.255Gi starting
2023/04/12 12:42:23 DEBUG : c693131a-d0fa-49ad-9e37-a2ed69e33969_12-04-23.tgz: multi-thread copy: stream 1/4 (0-28190900224) size 26.255Gi starting
2023/04/12 12:43:22 INFO  :
Transferred:      182.250 MiB / 105.019 GiB, 0%, 2.910 MiB/s, ETA 10h14m58s
Transferred:            0 / 1, 0%
Elapsed time:       1m2.7s
Transferring:
 * c693131a-d0fa-49ad-9e3…d69e33969_12-04-23.tgz:  0% /105.019Gi, 2.910Mi/s, 10h14m58s


2023/04/12 12:44:22 INFO  :
Transferred:      359.297 MiB / 105.019 GiB, 0%, 2.972 MiB/s, ETA 10h1m4s
Transferred:            0 / 1, 0%
Elapsed time:       2m2.7s
Transferring:
 * c693131a-d0fa-49ad-9e3…d69e33969_12-04-23.tgz:  0% /105.019Gi, 2.972Mi/s, 10h1m4s

2023/04/12 12:45:22 INFO  :
Transferred:      535.453 MiB / 105.019 GiB, 0%, 2.864 MiB/s, ETA 10h22m44s
Transferred:            0 / 1, 0%
Elapsed time:       3m2.7s
Transferring:
 * c693131a-d0fa-49ad-9e3…d69e33969_12-04-23.tgz:  0% /105.019Gi, 2.864Mi/s, 10h22m44s

Regarding your multi streams, it's already doing 4.

imho, you just have to wait.

maybe try to download the file using onedrive website, what speeds to you get?

Speeds on the site are 100MB down so I am not too fussed about that. I remember you can do ?download=1 on the end of the file link to download the file much quicker than rclone can, so I will remember in the future to just do that.

Thank you anyway. Just mighty annoying why it can't download quick but upload fine.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.