Rclone copy is extremely slow

What is the problem you are having with rclone?

My network is 5 Gbps. When I use rclone to copy, the download speed is just around 30 MB/s. If I do the same in AirExplorer, it must be around 100 MB/s. But thereotically, it must be over 300 MB/s, when I use IDM to download directly from Google Drive.

Run the command 'rclone version' and share the full output of the command.

rclone v1.60.1

  • os/version: Microsoft Windows 11 Pro 22H2 (64 bit)
  • os/kernel: 10.0.22621.819 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.19.3
  • go/linking: static
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Google Drive.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy --log-file rclone.log "GDrive":"test.mkv" --transfers=16 --drive-chunk-size=256M U: --progress -vv```

#### The rclone config contents with secrets removed.  
<!--  You should use 3 backticks to begin and end your paste to make it readable.   -->

[GDrive]
type = drive
scope = drive
token = {"access_token":"XXXXXXX","token_type":"Bearer","refresh_token":"XXXXXXXX","expiry":"2022-11-19T12:37:19.4742349-06:00"}
team_drive = XXXXXXXXX
root_folder_id =


#### A log from the command with the `-vv` flag  
<!-- You should use 3 backticks to begin and end your paste to make it readable.  Or use a service such as https://pastebin.com or https://gist.github.com/   -->

2022/11/19 12:42:42 DEBUG : rclone: Version "v1.60.1" starting with parameters ["rclone" "copy" "--log-file" "rclone.log" "GDrive:test.mkv" "--transfers=16" "--drive-chunk-size=256M" "U:" "--progress" "-vv"]
2022/11/19 12:42:42 DEBUG : Creating backend with remote "GDrive:test.mkv"
2022/11/19 12:42:42 DEBUG : Using config file from "C:\Users\XXXXXX\AppData\Roaming\rclone\rclone.conf"
2022/11/19 12:42:42 DEBUG : GDrive: detected overridden config - adding "{A6J6b}" suffix to name
2022/11/19 12:42:43 DEBUG : fs cache: adding new entry for parent of "GDrive:test.mkv", "GDrive{A6J6b}:"
2022/11/19 12:42:43 DEBUG : Creating backend with remote "U:"
2022/11/19 12:42:43 DEBUG : fs cache: renaming cache item "U:" to be canonical "//?/U:/"
2022/11/19 12:42:43 DEBUG : test.mkv: Need to transfer - File not found at Destination
2022/11/19 12:42:43 INFO : Writing sparse files: use --local-no-sparse or --multi-thread-streams 0 to disable
2022/11/19 12:42:43 DEBUG : test.mkv: Starting multi-thread copy with 4 parts of size 263.875Mi
2022/11/19 12:42:43 DEBUG : test.mkv: multi-thread copy: stream 4/4 (830078976-1106636308) size 263.746Mi starting
2022/11/19 12:42:43 DEBUG : test.mkv: multi-thread copy: stream 2/4 (276692992-553385984) size 263.875Mi starting
2022/11/19 12:42:43 DEBUG : test.mkv: multi-thread copy: stream 1/4 (0-276692992) size 263.875Mi starting
2022/11/19 12:42:43 DEBUG : test.mkv: multi-thread copy: stream 3/4 (553385984-830078976) size 263.875Mi starting
2022/11/19 12:43:25 DEBUG : test.mkv: multi-thread copy: stream 4/4 (830078976-1106636308) size 263.746Mi finished
2022/11/19 12:43:25 DEBUG : test.mkv: multi-thread copy: stream 2/4 (276692992-553385984) size 263.875Mi finished
2022/11/19 12:43:25 DEBUG : test.mkv: multi-thread copy: stream 3/4 (553385984-830078976) size 263.875Mi finished
2022/11/19 12:43:25 DEBUG : test.mkv: multi-thread copy: stream 1/4 (0-276692992) size 263.875Mi finished
2022/11/19 12:43:25 DEBUG : test.mkv: Finished multi-thread copy with 4 parts of size 263.875Mi
2022/11/19 12:43:36 DEBUG : test.mkv: md5 = 20aa8f0de1c5b02e7130b9eedecca77c OK
2022/11/19 12:43:36 INFO : test.mkv: Multi-thread Copied (new)
2022/11/19 12:43:36 INFO :
Transferred: 1.031 GiB / 1.031 GiB, 100%, 12.354 MiB/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 53.6s

2022/11/19 12:43:36 DEBUG : 11 go routines active

hello and welcome to the forum,

i do not see the GDrive: in that rclone copy?

the source M:/test.mkv - what is that, a local drive, a network drive, rclone mount or what?
and
the dest is U: - what is that, a local drive, a network drive, rclone mount or what?

as far as i can tell, that is not used, as not using gdrive.
would expect to see something like
detected overridden config - adding "{I9tHt}" suffix to name

Sorry for the confusion, I've put the wrong log and command.
I've updated the correct ones.
Please help me with your expert ideas.

U: is the local drive on the PC.

Thanks so much.

not sure what that means,
so IDM downloads 1GiB in approx. 3 seconds?

so AirExplorer downlaods 1GiB in approx. 10 seconds?

note: --transfers=16 - does nothing, as you are copying a single file.

Yes, Internet Download Manager downloads 1GB in approx. 3 seconds.
And AirExplorer for around 10 seconds.
I feel like Rclone has been capped at around 35MB/s.
I really want to fix it.

did you do this?
https://rclone.org/drive/#making-your-own-client-id

if had then your remote would look like

[remote]
type = drive
client_id = redacted
client_secret = redacted
scope = drive
token = redacted

Even if I tried with multithread by this command:
rclone copy --log-file rclone.log "GDrive":"New Folder" U: --ignore-existing --verbose --transfers 4 --checkers 8 --bwlimit 5000M --contimeout 60s --timeout 300s --retries 3 --low-level-retries 10 --drive-chunk-size 256M --stats 1s --progress

Here is the ridiculous result:

Transferred: 992.904 MiB / 1.300 TiB, 0%, 22.896 MiB/s, ETA 16h31m19s
Checks: 1 / 1, 100%
Transferred: 0 / 132, 0%
Elapsed time: 44.3s
Transferring:

  •            X.mkv:  1% /8.158Gi, 2.805Mi/s, 48m52s
    
  •           Y.mkv:  2% /15.079Gi, 9.890Mi/s, 25m18s
    
  •        Z.mkv:  2% /8.113Gi, 3.961Mi/s, 34m12s
    
  •        A.mkv:  1% /23.733Gi, 6.239Mi/s, 1h4m13s
    

see my last comment?

no need for these, ok?
if you are not sure what a flag does, best not to use it, just use defaults
--bwlimit 5000M --contimeout 60s --timeout 300s --retries 3 --low-level-retries 10

[GDrive]
type = drive
client_id = XXXXXX.apps.googleusercontent.com
client_secret = XXXXXX
scope = drive
token = {"access_token":"XXXXX","token_type":"Bearer","refresh_token":"XXXXX","expiry":"2022-11-19T14:26:34.1457745-06:00"}
team_drive = XXXXXXX
root_folder_id = 

please post the entire debug output, not just a snippet, ok?

I've made my own client ID.

I'm not sure if my command to get the debug output is correct or not, here it is:
rclone -vv copy "GDrive":"X.mp4" U: --drive-chunk-size 256M --progress > log.txt

And here is the log file:

https://pastebin.com/xy1yhh9x

--- you keep chagning the source, pick one source file and use that, ok?

-- remove --drive-chunk-size 256M

--- download the entire file and then post the rclone.log
rclone copy GDrive:X.mp4 U: --progress --log-level=DEBUG --log-file=rclone.log --stats=5h

that will establish a baseline performance and we can tweak from there.

Here is the log:

https://pastebin.com/i5etLEnT

my connection is 1Gbps

i condensed the output

rclone-v1.60.1-windows-amd64.exe copy gdrive01:zork\2GiB.file D:\dest -vv --progress
2022/11/19 15:23:46 DEBUG : rclone: Version "v1.60.1" starting with parameters ["C:\\data\\rclone\\rclone-v1.60.1-windows-amd64.exe" "copy" "gdrive01:zork\\2GiB.file" "D:\\dest" "-vv" "--progress"]
2022-11-19 15:24:06 DEBUG : 2GiB.file: Finished multi-thread copy with 4 parts of size 512Mi
2022-11-19 15:24:16 DEBUG : 2GiB.file: md5 = f6a70a3c0f7711e6a9d7e2b539b6264f OK
2022-11-19 15:24:16 INFO  : 2GiB.file: Multi-thread Copied (new)
Transferred:            2 GiB / 2 GiB, 100%, 56.342 MiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:        30.0s

I really don't know. I'm using the 5Gbps Fiber Connection.
It just takes me less than 15 seconds for the 2GB file by using IDM.
But by using rclone, it can take around 1.5 mins.
I don't have any clue to fix it.
Maybe, I just use AirExplorer as the middle solution.
:frowning:

Downloading things via IDM or browser isn't the same as downloading via the Drive API.

Try increasing the number of streams as that would more simulate your IDM.

--multi-thread-streams=8

edit: this is meant for @Animosity022
maybe it will make a difference with 5Gbps

ran this three times, same result, no practical difference.

rclone-v1.60.1-windows-amd64.exe copy gdrive01:zork\2GiB.file D:\dest -vv --progress --multi-thread-streams=8
2022/11/19 15:41:17 DEBUG : rclone: Version "v1.60.1" starting with parameters ["C:\\data\\rclone\\rclone-v1.60.1-windows-amd64.exe" "copy" "gdrive01:zork\\2GiB.file" "D:\\dest" "-vv" "--progress" "--multi-thread-streams=8"]
2022-11-19 15:41:36 DEBUG : 2GiB.file: Finished multi-thread copy with 8 parts of size 256Mi
2022-11-19 15:41:46 DEBUG : 2GiB.file: md5 = f6a70a3c0f7711e6a9d7e2b539b6264f OK
2022-11-19 15:41:46 INFO  : 2GiB.file: Multi-thread Copied (new)
Transferred:            2 GiB / 2 GiB, 100%, 60.036 MiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:        29.4s

might try

https://forum.rclone.org/t/rclone-25-50mbps-slow-transfer-speed/29157/2?u=asdffdsa

TCP Global Parameters
----------------------------------------------
Receive-Side Scaling State          : enabled
Receive Window Auto-Tuning Level    : normal
Add-On Congestion Control Provider  : default
ECN Capability                      : disabled
RFC 1323 Timestamps                 : allowed
Initial RTO                         : 1000
Receive Segment Coalescing State    : enabled
Non Sack Rtt Resiliency             : disabled
Max SYN Retransmissions             : 4
Fast Open                           : enabled
Fast Open Fallback                  : enabled
HyStart                             : enabled
Proportional Rate Reduction         : enabled
Pacing Profile                      : off

I've rebooted the PC.
The result is still the same.

Your windows box has a 10 Gig fiber card in it or something?

Where are you located roughly? US / non US? Is your Google Suite US / non US? EDU account?