Very slow performance when using rclone on windows to download from gdrive to unraid share (p:)

Jinxyface · April 27, 2021, 11:09pm

Hello guys, not new to Rclone, but I am new to the system I am using for local back ups (UnRAID). I was using a Synology DS1019+ but decided to make my own NAS for freedom to upgrade and whatnot. I was using Rclone/RcloneBrowser on my Windows machine to sync up my Synology NAS (P: drive) to my mounted GDrive (X: drive).

When uploading from Synology to GDrive (P: -> X:) I would get about 120MB/s

When downloading from GDrive to Synology (X: -> P:), I would also get the same speed

The Synology is decommissioned now and my unraid box is working (ver 6.9.2). I've create the array for my drives, created a user share, and mounted unRAID to the same letter as my old Synology (So P:)

But now when I download from GDrive to unRAID (X: -> new P:) I'm getting around 30MB/s. I have 5 drives all connected through a SAS2 LSI-9211-8i card. And this is only happening when it's X: -> P: through Rclone

If i download from Rclone to my local storage drive (E:) I get full speed, and then if I drag and drop the files from Windows to the unRAID (local E: to unRAID mount P:) share, it's full speed. It only seems to be when I remove the middle-man that is Windows that the spee drops off. Does anyone have any ideas?

Here is the mount I'm using:

mount --allow-other --dir-cache-time 72h --drive-chunk-size 64M --log-level INFO --vfs-read-chunk-size 32M --vfs-read-chunk-size-limit off gdrive: X: --config "C:\Users\Matthew.config\rclone\rclone.conf" --vfs-cache-mode writes --rc

No matter what I do I can't seem to break the 30MB/s speed when going from GDrive -> P:, but if I do GDrive -> Local Windows or Local Windows -> P: the speeds are fine.

asdffdsa · April 27, 2021, 11:25pm

hello and welcome to the forum,

so the only thing that changed is the hardware, nothing about rclone itself.

what version of rclone?
on windows, as per the docs, --allow-other does nothing.

E: local drive on windows?
P: network drive pointing to unraid?
X: points to gdrive?

what is the speed of rclone copy from unraid to gdrive, not using windows?

have you tried a rclone mount on the unraid box and copy from unraid to gdrive, using unraid GUI?

have you tried the unraid rclone plugin?

Jinxyface · April 28, 2021, 12:25am

I haven't tried Rclone on unRAID yet, I'm not too familair with setting up rclone on unix/linux systems. I can do some reading and try to set it up. Would rclone on unraid allow me to download from remote to local, or is it just local upload to remote?

asdffdsa · April 28, 2021, 12:37am

rclone does

local to remote
remote to local
local to local
remote to remote

this seems to be an issue with unraid, not rclone.
perhaps the hardware is slow, perhaps unraid is not configured correctly.

asdffdsa · April 28, 2021, 12:43am

not sure what that means?
if you are not running rclone on unraid, then windows is always in the middle?

Jinxyface · April 28, 2021, 12:49am

My unraid box has a Intel Xeon E5-2670 V3 and 16GB of ECC DDR4 RAM. It's definitely not a spec issue. I'm thinking it's unraid too, but I can't seem to find anyone else with the issue

asdffdsa · April 28, 2021, 12:51am

not sure what to tell you.
the best place for help would be at the unraid forum.

Jinxyface · April 28, 2021, 12:56am

The unraid forum told me it was an rclone issue and to come here. I'm really hoping this isn't just going to be a back and forth of both sides not trying to help.

ncw · April 28, 2021, 8:29am

You could do an experiment.. try using rclone copy to copy a big file to your P: drive direct from google and see what speed you get there. To make it a fair comparison you should use the flag --multi-thread-streams 0, so something like this

rclone copy --multi-thread-streams 0 --config "C:\Users\Matthew.config\rclone\rclone.conf" -P gdrive:/path/to/big/file P:testdir

And can you try copying that same file via the mount so

rclone copy --multi-thread-streams 0 --config "C:\Users\Matthew.config\rclone\rclone.conf" -P X:\path\to\big\file P:testdir2

That will give us a like-to-like comparison to see if the rclone mount is the problem or not.

How are you doing that copy? Is it with rclone copy? If so, likely rclone is using multi thread downloads which it doesn't use when copying from the mount.

Jinxyface · April 28, 2021, 6:33pm

Hello, just for clarity sake, on Windows I've been using RCloneBrowser to give myself a GUI. If I set the source from X: (mount) to E: (local storage), and hit run, it consistently performs at about 120MB/s.

If I set source to X: (mount) to destination P: (NAS), I get about 30MB/s

If I download from X to local E:, then drag/drop from E: to P:, both are 120MB/s, it's only when RCloneBrowser is going from Source X: to Destiation P: that it's slow. I've actually gotten RClone set up inside of unraid and did an rclone copy on some test files and it seems to run...slightly faster, at about 45-50MB/s, but still nowhere what it should be at.

asdffdsa · April 28, 2021, 6:52pm

now that rclone is running on unraid and you have done the comparison test suggested by ncw.
best not to complicate/confuse by using windows as a middle man.

on unraid,

what are the results of internet speed tests?
what rclone command are you running?
are you copying a single large file or lots of small files?
if testing a download, i use not use --multi-thread-streams 0 unless ncw suggests otherwise

Jinxyface · April 28, 2021, 6:55pm

what are the results of internet speed tests?

850-900mbps from within a Firefox docker container. I have gigabit

what rclone command are you running?

I was just doing rclone copy (sourcepath) (destpath)

are you copying a single large file or lots of small files?

I suppose they'd be multiple large files? It's Plex media, so lots of 1-3GB files

asdffdsa · April 28, 2021, 6:59pm

what is the exact rclone command you are testing with?

that would be super slow, did you mean 850-900Mbps

if lots of files, try --transfers=8 checkers=8 or even higher.

Jinxyface · April 28, 2021, 7:04pm

Yes you're right sorry, my shift key wasn't working. 850-900 MBps

if lots of files, try --transfers=8 checkers=8 or even higher.

I do transfers=10 and checkers=8

asdffdsa · April 28, 2021, 7:08pm

really hard to figure out what needs to be done, if you do not post the exact command used for that test.
if you test rclone and post a speed result, need to post the exact command used.

Jinxyface · April 28, 2021, 7:55pm

So I just ran this command:

rclone copy -vv "gdrive:Anime Movies/Blackfox (2019)/" "P:\Anime Movies\Blackfox (2019)"

And it appears to break the file down in chunks and download those chunks consecutively? This is from GDrive to NAS directly

This is my output from rclone

C:\rclone>rclone copy -vv "gdrive:Anime Movies/Blackfox (2019)/" "P:\Anime Movies\Blackfox (2019)"
2021/04/28 14:43:19 DEBUG : rclone: Version "v1.56.0-beta.5448.b78c9a65f" starting with parameters ["rclone" "copy" "-vv" "gdrive:Anime Movies/Blackfox (2019)/" "P:\\Anime Movies\\Blackfox (2019)"]
2021/04/28 14:43:19 DEBUG : Creating backend with remote "gdrive:Anime Movies/Blackfox (2019)/"
2021/04/28 14:43:19 DEBUG : Using config file from "C:\\Users\\Matthew\\.config\\rclone\\rclone.conf"
2021/04/28 14:43:20 DEBUG : fs cache: renaming cache item "gdrive:Anime Movies/Blackfox (2019)/" to be canonical "gdrive:Anime Movies/Blackfox (2019)"
2021/04/28 14:43:20 DEBUG : Creating backend with remote "P:\\Anime Movies\\Blackfox (2019)"
2021/04/28 14:43:20 DEBUG : fs cache: renaming cache item "P:\\Anime Movies\\Blackfox (2019)" to be canonical "//?/P:/Anime Movies/Blackfox (2019)"
2021/04/28 14:43:20 DEBUG : Local file system at //?/P:/Anime Movies/Blackfox (2019): Waiting for checks to finish
2021/04/28 14:43:20 DEBUG : Local file system at //?/P:/Anime Movies/Blackfox (2019): Waiting for transfers to finish
2021/04/28 14:43:20 INFO  : Writing sparse files: use --local-no-sparse or --multi-thread-streams 0 to disable
2021/04/28 14:43:20 DEBUG : Blackfox (2019).mkv: Starting multi-thread copy with 4 parts of size 281.125Mi
2021/04/28 14:43:20 DEBUG : Blackfox (2019).mkv: multi-thread copy: stream 4/4 (884342784-1179004895) size 281.012Mi starting
2021/04/28 14:43:20 DEBUG : Blackfox (2019).mkv: multi-thread copy: stream 1/4 (0-294780928) size 281.125Mi starting
2021/04/28 14:43:20 DEBUG : Blackfox (2019).mkv: multi-thread copy: stream 2/4 (294780928-589561856) size 281.125Mi starting
2021/04/28 14:43:20 DEBUG : Blackfox (2019).mkv: multi-thread copy: stream 3/4 (589561856-884342784) size 281.125Mi starting
2021/04/28 14:44:10 DEBUG : Blackfox (2019).mkv: multi-thread copy: stream 4/4 (884342784-1179004895) size 281.012Mi finished
2021/04/28 14:44:10 DEBUG : Blackfox (2019).mkv: multi-thread copy: stream 2/4 (294780928-589561856) size 281.125Mi finished
2021/04/28 14:44:10 DEBUG : Blackfox (2019).mkv: multi-thread copy: stream 1/4 (0-294780928) size 281.125Mi finished
2021/04/28 14:44:10 DEBUG : Blackfox (2019).mkv: multi-thread copy: stream 3/4 (589561856-884342784) size 281.125Mi finished
2021/04/28 14:44:10 DEBUG : Blackfox (2019).mkv: Finished multi-thread copy with 4 parts of size 281.125Mi
2021/04/28 14:44:20 INFO  :
Transferred:        1.098Gi / 1.098 GiByte, 100%, 18.792 MiByte/s, ETA 0s
Transferred:            0 / 1, 0%
Elapsed time:       1m0.6s
Transferring:
 Blackfox (2019).mkv:100% /1.098Gi, 13.051Mi/s, 0s

2021/04/28 14:44:22 DEBUG : Blackfox (2019).mkv: MD5 = 7a5db9440e8aa4e895df398d9253dcf2 OK
2021/04/28 14:44:22 INFO  : Blackfox (2019).mkv: Multi-thread Copied (new)
2021/04/28 14:44:22 INFO  :
Transferred:        1.098Gi / 1.098 GiByte, 100%, 18.031 MiByte/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:       1m3.1s

So it looks like it took about a minute to download a 1GB file, which is relatively slow as I should be able to do that in a few seconds on my gigabit.

If I run the same command but to my local storage E: drive, I get this:

C:\rclone>rclone copy -vv "gdrive:Anime Movies/Blackfox (2019)/" "E:\"
2021/04/28 14:53:28 DEBUG : rclone: Version "v1.56.0-beta.5448.b78c9a65f" starting with parameters ["rclone" "copy" "-vv" "gdrive:Anime Movies/Blackfox (2019)/" "E:\""]
2021/04/28 14:53:28 DEBUG : Creating backend with remote "gdrive:Anime Movies/Blackfox (2019)/"
2021/04/28 14:53:28 DEBUG : Using config file from "C:\\Users\\Matthew\\.config\\rclone\\rclone.conf"
2021/04/28 14:53:29 DEBUG : fs cache: renaming cache item "gdrive:Anime Movies/Blackfox (2019)/" to be canonical "gdrive:Anime Movies/Blackfox (2019)"
2021/04/28 14:53:29 DEBUG : Creating backend with remote "E:\""
2021/04/28 14:53:29 DEBUG : fs cache: renaming cache item "E:\"" to be canonical "//?/E:/\""
2021/04/28 14:53:29 DEBUG : Local file system at //?/E:/": Waiting for checks to finish
2021/04/28 14:53:29 DEBUG : Local file system at //?/E:/": Waiting for transfers to finish
2021/04/28 14:53:29 INFO  : Writing sparse files: use --local-no-sparse or --multi-thread-streams 0 to disable
2021/04/28 14:53:29 DEBUG : Blackfox (2019).mkv: Starting multi-thread copy with 4 parts of size 281.125Mi
2021/04/28 14:53:29 DEBUG : Blackfox (2019).mkv: multi-thread copy: stream 4/4 (884342784-1179004895) size 281.012Mi starting
2021/04/28 14:53:29 DEBUG : Blackfox (2019).mkv: multi-thread copy: stream 2/4 (294780928-589561856) size 281.125Mi starting
2021/04/28 14:53:29 DEBUG : Blackfox (2019).mkv: multi-thread copy: stream 3/4 (589561856-884342784) size 281.125Mi starting
2021/04/28 14:53:29 DEBUG : Blackfox (2019).mkv: multi-thread copy: stream 1/4 (0-294780928) size 281.125Mi starting
2021/04/28 14:53:34 DEBUG : Blackfox (2019).mkv: multi-thread copy: stream 3/4 (589561856-884342784) size 281.125Mi finished
2021/04/28 14:53:37 DEBUG : Blackfox (2019).mkv: multi-thread copy: stream 4/4 (884342784-1179004895) size 281.012Mi finished
2021/04/28 14:53:39 DEBUG : Blackfox (2019).mkv: multi-thread copy: stream 2/4 (294780928-589561856) size 281.125Mi finished
2021/04/28 14:53:44 DEBUG : Blackfox (2019).mkv: multi-thread copy: stream 1/4 (0-294780928) size 281.125Mi finished
2021/04/28 14:53:48 DEBUG : Blackfox (2019).mkv: Finished multi-thread copy with 4 parts of size 281.125Mi
2021/04/28 14:53:58 DEBUG : Blackfox (2019).mkv: MD5 = 7a5db9440e8aa4e895df398d9253dcf2 OK
2021/04/28 14:53:58 INFO  : Blackfox (2019).mkv: Multi-thread Copied (new)
2021/04/28 14:53:58 INFO  :
Transferred:        1.098Gi / 1.098 GiByte, 100%, 39.350 MiByte/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:        29.1s

So just under half the time.

Here's Rclonebrowser from X: (mount) to E: (local storage)

And here is Rclonebrowser from X: (mount) to P: (unRAID NAS)

asdffdsa · April 28, 2021, 8:15pm

we agree that running rclone on the win box is faster than the unraid box.
so let's forget about using windows machine as the middleman and just run rclone on the unraid box only

perhaps, test using the lastest stable rclone and test using --progress for real-time speed updates as rclone is downloading.

rclone is running all the threads at the same time.

Jinxyface · April 28, 2021, 8:34pm

So running:

rclone copy --progress "gdrive:Anime/ReZero" "/mnt/disk1/Media/Anime"

from the unraid box itself gives me this

So I guess I'll just have to do this from unraid itself. I wish unraid had an rclonebrowser equivelant lmao. I'm not too well versed in terminal

asdffdsa · April 28, 2021, 8:38pm

that is good speed.

rclone browser runs on linux.
or
run rclone mount on unraid and use unraid gui to transfer files and use some docker image that exposes a file manager, unraid plugin such as double commander which runs on runs on unraid.
i use double commander on all my computers - linux and windows. on windows, it is a portable app just like rclone on windows.

Jinxyface · April 28, 2021, 8:49pm

RCloneBrowser might run on Linux but I would have no idea how to build/install it on unRAID since it's not a "standard" linux. It can only run apps that are installed from a repo.

I've never checked the rclone GUI itself. I'll have to give that a try