Slow upload speeds, or complete failure

Help and Support
1

I've seen multiple threads about slow upload speed, and have read through them thouroughly, but I haven't found anything that seems to help my specific scenario. I have scoured rclone forums, and google but I haven't been able to find anything useful. I'm hoping somebody here could shed some light.

What is the problem you are having with rclone?

The problem seems to be with a combination of uploading, combined with the mounting. Either I get really slow upload speeds (3.8MB/s), or if I change the code a little bit, I can get 25MB/s but it seems like the drive unmounts itself and then the copying fails.

What is your rclone version (output from rclone version)

rclone v1.52.0
- os/arch: windows/amd64
- go version: go1.14.3

Which OS you are using and how many bits (eg Windows 7, 64 bit)

The issue happens both on my HTPC (Plex server) with modest system hardware (Win 7 x64, 8gb RAM, built in GPU), or even on my gaming rig (Win10 x64, 32GB ram, Ryzen 7 2700x, RTX 2080 GPU). I have a 400 MB down / 50 MB up internet connection.

Which cloud storage system are you using? (eg Google Drive)

Google Team Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount --allow-other gcrypt: X: and
rclone copy --progress F:\Movies X:\Movies yields a reliable 3.8MB/s, and the upload completes but it takes over 50 hours to upload 750GB.

rclone mount gcrypt: X: --allow-other --cache-db-purge --buffer-size 32M --use-mmap --dir-cache-time 72h --drive-chunk-size 16M --timeout 1h --vfs-cache-mode minimal --vfs-read-chunk-size 128M --vfs-read-chunk-size-limit 1G & with the same copy command yields speeds that vary from 40MB/s to 25MB/s, but it always fails aprox 10-20 minutes into the uploads. It starts to read a bunch of errors that it can not access the files, and the X: drive is visible in "This PC" but when you click it, it is unable to load or open.

(Admittedly, I found this code online and I'm not clear exactly on what it does)

The rclone config contents with secrets removed.

type = drive
client_id = REDACTED
client_secret = REDACTED
scope = drive
use_trash = false
chunk_size = 16M
token =REDACTED

[gcache]
type = cache
remote = gdrive:/gdrive
plex_url = http://10.0.0.202:32400
plex_username = REDACTED
plex_password = REDACTED
chunk_size = 16M
info_age = 2d
chunk_total_size = 20G
plex_token = REDACTED

[gcrypt]
type = crypt
remote = gcache:
password = REDACTED
password2 = REDACTED

A log from the command with the -vv flag

f:\Users\XXX\Downloads\rclone>rclone copy \\HTPC-PC\Movies\2 X:\Movies\2 --progress -vv
2020/06/11 20:12:32 DEBUG : rclone: Version "v1.52.0" starting with parameters ["rclone" "copy" "\\\\HTPC-PC\\Movies\\2" "X:\\Movies\\2" "--progress" "-vv"]
2020/06/11 20:12:32 DEBUG : Using config file from "C:\\Users\\XXX\\.config\\rclone\\rclone.conf"
2020/06/11 20:12:32 DEBUG : fs cache: renaming cache item "\\\\HTPC-PC\\Movies\\2" to be canonical "//?/UNC/HTPC-PC/Movies/2"
2020/06/11 20:12:32 DEBUG : fs cache: renaming cache item "X:\\Movies\\2" to be canonical "//?/X:/Movies/2"
2020-06-11 20:12:32 DEBUG : Local file system at //?/X:/Movies/2: Waiting for checks to finish
2020-06-11 20:12:32 DEBUG : Local file system at //?/X:/Movies/2: Waiting for transfers to finish
2020-06-11 20:24:21 DEBUG : Madagascar (2005).mkv: MD5 = 231e560927c9c3665b9eaf6a2799252a OK
2020-06-11 20:24:21 INFO  : Madagascar (2005).mkv: Copied (new)
2020-06-11 20:36:38 DEBUG : Madagascar 2 - Escape 2 Africa (2008) 1080p.mkv: MD5 = eeffc6424d3296e652e55a285f502f71 OK
2020-06-11 20:36:38 INFO  : Madagascar 2 - Escape 2 Africa (2008) 1080p.mkv: Copied (new)
2020-06-11 20:44:26 DEBUG : LOTR4 - The Hobbit - An Unexpected Journey (2012) 1080p.mkv: MD5 = ef85ecbcac69576fc2305fb20de1f71c OK
2020-06-11 20:44:26 INFO  : LOTR4 - The Hobbit - An Unexpected Journey (2012) 1080p.mkv: Copied (new)
2020-06-11 20:45:00 DEBUG : LOTR5 - The Hobbit - The Desolation Of Smaug 1080p (2013).mkv: MD5 = db3e87d5d94a8017aa196faaaa5a2ff1 OK
2020-06-11 20:45:00 INFO  : LOTR5 - The Hobbit - The Desolation Of Smaug 1080p (2013).mkv: Copied (new)
2020-06-11 20:52:24 ERROR : Madagascar 3 Europe's Most Wanted (2012) 1080p.mkv: Failed to copy: chtimes \\?\X:\Movies\2\Madagascar 3 Europe's Most Wanted (2012) 1080p.mkv: The I/O operation has been aborted because of either a thread exit or an application request.
Transferred:       57.210G / 738.549 GBytes, 8%, 24.100 MBytes/s, ETA 8h2m29s
Errors:                 1 (retrying may help)
Transferred:            4 / 95, 4%
Elapsed time:     40m30.8s
Transferring:
 *         Mission Impossible 1 (1996) 1080P.mkv:  0% /9.108G, 0/s, -
 * LOTR6 - The Hobbit - T…rmies 1080p (2014).mkv: 95% /14.963G, 0/s, 0s
 * Mighty.Morphin.Power.R….DD5.1.H.264-AJP69.mkv: 35% /2.637G, 0/s, 2001635h15m8.908383744s
 * man.of.steel.2013.1080…luray.x264-sector7.mkv: 12% /9.830G, 0/s, 840493h39m48.0896768s
2

hello and welcome to the forum,

  • is there a specific reason you are using the gcache? 99.99% of the time, there is on need for it. i have never used it and i stream from a mount every day.
  • is there a specific reason you are copying the the mounted drive and not straight to gcrypt"
    rclone \\HTPC-PC\Movies\2 gcrypt:
  • what version of winfsp are you using? make sure to use the latest stable version
3

I am using a cache because I have no clue why. I did it because all the tutorials I read online say to do so. I didn't think to use gcrypt: as the copy location instead of X:, I wonder if that will make any difference. I am using the latest version of winfsp as I just installed it, and learned about rclone this week. Any help is appreciated.

I just tried forgoing the mount, and just using the copy command

rclone copy \\HTPC-PC\Movies\3 grcypt:\3

I get transfer rates between 3.9-4.0 MB/s. It doesn't even make sense. I am currently 40+ hours into a transfer on my HTPC using the slow but reliable method I mentioned in the OP. I tried the mountless copy on my gaming PC (the one with 32GB RAM in the OP), which is getting 4MB/s. But the transfers on the HTPC haven't slowed down at all... so effectively I am transferring at ~8MB/s because I am using 2 PCs coming from the same IP. What is up with that?m If google was throttling me, wouldn't the # of PCs not matter.

4

https://rclone.org/cache/#status

based on your config, you are still using the cache
gcrypt->gcache->gdrive

please try gcrypt->gdrive

there many outdate and lame "guides" on reddit and elsewhere that still recommened using a cache.
can you post the link you used?

5

You're right it would still be using the cache. Is the cache really not recommended now? Everything I read is debatable at best? I just tried to post links, and I got an error saying "Sorry you cant include links in your post" So I just copied it into paste bin

https://pastebin.com/qX5jxq2j

6

there is not need for the cache, it is the cause of many problems here at the forum.

if you want to use it, good luck.

7

I am definitely open to forgoing the cache. I thought the cache was used for 1) better performance 2)to prevent Plex from bombarding the GDrive and getting a temp ban when scanning.

Is there a way to take everything out from the cache easily without having to re-upload everything? At this point I am somewhere near 2TB of data on the drive. I wouldn't want to delete everything just to have the same upload speeds and have to repeat.

Thank you for your help. I really am open to any suggestions, I am not fixed on anything 1 way or another.

8

no need to re-upload.
we are going to add a new remote to the config file
for sure, no deletions, so you current setup will continue to work as is.

for gcrypt, do you have password and password2?

9

yup the encryption key and the salt. I have them both from what I entered in the actual cmd config, and I can get the encrypted part from the actual config file if needed.

10

we need to find the root of the crypt on gdrive.

do a
rclone lsd gdrive:
and
rclone lsd gdrive:gdrive

11

rclone lsd gdrive: returns:
-1 2020-06-05 00:08:19 -1 gdrive
rclone lsd gdrive:gdrive returns:

          -1 2020-06-06 16:14:15        -1 6nu5akjq0k1v3udplr4igmammo
          -1 2020-06-13 12:47:33        -1 eqij9s2a05v8hqq9ai81sb4kuk
          -1 2020-06-06 16:14:27        -1 u9mn241g0j1h03efhvk0lkcnrs
12

rclone lsd gcrypt:

13 
      -1 2020-06-06 16:14:15        -1 Movies
      -1 2020-06-06 16:14:27        -1 TV Shows

sorry there was another folder that I deleted between those 2, thats why it doesnt match. the 06/13 folder was just deleted. it was an empty folder

14

good, as i was about to tell you i was confused as to the root of the crypt, due to a mismatch on folder count.

to be sure,
rclone lsd gdrive:gdrive

15 
      -1 2020-06-06 16:14:15        -1 6nu5akjq0k1v3udplr4igmammo
      -1 2020-06-06 16:14:27        -1 u9mn241g0j1h03efhvk0lkcnrs
16

good, that matches.

create a new remote, perhaps call it gcrypt2
set the remote to gdrive:gdrive
and for the rest of the config values, use the exact same settings from gcrypt
then do a rclone lsd gcrypt2:

17 
      -1 2020-06-06 16:14:15        -1 Movies
      -1 2020-06-06 16:14:27        -1 TV Shows
18

so that worked!

now let's test the speed, do something like.
rclone c:\path\to\local\folder gcrypt2:testfolder --progress -v

19 
f:\Users\Jeff\Downloads\rclone>rclone copy \\HTPC-PC\Movies\3 gcrypt:\test --progress -v
2020/06/13 14:25:16 INFO  : gcache: Cache DB path: C:\Users\Jeff\AppData\Local\rclone\cache-backend\gcache.db
2020/06/13 14:25:16 INFO  : gcache: Cache chunk path: C:\Users\Jeff\AppData\Local\rclone\cache-backend\gcache
2020/06/13 14:25:16 INFO  : gcache: Chunk Memory: true
2020/06/13 14:25:16 INFO  : gcache: Chunk Size: 16M
2020/06/13 14:25:16 INFO  : gcache: Chunk Total Size: 20G
2020/06/13 14:25:16 INFO  : gcache: Chunk Clean Interval: 1m0s
2020/06/13 14:25:16 INFO  : gcache: Workers: 4
2020/06/13 14:25:16 INFO  : gcache: File Age: 2d
2020/06/13 14:25:17 INFO  : gcache: Cache DB path: C:\Users\Jeff\AppData\Local\rclone\cache-backend\gcache.db
2020/06/13 14:25:17 INFO  : gcache: Cache chunk path: C:\Users\Jeff\AppData\Local\rclone\cache-backend\gcache
2020/06/13 14:25:17 INFO  : gcache: Chunk Memory: true
2020/06/13 14:25:17 INFO  : gcache: Chunk Size: 16M
2020/06/13 14:25:17 INFO  : gcache: Chunk Total Size: 20G
2020/06/13 14:25:17 INFO  : gcache: Chunk Clean Interval: 1m0s
2020/06/13 14:25:17 INFO  : gcache: Workers: 4
2020/06/13 14:25:17 INFO  : gcache: File Age: 2d
2020-06-13 14:26:16 INFO  : gdrive/d0kfv67dk87s55rnc4tlpdjai0: received cache expiry notification
2020-06-13 14:26:18 INFO  : : received cache expiry notification
2020-06-13 14:28:01 INFO  : Signal received: interrupt
2020-06-13 14:28:01 INFO  : plex: stopped Plex watcher
2020-06-13 14:28:01 INFO  : plex: stopped Plex watcher
2020-06-13 14:28:01 INFO  : Exiting...
Transferred:      651.589M / 146.071 GBytes, 0%, 3.992 MBytes/s, ETA 10h21m43s
Transferred:            0 / 18, 0%
Elapsed time:      2m43.2s
Transferring:
 *              The Expendables (2010) 1080p.mkv:  1% /9.095G, 1.043M/s, 2h26m15s
 *            The Expendables 2 (2012) 1080p.mkv:  2% /8.118G, 1.004M/s, 2h15m13s
 *   The Fast and The Furious 1 (2001) 1080p.wmv:  2% /7.733G, 1017.650k/s, 2h10m2s
 * Donnie Brasco (1997) E…BluRay x265 afm72).mkv:  2% /5.746G, 982.899k/s, 1h39m23s

seems to be about the same rate.

20

i thought that new remote was named gcrypt2: