Copying sonarr / radarr files to dropbox encrypted are slow

Help and Support
1

What is the problem you are having with rclone?

When I use a rclone copy, everything is good, my connection is at 60mo/s like everytime, but when *arr are doing their job of moving files after a local download, the upload is between 5 and 8 mo/s. I tried so much configurations using systemd but nothing seems to solve this.

Run the command 'rclone version' and share the full output of the command.

rclone v1.61.1

  • os/version: ubuntu 20.04 (64 bit)
  • os/kernel: 5.15.0-58-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.19.4
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Dropbox with a crypt remote

The command you were trying to run (eg rclone copy /tmp remote:tmp)

[Unit]
Description=Encrypted Dropbox (rclone)
AssertPathIsDirectory=/volume1/Plex
Before=network-online.target docker.socket firewalld.service containerd.service docker.service

[Service]
CPUQuota=60%
Type=simple
ExecStart=/usr/bin/rclone mount \
        --config=/root/.config/rclone/rclone.conf \
        --allow-other \
        --vfs-cache-mode full \
        --vfs-cache-max-size 200G \
        --dir-perms 0777 \
        --file-perms 0777 \
        --umask 000 \
        --tpslimit 12 \
        --cache-chunk-path=/volume1/tmp/rclone/chunks \
        --cache-workers=8 \
        --cache-writes \
        --cache-dir=/volume1/tmp/rclone/vfs \
        --cache-db-path=/volume1/tmp/rclone/db \
        --no-modtime \
        --stats=0 \
        --dir-cache-time=60m \
        --cache-info-age=60m encrypt:/Plex /volume1/Plex
ExecStop=/bin/fusermount -u /volume1/Plex
Restart=always
RestartSec=10

[Install]
WantedBy=default.target

The rclone config contents with secrets removed.

[encrypt]
type = crypt
remote = dropbox:/<my name>
password = **CONFIDENTIAL**
password2 = **CONFIDENTIAL**

[dropbox]
type = dropbox
client_id = **CONFIDENTIAL**
client_secret = **CONFIDENTIAL**
token = **CONFIDENTIAL**

A log from the command with the -vv flag

I got a 5mo file... it can't go on pastebin. Published it here :
https://1fichier.com/?ga8bv9509ic7gztnsju3

2

There is a bunch of stuff that does nothing on your mount.

That's all for the cache backend which you aren't using.

Dropbox is a polling remote so making that 60 minutes makes it poll more. Ideally, you want to set that huge.

Why no modtime?

Can you share some output of a rclone copy vs the mount speed?

3

I am not too familiar with this tool, so I used a config that I found on the internet and modified it to fit with what I wanted.

I removed all of what you said (all cache) and don't really understand what you are saying about the --dir-cache-time=60m.

--no-modtime was because I read this Don't read/write the modification time (can speed things up) on the mount page

About the output of the file transfer speed, I can do it for the copy using --progress, but how can I do this using the log file in a mount ?

4

A polling remote means it will automatically detect changes.

I use:

--vfs-cache-max-age 9999h

That will make Plex pretty annoyed without mod times and you'll end up rescanning things quite a bit.

How about we start there? Share the output from a rclone copy.

5

I just checked and I think the problem can be outside rclone.
On my NAS, I got 60mo/s download and upload using rclone copy. But on my ubuntu computer, I got 60mo/s in download but on upload I got 18mo/s and it decrease to 5mo/s after few seconds

6

What's a mo/s?

What's your internet rated at?

7

oh yeah sorry, mo/s is the french conversion of MB/s

my internet connection is rated at 500Mb/s

8

So it's tough to compare a download to an upload because downloads are generally multi streamed so you get 4 streams downloading.

I tend to max out my gigabit download without much of an issue.

[felix@gemini data]$ rclone copy DB:jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv . -P -vv
2023/01/17 15:42:37 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2023/01/17 15:42:37 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "copy" "DB:jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv" "." "-P" "-vv"]
2023/01/17 15:42:37 DEBUG : Creating backend with remote "DB:jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv"
2023/01/17 15:42:37 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2023/01/17 15:42:37 DEBUG : fs cache: adding new entry for parent of "DB:jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv", "DB:"
2023/01/17 15:42:37 DEBUG : Creating backend with remote "."
2023/01/17 15:42:37 DEBUG : fs cache: renaming cache item "." to be canonical "/data"
2023-01-17 15:42:37 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: Need to transfer - File not found at Destination
2023-01-17 15:42:37 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: Starting multi-thread copy with 4 parts of size 358.812Mi
2023-01-17 15:42:37 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: multi-thread copy: stream 4/4 (1128726528-1504953150) size 358.798Mi starting
2023-01-17 15:42:37 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: multi-thread copy: stream 2/4 (376242176-752484352) size 358.812Mi starting
2023-01-17 15:42:37 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: multi-thread copy: stream 3/4 (752484352-1128726528) size 358.812Mi starting
2023-01-17 15:42:37 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: multi-thread copy: stream 1/4 (0-376242176) size 358.812Mi starting
2023-01-17 15:42:52 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: multi-thread copy: stream 2/4 (376242176-752484352) size 358.812Mi finished
2023-01-17 15:42:53 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: multi-thread copy: stream 1/4 (0-376242176) size 358.812Mi finished
2023-01-17 15:42:53 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: multi-thread copy: stream 3/4 (752484352-1128726528) size 358.812Mi finished
2023-01-17 15:42:53 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: multi-thread copy: stream 4/4 (1128726528-1504953150) size 358.798Mi finished
2023-01-17 15:42:53 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: Finished multi-thread copy with 4 parts of size 358.812Mi
2023-01-17 15:42:57 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: dropbox = b4c95cf116a73804e5edbf37975ffb8759ab4942cf362c2859a60bf6fdac94b4 OK
2023-01-17 15:42:57 INFO  : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: Multi-thread Copied (new)
Transferred:   	    1.402 GiB / 1.402 GiB, 100%, 73.985 MiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:        19.8s
2023/01/17 15:42:57 INFO  :
Transferred:   	    1.402 GiB / 1.402 GiB, 100%, 73.985 MiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:        19.8s

2023/01/17 15:42:57 DEBUG : 15 go routines active
2023/01/17 15:42:57 INFO  : Dropbox root '': Committing uploads - please wait...

But uploads are not multithreaded so it's only 1 stream.

[felix@gemini data]$ rclone copy jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv DB: -P
Transferred:   	    1.402 GiB / 1.402 GiB, 100%, 34.149 MiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:        42.0s

I tend to get about 30-35 MB/s per stream in my experience from Dropbox on a dedicated server and a very managed/dedicated Verizon FIOS fiber connection from my house.

18 to 5 seems strange as is it a home internet or something else? Do they do any throttling there?

9

Yeah it is a home connection. And nope, nothing is on the server except my NAS and my computer, and nobody is playing any console at home.
Weird but if for you that's normal the "low" upload speed, then it must surely be normal.

10

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.