One File Copying Much Slower

Help and Support
1

What is the problem you are having with rclone?

When copying a large number of files, I frequently notice that, with four files are moving at any one time, one of them is moving much slower than the rest. There's an example screen shot at /a/oTwA55y on imgur dot com.

What is your rclone version (output from rclone version)

rclone v1.56.2
- os/version: debian 10.11 (64 bit)
- os/kernel: 4.19.0-18-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.16.8
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Jotta right now -- but this also happens on other destinations, including LAN-based storage appliances, and Lima Labs (an SFTP cloud service).

The command you were trying to run (eg rclone copy /tmp remote:tmp)

me@machine:~$ rcl copy -P LAN-Source:Backups/duplicati/ Jotta:duplicati

("rcl" is an alias for rclone that executes my --password-command)

The rclone config contents with secrets removed.

[Jotta]
type = jottacloud
configVersion = 1
client_id = jottacli
client_secret = 
tokenURL = https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/token
token = { REDACTED }
device = 
mountpoint = 
--------------------

A log from the command with the -vv flag

No text log available yet, but see screenshot at /a/oTwA55y on imgur dot com.
2

hello and welcome to the forum,

by default rclone transfers four files at a time, that can be tweaked using --transfers and --checkers

screenshots are not too helpful, best to use a copy/paste rclone output into a post.

3

You may however also be limited by your cloud storage, so do it in small steps and with some caution.

I use OneDrive and it will not do more than 4 transfers at the time, if I set --transfers higher then OneDrive will start throttling my account and actually make my transfers much slower. (Try searching the forum for OneDrive throttling).

I don't know Jottacloud, but parhaps it only allows 3 transfers at a time - and what you are seeing is light throttling.

Probably best to test your Jottacloud, LAN and SFTP individually while copying local data. That will make it easier to find the storage (or network component) limiting your transfers.

4

I would agree with this but for the fact that what I am seeing is a behavior that comes & goes, and is consistently inconsistent across destinations.

That is to say, I sometimes see all four transfers going at multi-megabit speeds.

5

I'll get a log as soon as I can but if you look at my unhelpful screen shot you would see that the current transfer is quite large and has several days to run. I'm reluctant to stop it without a known-useful fix.

6

with some backends, such as s3, rclone must perform a checksum before the uploads starts.
so uploading a large file will take extra time.

using a log, you can see what rclone is doing......

7

Frustrating, but it is hard to help you when none of us can see a pattern.

Is your source a local disk?

8

all the files are quite close to the same size. That's a property of duplicati.

9

The source this time is a large Samba volume on my LAN. But I also see this behavior from a local SSD and from a locally-attached USB3 spinning-rust drive.

10

the only way to figure this out is to use a debug log.

when you see a particular file is slow, upload the debug log file and we can take a look.

11

OK then it might be a week or so until I can organize that

12

i was not able to find that screenshot, could not figure out the url.
can you copy/paste the screenshot into a post?

13

Sorry - I am not (yet?) allowed to post images here.

Try https :// imgur (dot) com /a/oTwA55y

14

could it be that you are running out of bandwidth?
can you post the output of a speedtest, such as speedtest.net?

15

i see that they support rclone.
if you transfer a file to them, is rclone able to checksum the file?

16

Speedtest by Ookla

     Server: Empire Access - Prattsburgh, NY (id = 24883)
        ISP: Level 3 Communications
    Latency:    27.20 ms   (0.89 ms jitter)
   Download:   476.14 Mbps (data used: 511.5 MB)                               
     Upload:   462.24 Mbps (data used: 696.8 MB)                               
Packet Loss:     0.6%
 Result URL: https://www.speedtest.net/result/c/110086ee-e64b-46d5-bde2-2cd65581d7e8
17

Yes, although Lima only supports MD5 :frowning:

18

Duplicati used to support Jottacloud as destination, but perhaps not with the new authentication methods? Or do you deliberately use a local disk as destination and want to use rclone to upload a second copy to cloud?

In my experience I can set transfers quite high, even as high as 16, and see increase in total upload speed. Depending on the data, of course. Don't think it leads to throttling.

19

This.

I'm throttled at Jotta because of total volume but this problem is not restricted to Jotta. I am not throttled at Lima. And certainly not on the QNap in my basement.

Also: in a throttling situation, I would still expect the four transfers to be running at similar-ish speeds. Not three at X and one at 0.1*X

20

off topic.....

About us

We’re a small family business based in Winchester Va. Our servers are in Va, Canada, the UK, France, Poland and Australia.
Lima Labs is a CircleNet LLC company.

"Circlenet CEO gets 4 Months in Prison for Hiding Money in Swiss Bank"
http://www.usinternationaltaxadvisors.com/circlenet-ceo-gets-four-months-in-prison-for-hiding-money-in-swiss-bank/

https://www.justice.gov/usao-ndga/pr/atlanta-internet-entrepreneur-convicted-hiding-income-and-assets-swiss-bank-account