Sync starts well, but then slows down and comes to a halt - outgoing port problem?

Suspected Bug
1

What is the problem you are having with rclone?

The sync starts fine at a very good speed. But then it gradually slows until a complete halt (0b/s).
The provider says it can be nothing but rclone not closing connections, so I've added a couple of parameters to handle connection closure - without any success.

However. After studying the logs, I've noticed an interesting pattern. There are no significant errors until rclone decides to use port 60000+. First time it's port 64009:

2023/11/26 08:58:06 DEBUG : pacer: low level retry 1/10 (error write tcp 192.168.0.105:64009->81.171.6.176:21: wsasend: An existing connection was forcibly closed by the remote host.)
2023/11/26 08:58:06 DEBUG : pacer: Rate limited, increasing sleep to 20ms

Then a few successful connections - and again:

2023/11/26 08:58:08 DEBUG : pacer: low level retry 1/10 (error write tcp 192.168.0.105:64009->81.171.6.176:21: wsasend: An existing connection was forcibly closed by the remote host.)
2023/11/26 08:58:08 DEBUG : pacer: Rate limited, increasing sleep to 20ms

This goes on for a while, and then it tries to use 64012 - unsuccessfully:

2023/11/26 08:58:15 DEBUG : pacer: low level retry 1/10 (error write tcp 192.168.0.105:64012->81.171.6.176:21: wsasend: An existing connection was forcibly closed by the remote host.)
2023/11/26 08:58:15 DEBUG : pacer: Rate limited, increasing sleep to 151.875ms
2023/11/26 08:58:16 DEBUG : pacer: low level retry 2/10 (error write tcp 192.168.0.105:64009->81.171.6.176:21: wsasend: An existing connection was forcibly closed by the remote host.)
2023/11/26 08:58:16 DEBUG : pacer: Rate limited, increasing sleep to 303.75ms

Over time, the amount of "faulty" ports grows - but rclone doesn't seem to consider them so, instead, it starts uses them indefinitely, which ultimately comes to hours of banging against the wall with the same ports, like this:

[Album] imgur.com

Run the command 'rclone version' and share the full output of the command.

rclone v1.62.2
- os/version: Microsoft Windows 10 Pro 22H2 (64 bit)
- os/kernel: 10.0.19045.3324 Build 19045.3324.3324 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.20.2
- go/linking: static
- go/tags: cmount

(I know it's not the latest, but I updated and tried on a latest one with the same result.)

Which cloud storage system are you using? (eg Google Drive)

FTP (hostingbydesign)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone.exe sync !remote!:!item! %backupPath%!item! -vv --check-first --checkers 5 --transfers 3 --ftp-concurrency 6 --ftp-idle-timeout 0m10s --ftp-close-timeout 0m20s

The rclone config contents with secrets removed.

[tmn_nl]
type = ftp
host = bouncer.ams.nl.seedbox.io
user = 
pass = 
tls = false
explicit_tls = true

A log from the command with the -vv flag

2

welcome to the forum,

fwiw, i have been using seedbox.io for years; given they support sftp and webdav, never once tried ftp.

3

Thanks! :wave:t2:
Now this is interesting. I haven't found any mention of webdav in any of their knowledgebases or on Discord. Welcome mail doesn't mention it either. How do you use it exactly?
As for SFTP, I'll have to try it, however I have a feeling it won't cure this particular problem.

4

yes, very interesting!

rclone config redacted seedboxio.dav:
[seedboxio.dav]
type = webdav
url = https://redacted.seedbox.io/dav/
vendor = other
user = XXX
pass = XXX

rclone lsl seedboxio.dav:
        0 2023-11-26 14:57:13.000000000 file.ext
5

Just checked, yeah, SFTP made it even worse. Getting stuck almost immediately.

2023/11/26 21:59:54 DEBUG : pacer: low level retry 4/10 (error couldn't connect SSH: dial tcp 178.162.148.111:22: connectex: No connection could be made because the target machine actively refused it.)
6

Pardon me asking, but what exactly do you use as a hostname there? The main address of the server you're located at, or a bouncer? And are you on shared plan?

7

https://redacted.seedbox.io/dav/

yes

no bouncers. i am using webdav and sftp.

here is the output the most recent rclone move command, where source is seedboxio.dav:

Transferred:        2.840 GiB / 2.840 GiB, 100%, 91.703 MiB/s, ETA 0s
Checks:                18 / 18, 100%
Deleted:                9 (files), 5 (dirs)
Renamed:                9
Transferred:            9 / 9, 100%
Elapsed time:        31.9s
8

That's a good speed right there.
However this must mean that you probably use some of the servers from their previous tariffs, because the current servers are on "itsby.design" domain (seedbox.io I've provided in the first post is the bouncer); I tried mine with webdav and get an error.

2023/11/26 22:09:17 ERROR : webdav root '.test': error reading source root directory: directory not found
9

Actually I have to take that one back. Turns out SFTP has separate parameters for thread limitation; after change, the process is going strong for 40 minutes already.

The initial problem, I think, should still be addressed though.

10

Did a bit of debugging with the provider. They say straight away that I have too many open connections. I quote,

You are hitting the limits on connections, both on SFTP and on the FTPS daemon, I see you at one point had over 200 connection attempts at the same time to the bouncer server - That might be an rclone bug, I am not sure but that is all I can see on our end which is causing your timeouts as our system wont accept that many connections.

Even though I had --ftp-concurrency 3. Here is the rclone log. You can see how the speed is more or less solid for an hour, and then quickly drops to 0 with errors:

multi-thread copy: chunk 105/579 failed: multi-thread copy: failed to open source: open: context canceled

Sharing from pcloud because the log is too big for pastebin.

11

as i mentioned, i have used seedbox.io for 4+ years, never had issues.
i never used ftp, never used bounce servers, i try to avoid sftp.

ftp/sftp are very old, limited protocols. i use webdav

here is a log snippet from a recent download from seedbox.io using webdav

Transferred:        5.262 GiB / 5.262 GiB, 100%, 89.722 MiB/s, ETA 0s
Checks:                16 / 16, 100%
Deleted:                8 (files), 7 (dirs)
Renamed:                8
Transferred:            8 / 8, 100%
Elapsed time:        59.1s
12

One thing I can do is actively envy your speed once again, but that's it; I have no ability to use webdav, because like I suspected, it's only available on older plans. As they confirmed on Discord, I quote,

Webdav was supported on the old packages, its not been included in the packages thats been sold in the past +2 years

So, I guess their opinion on which protocol is outdated is different from yours ¯_(ツ)_/¯ Not arguing here, I don't have any opinion on the matter, simply stating the fact.

So, it very much means that in terms of rclone, I'm stuck with one of two options:

  • SFTP to the main server - it used to work for a while, like I said before, but it doesn't any more; now I don't even have patience to see it through the checks stage. Like, ~150 files checked in over an hour, c'mon man. 2023-11-29_14-21-22.txt - pCloud

  • FTP to the bouncer: brings up the problems with unclosed connections, sometimes sooner, sometimes later, but at some point I end up with 0b/s. 2023-11-29_13-05-31.txt - pCloud

I mean I could still use something else, but with rclone's unparalleled configurability and attention to detail, I'd rather use it instead of something else.

13

ok, good to know.

is there is a rclone issue or a seedbox.io sftp issue?
have you tested another sftp tool, does it also slow down and come to a halt?

my plan also offers vpn access to the files, i have not used it in a long time but i remember getting good results.

14

here's my two pennies worth comment. I have to use sftp often for various servers (often low spec) and I have never seen issues like yours. Also using rclone - mostly using defaults settings. Sometimes running overnight transfers lasting many hours.

Your sftp provider has some very serious limits or issues with handling load IMO. Maybe rclone could do something better here.. not sure. If it was some serious bug in ftp/sftp implementation we would hear about it a lot from many people - but we do not.

16

I'd say these are two little bit different issues (SFTP vs FTP), I compared with how WinSCP performs - while definitely not a tool for sync, it shows stable speed. Not as high as I'd like (but that's a matter to discuss with my ISP), sometimes lower, sometimes higher, but no other tool has shown either "coming to a complete halt" or "reading checksums way too slowly" problems.

17

I understand, what you say makes a lot of sense, but frankly I don't know what to think any more. The provider has been very persuasive in regards of "we have no reason to throttle your speed" argument, and they've clearly said that I've had more than 200 open connections at some point. It's very possible that it's me doing something wrong - so I'd appreciate if someone could show me where to look for my error.

18

Make sure you are using rclone 1.65 as there was a bug with SFTP opening too many connections which is fixed there.

1 Like
19

Oh. You know what, I use the previous one. I'll update straight away. Please tell me, is there something I should do to close the previous connections?

20

Stopping rlcone should close all open connections.

21

OK, so here's the situation post-update:

SFTP: seems to perform much better, first 149 checks are done within just a minute, but then gets hopelessly stuck as before. 2023-11-29_18-40-06.txt - pCloud

FTP: breezes through all 700+ checks within seconds just like it did before. Within minutes transfer speed degrades to 0. 2023-11-29_18-55-14.txt - pCloud

So, unfortunately, update brought no changes in my case.

UPD: for an unknown reason pastebin removes my logs, so I replaced the links with my cloud.