Sync starts well, but then slows down and comes to a halt - outgoing port problem?

[kapitainsky]

Try to limit number and "depth" of checks:

--checkers 1 --max-backlog 10

Maybe default checkers settings are too aggressive for your server.

Also --tpslimit works with the sftp backend so you can try to throttle it, e.g.: --tpslimit 10 for the start

[asdffdsa (jojothehumanmonkey)]

@Basil_Vlasyuk

hopelessly stuck as before

that sftp logs looks ok to me. rclone is not stuck, no errors, no issues, no warning, no pacers, no drop packets, etc..
the cheap seedbox server is the problem.

sftp is very slow protocol, has high latency when there are a lot of files to compare.
in addition, rclone is calculating checksums and that takes time.
for testing, can disable using --ignore-checksum

at seedbox, what plan are you on, do you have access to ssh/shell?

[Basil_Vlasyuk]

I'll try that, thanks. However I don't use defaults; here is what I do use now:

rclone.exe sync !remote!:!item! %backupPath%!item! --log-file !logFile! --log-level DEBUG --check-first --checkers 2 --transfers 2 --sftp-concurrency 3  --track-renames

[Basil_Vlasyuk]

the cheap seedbox server is the problem.

I mean it's a simple explanation, but there's no evidence to support that this is the problem. They wouldn't just make up that I have hundreds of unclosed connections to avoid saying "it's a cheap seedbox, what do you want?". Right?

In addition, rclone is calculating checksums and that takes time.

Yeah, I get it, but not THIS amount of time. Plus, why do the first ~150 checks run within a minute and then the following ones take several minutes each? And why on the FTP all 700 checks are done within a minute?

for testing, can disable using --ignore-checksum

I'll try that as well, thanks.

at seedbox, what plan are you on, do you have access to ssh/shell?

I have a 4TB shared plan with SSH access.

[asdffdsa (jojothehumanmonkey)]

thanks for the logs, it helps.

@Basil_Vlasyuk

I mean it's a simple explanation, but there's no evidence to support that this is the problem. They wouldn't just make up that I have hundreds of unclosed connections to avoid saying "it's a cheap seedbox, what do you want?". Right?

well, it is a cheap shared seedbox, not a full dedicated machine/vm and it is that slow seedbox that is slow to calculate checksums.

in the end, it is about what flags you choose to use and choose not to use.

@Basil_Vlasyuk

Yeah, I get it, but not THIS amount of time. Plus, why do the first ~150 checks run within a minute and then the following ones take several minutes each?

And why on the FTP all 700 checks are done within a minute?

ftp does not calculate checksums so i assume that is why it is faster.
as for the other ftp issues, no idea. i avoid ftp like the plague, not looking at debug log.

so back to the sftp log and how rclone decides to transfers a file.
rclone can be smart or dumb, it depends on the flags you choose to use or not use.

based on your debug log snippet.

1. rclone does a quick compare of modtime and finds a mismatch
2. now, what is rclone to do?
   a. dumb mode - rclone re-transfer the file.
   b. smart mode - often modtimes might not match, but the files content match.
   rclone ask the slow, shared server to calculate the file checksum.
3. rclone determines the contents match, so no need to re-upload the file, can quickly update the dest modtime.

take the real world example, Community - S02E11 - Abeds Uncontrollable Christmas WEBDL-1080p.mkv

1. Modification times differ by -1h0m8s: 2023-06-14 22:35:08 +0300 EEST, 2023-06-14 21:35:00 +0300 EEST
2. Remote command result: f99d30f1ad8fe63283c84fa2e97f3b3f
3. Updated modification time in destination

in the log snippet, there are a number of mismatch modtimes, forcing rclone to calculate checksums.

did you notice that difference is always the same, off by 1h0m and a few seconds?

• to ignore that, might test --modify-window
• maybe an issue with timezones? if so, fixing that would prevent rclone from calculating checksums in the first place.

---

@Basil_Vlasyuk

I have a 4TB shared plan with SSH access.

might test/try running rclone on the seedbox using rclone serve webdav
and then on the client, create a rclone remote pointing to that webdav server.

[Basil_Vlasyuk]

Wow. That right there is a bunch of useful insights one after the other. Thanks a LOT for taking the time to write this.

well, ftp does not calculate checksums so i assume that is why it is faster.

Had no idea. And yeah, that definitely explains it.

did you notice that difference is always the same, off by 1h0m and a few seconds?

Again, that slipped from my attention - but I have a good explanation, I've had a daytime saving shift recently.

So, for the time being I'm gonna try --ignore-checksum as well as --modify-window. That doesn't fix that FTP thing, but it could just fix the SFTP problem. Thanks again, your input has been tremendously helpful!

[asdffdsa (jojothehumanmonkey)]

@Basil_Vlasyuk

Thanks a LOT for taking the time to write this.

lol, truth is, i did it for Columbo.

in your debug log, i saw Columbo - S04E01 - An Exercise in Fatality
there is a moment in that episode, that will always be frozen in my brain.

so you see, i felt obligated to help you...
https://columbophile.com/2018/05/27/episode-review-columbo-an-exercise-in-fatality/

[Basil_Vlasyuk]

This is literally the first time in my life when I meet another Columbo fan! We are going extinct, I'm afraid
By "the moment" you must mean the showdown, yeah? It's a good one, however my personal favorite is no doubt the one in "Stitch in Crime".

---

I tried something with SFTP: --ignore-checksum, --modify-window 1h1m and, since I read in the docs that ignore-checksum skips post-copy checksum verifying, I also added --size-only. It still ran md5sum for one file for good 30 minutes (not sure why - I thought I disabled checksums for all possible scenarios), and then it finally got to transfers. It was slow and held it for ~1 hour before giving up, but this time there are articulate errors about rate limits. 2023-11-29_22-22-49.txt - pCloud

Now your other advice:

might test/try running rclone on the seedbox using rclone serve webdav

Thinking about it, it's so good it might be illegal. There is no way it can fail; they can't refuse to serve you if you serve yourself.
So it's running now. The speed is, of course, nowhere near yours - ranging anywhere between 3 MB/s and 20 MB/s, but it does run for almost an hour now without failing. So let's see how this one performs.

[Basil_Vlasyuk]

Just minutes after I wrote about this method being sure-fire, I got rate-limited...

[asdffdsa (jojothehumanmonkey)]

fwiw, there are so many seebox companies out there to choose from.

it seems our plans are very different.
i use cheap seedboxes - for downloading torrents. no ssh, no shell, no plex, no streaming.
so the download speeds have to be good.
i have a emby server at hetzner. so when a torrent completes, i move the new torrents from seedboxio to hetzner storagebox.

i think with your plan, they want you to keep the files hosted on their system and use their plex app to stream it.

@Basil_Vlasyuk

Now your other advice:

might test/try running rclone on the seedbox using rclone serve webdav
Thinking about it, it's so good it might be illegal. There is no way it can fail; they can't refuse to serve you if you serve yourself.
So it's running now

sorry, confused, as the log you posted was for sftp, not webdav?

@Basil_Vlasyuk

It still ran md5sum for one file for good 30 minutes (not sure why - I thought I disabled checksums for all possible scenarios),

you can disable checksums by modifying the config file.

[seedboxio.sftp]
type = sftp
host = redcated.seedbox.io
user = redcated
pass = redcated
md5sum_command = none
sha1sum_command = none
shell_type = unix

---

@Basil_Vlasyuk

Stitch in Crime

yeah, but as a life long star trek fan, anytime Leonard Nimoy smiles, i freak out!

[Basil_Vlasyuk]

@asdffdsa

sorry, confused, as the log you posted was for sftp, not webdav?

Yeah, I was quite beat by that time so I just posted sftp link. I tried sftp with several additional arguments and it didn't work out, that's what that log demonstrates.

As for WebDAV situation, after some experiments I realized that it says "rate limited" when there's simply connection trouble - most likely server process is down (which, in turn, happened because I was tampering with VPN, and with every switch ssh terminal would die). So I've created systemd service and the problem didn't appear ever since.

So I'd say my personal case is covered by WebDAV (thank you once again for your suggestion!), and as for the initial problem with FTP, well, it still stands - and it's up to @ncw now to decide if it deserves to be further investigated.

[ncw (Nick Craig-Wood)]

@Basil_Vlasyuk

as for the initial problem with FTP, well, it still stands - and it's up to @ncw now to decide if it deserves to be further investigated.

It looks like rate limiting or firewalling at the provider, so not something rclone has direct control over.

[Basil_Vlasyuk]

I apply heavy limitations on my transfers though. Even with 2 checkers, 2 transfers and strict close timeouts after time I still get rate limited. I've been in touch with the provider and, from what they've seen, I've had hundreds of open connections at a time. They suggest rclone might for some reason not close the connections in this case.

[ncw (Nick Craig-Wood)]

It is possible. You can check what open connections rclone has

Try netstat - something like netstat -tno