AWS SFTP Service

having an odd issue, It's not actually rclone per-se but might be relevant, when trying to download a 18 gig file via SFTP from amazon's hosted/managed sftp service - at random points any multi-part download fails:

2019-07-08 18:27:26 DEBUG : multi-thread copy: stream 31/50 failed: context canceled Failed to copy: multpart copy: wrote 147456000 bytes but expected to write 372375552
2019-07-08 18:27:32 ERROR : Attempt 1/3 failed with 2 errors and: multpart copy: wrote 147456000 bytes but expected to write 372375552

and when it crashes: Discarding closed SSH connection: read tcp> wsarecv: An existing connection was forcibly closed by the remote host.

again pretty sure it's just aws' sftp implementation so was wondering if anybody has seen anything comparable, thanks

(I've tried this with multiple stream numbers just to see) obviously trying to speed up the transfer

I've had the comparable problems with a few SFTP clients (filezilla, cyberduck) so I know it's not rclone particularly just wondering if anybody else has encountered this particular issue

I tried using cache thinking it might somehow be able to resume the transfer but that didn't seem to help


UPDATE: noticed this gem when the transfer starts:

2019-07-08 18:44:49 DEBUG : Sizes differ (src 18615521247 vs dst 18092523520)
2019-07-08 18:44:51 DEBUG : Starting multi-thread copy with 20 parts of size 887.688M
2019-07-08 18:44:51 DEBUG : multi-thread copy: stream 20/20 (17685348352-18615521247) size 887.082M starting
2019-07-08 18:44:51 DEBUG : multi-thread copy: stream 1/20 (0-930807808) size 887.688M starting


Apologies for the delay in responding, I've been on vacation.

Can you post a complete log with DEBUG? Feel free to redact names of files etc.

I suspect the real error is previous to the one you posted - thanks!

sorry for my own delay, I am actually not working with this client anymore so unfortunately can't help :frowning:

1 Like

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.