Rclone copy from ftp to s3 gets an error code: `SlowDown`

I'm trying to sync a lot of files from an FTP server to a Minio bucket.

Everything seems to work just fine, but after an hour or so the following messages appear in the rclone log:

2020/04/19 17:07:20 ERROR : Attempt 2/3 failed with 8 errors and: s3 upload: 503 Service Unavailable: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SlowDown</Code><Message>Please reduce your request</Message><Key>some-random-dir/PREFIX7037/log_200419_182143.zip</Key><BucketName>vilmos-test-rinfra-637</BucketName><Resource>/vilmos-test-rinfra-637/some-random-dir/PREFIX7037/log_200419_182143.zip</Resource><RequestId>16074779FBE60FED</RequestId><HostId>b2c91653-fdbe-466d-8aed-ad791160bc01</HostId></Error>
2020/04/19 17:07:22 INFO  : 
Transferred:   	   39.718M / 39.718 MBytes, 100%, 138.227 kBytes/s, ETA 0s
Checks:               868 / 868, 100%
Elapsed time:      4m54.2s

2020/04/19 17:08:22 INFO  : 
Transferred:   	   39.718M / 39.718 MBytes, 100%, 138.227 kBytes/s, ETA 0s
Checks:               868 / 868, 100%
Elapsed time:      4m54.2s

2020/04/19 17:09:22 INFO  : 
Transferred:   	   39.718M / 39.718 MBytes, 100%, 138.227 kBytes/s, ETA 0s
Checks:               868 / 868, 100%
Elapsed time:      4m54.2s

2020/04/19 17:10:22 INFO  : 
Transferred:   	   39.718M / 39.718 MBytes, 100%, 138.227 kBytes/s, ETA 0s
Checks:               868 / 868, 100%
Elapsed time:      4m54.2s

^[2020/04/19 17:11:19 ERROR : PREFIX3052/log_200419_182057.zip: Failed to copy: s3 upload: 503 Service Unavailable: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SlowDown</Code><Message>Please reduce your request</Message><Key>some-random-dir/PREFIX3052/log_200419_182057.zip</Key><BucketName>vilmos-test-rinfra-637</BucketName><Resource>/vilmos-test-rinfra-637/some-random-dir/PREFIX3052/log_200419_182057.zip</Resource><RequestId>160747BEE4CA0A51</RequestId><HostId>b2c91653-fdbe-466d-8aed-ad791160bc01</HostId></Error>
2020/04/19 17:11:22 INFO  : 
Transferred:   	   40.056M / 40.067 MBytes, 100%, 131.754 kBytes/s, ETA 0s
Errors:                 1 (retrying may help)
Checks:               868 / 868, 100%
Transferred:            0 / 1, 0%
Elapsed time:      5m11.3s
Transferring:
 *              PREFIX3118/log_200419_182456.zip:  0% /11.578k, 0/s, -

2020/04/19 17:11:35 ERROR : PREFIX3118/log_200419_182456.zip: Failed to copy: s3 upload: 503 Service Unavailable: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SlowDown</Code><Message>Please reduce your request</Message><Key>some-random-dir/PREFIX3118/log_200419_182456.zip</Key><BucketName>vilmos-test-rinfra-637</BucketName><Resource>/vilmos-test-rinfra-637/some-random-dir/PREFIX3118/log_200419_182456.zip</Resource><RequestId>160747C29EA6EC8D</RequestId><HostId>b2c91653-fdbe-466d-8aed-ad791160bc01</HostId></Error>
2020/04/19 17:12:22 INFO  : 
Transferred:   	   40.135M / 40.135 MBytes, 100%, 127.034 kBytes/s, ETA 0s
Errors:                 2 (retrying may help)
Checks:               868 / 868, 100%
Elapsed time:      5m23.5s

The command I run is:

rclone -v copy --include PREFIX*/log_*.zip ftp-name:dir/in-ftp minio:vilmos-test-rinfra-637/some-random-dir

Is there a way to set the rate-limit on the minio component? A lot of these files (100+) gets copied after all, but it's a bit hard to debug my rclone flow, if the minio throws these errors all the time. I don't mind if the copy will happen slower.

hello,
this does not look like a bug,
it would have been better to post using the question template, not the bug template.

you need to tweak your rclone command
if the minio, your own server or hosted?
if hosted, by whom?

slowdown, that is a s3 error code.

Error Code Description HTTP Status Code SOAP Fault Code Prefix

SlowDown Reduce your request rate. 503 Slow Down Server

you try this flag to reduce transactions per second
https://rclone.org/docs/#tpslimit-float

or number of transfers run in parallel.
https://rclone.org/docs/#transfers-n

sorry about the category, i've recategorized this post.

the minio server is hosted by my organization, but i'd be nicer to tweak the rclone command (set the rate limit on the rclone side, or whatsoever) rather than tweaking the minio server. is there a way to do that?

yes, in my post above, i gave you two options

I think the problem is you don't really know what to slow down. Do you? Transactions per second/minute/hour? Total transferred data? Api hits? Etc.

Is there published limits? Both parameters @asdffdsa said are correct but it would be nice if you knew what target you're trying to hit.

@asdffdsa Thanks for the options, I've missed those in your post. My bad :slightly_frowning_face:

I'll talk whoever operates our minio server, and come up with a solution, but I think I have every info from the rclone side :slightly_smiling_face:

Thanks again.

sure, glad to help.
let us know how things work out.

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.