I'm trying to sync a lot of files from an FTP server to a Minio bucket.
Everything seems to work just fine, but after an hour or so the following messages appear in the rclone log:
2020/04/19 17:07:20 ERROR : Attempt 2/3 failed with 8 errors and: s3 upload: 503 Service Unavailable: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SlowDown</Code><Message>Please reduce your request</Message><Key>some-random-dir/PREFIX7037/log_200419_182143.zip</Key><BucketName>vilmos-test-rinfra-637</BucketName><Resource>/vilmos-test-rinfra-637/some-random-dir/PREFIX7037/log_200419_182143.zip</Resource><RequestId>16074779FBE60FED</RequestId><HostId>b2c91653-fdbe-466d-8aed-ad791160bc01</HostId></Error>
2020/04/19 17:07:22 INFO :
Transferred: 39.718M / 39.718 MBytes, 100%, 138.227 kBytes/s, ETA 0s
Checks: 868 / 868, 100%
Elapsed time: 4m54.2s
2020/04/19 17:08:22 INFO :
Transferred: 39.718M / 39.718 MBytes, 100%, 138.227 kBytes/s, ETA 0s
Checks: 868 / 868, 100%
Elapsed time: 4m54.2s
2020/04/19 17:09:22 INFO :
Transferred: 39.718M / 39.718 MBytes, 100%, 138.227 kBytes/s, ETA 0s
Checks: 868 / 868, 100%
Elapsed time: 4m54.2s
2020/04/19 17:10:22 INFO :
Transferred: 39.718M / 39.718 MBytes, 100%, 138.227 kBytes/s, ETA 0s
Checks: 868 / 868, 100%
Elapsed time: 4m54.2s
^[2020/04/19 17:11:19 ERROR : PREFIX3052/log_200419_182057.zip: Failed to copy: s3 upload: 503 Service Unavailable: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SlowDown</Code><Message>Please reduce your request</Message><Key>some-random-dir/PREFIX3052/log_200419_182057.zip</Key><BucketName>vilmos-test-rinfra-637</BucketName><Resource>/vilmos-test-rinfra-637/some-random-dir/PREFIX3052/log_200419_182057.zip</Resource><RequestId>160747BEE4CA0A51</RequestId><HostId>b2c91653-fdbe-466d-8aed-ad791160bc01</HostId></Error>
2020/04/19 17:11:22 INFO :
Transferred: 40.056M / 40.067 MBytes, 100%, 131.754 kBytes/s, ETA 0s
Errors: 1 (retrying may help)
Checks: 868 / 868, 100%
Transferred: 0 / 1, 0%
Elapsed time: 5m11.3s
Transferring:
* PREFIX3118/log_200419_182456.zip: 0% /11.578k, 0/s, -
2020/04/19 17:11:35 ERROR : PREFIX3118/log_200419_182456.zip: Failed to copy: s3 upload: 503 Service Unavailable: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SlowDown</Code><Message>Please reduce your request</Message><Key>some-random-dir/PREFIX3118/log_200419_182456.zip</Key><BucketName>vilmos-test-rinfra-637</BucketName><Resource>/vilmos-test-rinfra-637/some-random-dir/PREFIX3118/log_200419_182456.zip</Resource><RequestId>160747C29EA6EC8D</RequestId><HostId>b2c91653-fdbe-466d-8aed-ad791160bc01</HostId></Error>
2020/04/19 17:12:22 INFO :
Transferred: 40.135M / 40.135 MBytes, 100%, 127.034 kBytes/s, ETA 0s
Errors: 2 (retrying may help)
Checks: 868 / 868, 100%
Elapsed time: 5m23.5s
The command I run is:
rclone -v copy --include PREFIX*/log_*.zip ftp-name:dir/in-ftp minio:vilmos-test-rinfra-637/some-random-dir
Is there a way to set the rate-limit on the minio component? A lot of these files (100+) gets copied after all, but it's a bit hard to debug my rclone flow, if the minio throws these errors all the time. I don't mind if the copy will happen slower.