Problem with uploading to S3 Scaleway

Help and Support
1

Hi! I try to upload a file to S3 Scaleway

Everything works good with small files (e.g. 600 Mb in size). But I can't upload a file of 7 Gb.
After uploading 4.8 Gb I begin to see errors. Verbose output is below.

It starts working good:

2020/04/02 15:38:25 DEBUG : rclone: Version "v1.51.0" starting with parameters ["/usr/bin/rclone" "copy" "/var/backup/web1" "scaleway:backup.server/example.com" "--include" "manual-.7z" "--include" "web[0-9].7z" "--include" "db_*.7z" "--s3-storage-class" "GLACIER" "-vv"]
2020/04/02 15:38:25 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2020/04/02 15:38:25 INFO : S3 bucket backup.server path example.com: Waiting for checks to finish
2020/04/02 15:38:25 INFO : S3 bucket backup.server path example.com: Waiting for transfers to finish
2020/04/02 15:38:50 DEBUG : manual-web1_2020-04-02_09-41.tar.7z: multipart upload starting chunk 1 size 5M offset 0/7.004G
2020/04/02 15:38:50 DEBUG : manual-web1_2020-04-02_09-41.tar.7z: multipart upload starting chunk 2 size 5M offset 5M/7.004G
2020/04/02 15:38:50 DEBUG : manual-web1_2020-04-02_09-41.tar.7z: multipart upload starting chunk 3 size 5M offset 10M/7.004G
2020/04/02 15:38:50 DEBUG : manual-web1_2020-04-02_09-41.tar.7z: multipart upload starting chunk 4 size 5M offset 15M/7.004G
2020/04/02 15:38:50 DEBUG : manual-web1_2020-04-02_09-41.tar.7z: multipart upload starting chunk 5 size 5M offset 20M/7.004G
2020/04/02 15:38:50 DEBUG : manual-web1_2020-04-02_09-41.tar.7z: multipart upload starting chunk 6 size 5M offset 25M/7.004G
2020/04/02 15:38:50 DEBUG : manual-web1_2020-04-02_09-41.tar.7z: multipart upload starting chunk 7 size 5M offset 30M/7.004G
2020/04/02 15:38:50 DEBUG : manual-web1_2020-04-02_09-41.tar.7z: multipart upload starting chunk 8 size 5M offset 35M/7.004G
2020/04/02 15:38:50 DEBUG : manual-web1_2020-04-02_09-41.tar.7z: multipart upload starting chunk 9 size 5M offset 40M/7.004G

Then I see the problems after about 4.8 Gb:

2020/04/02 15:34:39 DEBUG : manual-web1_2020-04-02_09-41.tar.7z: multipart upload starting chunk 997 size 5M offset 4.863G/7.004G
2020/04/02 15:34:39 DEBUG : manual-web1_2020-04-02_09-41.tar.7z: multipart upload starting chunk 998 size 5M offset 4.868G/7.004G
2020/04/02 15:34:39 DEBUG : manual-web1_2020-04-02_09-41.tar.7z: multipart upload starting chunk 999 size 5M offset 4.873G/7.004G
2020/04/02 15:34:39 DEBUG : manual-web1_2020-04-02_09-41.tar.7z: multipart upload starting chunk 1000 size 5M offset 4.878G/7.004G
2020/04/02 15:34:39 DEBUG : manual-web1_2020-04-02_09-41.tar.7z: multipart upload starting chunk 1001 size 5M offset 4.883G/7.004G
2020/04/02 15:34:39 DEBUG : pacer: low level retry 1/10 (error InvalidArgument: Part number must be an integer between 1 and 1000, inclusive
status code: 400, request id: tx54fe9d28acaa4cff81982-005e86060f, host id: tx54fe9d28acaa4cff81982-005e86060f)
2020/04/02 15:34:39 DEBUG : pacer: Rate limited, increasing sleep to 10ms
2020/04/02 15:34:39 DEBUG : pacer: Reducing sleep to 0s
2020/04/02 15:34:39 DEBUG : manual-web1_2020-04-02_09-41.tar.7z: multipart upload starting chunk 1002 size 5M offset 4.888G/7.004G
2020/04/02 15:34:39 DEBUG : manual-web1_2020-04-02_09-41.tar.7z: multipart upload starting chunk 1003 size 5M offset 4.893G/7.004G
2020/04/02 15:34:39 DEBUG : manual-web1_2020-04-02_09-41.tar.7z: multipart upload starting chunk 1004 size 5M offset 4.897G/7.004G
2020/04/02 15:34:39 DEBUG : pacer: low level retry 2/10 (error InvalidArgument: Part number must be an integer between 1 and 1000, inclusive
status code: 400, request id: txea19acacbb404cec86fa1-005e86060f, host id: txea19acacbb404cec86fa1-005e86060f)
2020/04/02 15:34:39 DEBUG : pacer: Rate limited, increasing sleep to 10ms
2020/04/02 15:34:39 DEBUG : pacer: low level retry 1/10 (error InvalidArgument: Part number must be an integer between 1 and 1000, inclusive
status code: 400, request id: txe13b663cb0784dc3aa0e7-005e86060f, host id: txe13b663cb0784dc3aa0e7-005e86060f)
2020/04/02 15:34:39 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/04/02 15:34:39 DEBUG : pacer: low level retry 1/10 (error InvalidArgument: Part number must be an integer between 1 and 1000, inclusive
status code: 400, request id: tx00f4ab1743ed4331b0891-005e86060f, host id: tx00f4ab1743ed4331b0891-005e86060f)
2020/04/02 15:34:39 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2020/04/02 15:34:40 DEBUG : pacer: low level retry 1/10 (error InvalidArgument: Part number must be an integer between 1 and 1000, inclusive
status code: 400, request id: tx109418ec2d844f49ac6bc-005e86060f, host id: tx109418ec2d844f49ac6bc-005e86060f)
2020/04/02 15:34:40 DEBUG : pacer: Rate limited, increasing sleep to 80ms
2020/04/02 15:34:40 DEBUG : pacer: low level retry 3/10 (error InvalidArgument: Part number must be an integer between 1 and 1000, inclusive
status code: 400, request id: tx98cb42029cf548eb81287-005e86060f, host id: tx98cb42029cf548eb81287-005e86060f)
2020/04/02 15:34:40 DEBUG : pacer: Rate limited, increasing sleep to 160ms
2020/04/02 15:34:40 DEBUG : pacer: low level retry 2/10 (error InvalidArgument: Part number must be an integer between 1 and 1000, inclusive
status code: 400, request id: tx6610a3bb78ab4ad7b7ac3-005e860610, host id: tx6610a3bb78ab4ad7b7ac3-005e860610)
2020/04/02 15:34:40 DEBUG : pacer: Rate limited, increasing sleep to 320ms
2020/04/02 15:34:40 DEBUG : pacer: low level retry 2/10 (error InvalidArgument: Part number must be an integer between 1 and 1000, inclusive
status code: 400, request id: tx26e0cf0888504a30bc99d-005e860610, host id: tx26e0cf0888504a30bc99d-005e860610)
2020/04/02 15:34:40 DEBUG : pacer: Rate limited, increasing sleep to 640ms
2020/04/02 15:34:40 DEBUG : pacer: low level retry 2/10 (error InvalidArgument: Part number must be an integer between 1 and 1000, inclusive
status code: 400, request id: tx0146bf4eba434652bfc82-005e860610, host id: tx0146bf4eba434652bfc82-005e860610)
2020/04/02 15:34:40 DEBUG : pacer: Rate limited, increasing sleep to 1.28s
2020/04/02 15:34:41 DEBUG : pacer: low level retry 3/10 (error InvalidArgument: Part number must be an integer between 1 and 1000, inclusive
status code: 400, request id: tx04196ed147aa4a5c8e622-005e860611, host id: tx04196ed147aa4a5c8e622-005e860611)
2020/04/02 15:34:41 DEBUG : pacer: Rate limited, increasing sleep to 2s
2020/04/02 15:34:41 DEBUG : pacer: low level retry 4/10 (error InvalidArgument: Part number must be an integer between 1 and 1000, inclusive
status code: 400, request id: tx06936e1b50464587b1d6a-005e860611, host id: tx06936e1b50464587b1d6a-005e860611)
2020/04/02 15:34:43 DEBUG : pacer: low level retry 3/10 (error InvalidArgument: Part number must be an integer between 1 and 1000, inclusive
status code: 400, request id: tx178f317d5d5a4463b468c-005e860612, host id: tx178f317d5d5a4463b468c-005e860612)
2020/04/02 15:34:44 DEBUG : pacer: low level retry 3/10 (error InvalidArgument: Part number must be an integer between 1 and 1000, inclusive
status code: 400, request id: tx6dd8041aeedf484b90442-005e860614, host id: tx6dd8041aeedf484b90442-005e860614)
2020/04/02 15:34:47 DEBUG : pacer: low level retry 4/10 (error InvalidArgument: Part number must be an integer between 1 and 1000, inclusive
status code: 400, request id: tx5895e06339a548a5b3494-005e860617, host id: tx5895e06339a548a5b3494-005e860617)

After this error, I see no-stopping error messages, and the upload speed becomes very slow, about 1Mb per minute or two!

Can you help me please?

2

I could solve this problem by changing chunk size:
--s3-chunk-size 100M

In this case the transfer needs only 72 chunks to complete. This is the problem of Scaleway: they have a limit of 1000 chunks and max size 5Tb per file https://www.scaleway.com/en/faq/how-can-i-upload-large-objects/

1 Like
3

That looks like a bug in Scaleway's implementation of the S3 protocol... AWS S3 allows 1 to 10,000 here.

Eg from https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html

Part numbers can be any number from 1 to 10,000, inclusive.

Maybe you can ask them about that.

You can work around it by setting --s3-chunk-size large enough so that you get less than 1000 parts for your 7 GB file --s3-chunk-size 16M should be enough.

Note that the parts are buffered in memory and there can potentially be --transfers * --s3-chunk-size * --s3-concurrency of them so you don't want to set it too big!

4

We could make this a parameter, eg --s3-multipart-max-chunks which would default to 10000 but you could set it to 1000 for Scaleway. Rclone auto adapts the chunk size so the transfer fits withing the max chunks number already.

It would be quite simple if you wanted to send a pull request!

5

Thank you very much for a prompt response and comments! For me increasing the --s3-chunk-size is a comfortable solution.

1 Like
6

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.