Configure the 10,000 chunks limit?

psycho · June 10, 2020, 3:04pm

What is the problem you are having with rclone?

The 10,000 chunks limit mentioned in --s3-chunk-size documentation is not correct when using Scaleway object storage because the maximum number of parts is then 1000.
Consequently, trying to upload a file of 45G with --s3-chunk-size 32M logically fails because it would require to upload 1400+ chunks.
I am looking for a way to change this 10,000 limit to 1000 so rclone will actually

automatically increase the chunk size when uploading a large file of known size to stay below the <configured> chunks limit

but I can't find an option to do this. Did I miss something or is it not possible (yet ?) ?

What is your rclone version (output from `rclone version`)

$ rclone version
rclone v1.51.0
- os/arch: linux/386
- go version: go1.14

Which OS you are using and how many bits (eg Windows 7, 64 bit)

$ uname -a
Linux Syno 3.2.40 #25426 SMP PREEMPT Tue May 12 04:38:00 CST 2020 i686 GNU/Linux synology_evansport_214play

Which cloud storage system are you using? (eg Google Drive)

Scaleway

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone sync /volume2/backups encrypted_scaleway:backups

The rclone config contents with secrets removed.

[scaleway]
type = s3
provider = Other
env_auth = false
access_key_id = XXXXX
secret_access_key = XXXXX
region = fr-par
endpoint = s3.fr-par.scw.cloud
acl = private
force_path_style = false
storage_class = GLACIER
chunk_size = 32M

[encrypted_scaleway]
type = crypt
remote = scaleway:XXXXX
password = XXXXX
directory_name_encryption = false

A log from the command with the `-vv` flag

2020/06/10 13:25:36 DEBUG : pacer: low level retry 1/10 (error InvalidArgument: Part number must be an integer between 1 and 1000, inclusive
        status code: 400, request id: tx86ca18460d1249898e660-005ee0c32f, host id: tx86ca18460d1249898e660-005ee0c32f)

Animosity022 · June 10, 2020, 3:09pm

I think that doesn't exist. The 10,000 is a hard coded number so you need something to override it unless I'm mistaken:

github.com

rclone/rclone/blob/2ea15a72bc76f07eb3ccc0d62f0f19a3e4152a18/backend/s3/s3.go#L920


			Help:     `Whether to use mmap buffers in internal memory pool.`,
		},
		}})
}

// Constants
const (
	metaMtime           = "Mtime"                // the meta key to store mtime in - eg X-Amz-Meta-Mtime
	metaMD5Hash         = "Md5chksum"            // the meta key to store md5hash in
	maxSizeForCopy      = 5 * 1024 * 1024 * 1024 // The maximum size of object we can COPY
	maxUploadParts      = 10000                  // maximum allowed number of parts in a multi-part upload
	minChunkSize        = fs.SizeSuffix(1024 * 1024 * 5)
	defaultUploadCutoff = fs.SizeSuffix(200 * 1024 * 1024)
	maxUploadCutoff     = fs.SizeSuffix(5 * 1024 * 1024 * 1024)
	minSleep            = 10 * time.Millisecond // In case of error, start at 10ms sleep.

	memoryPoolFlushTime = fs.Duration(time.Minute) // flush the cached buffers after this long
	memoryPoolUseMmap   = false
)

// Options defines the configuration for this backend

psycho · June 10, 2020, 3:16pm

Erf, that's what I was affraid of. Thanks for your response !

Should I create an issue to request this feature or is it too specific ?

Animosity022 · June 10, 2020, 3:19pm

No reason not to request, it "feels" like an easy one to me but then again, I don't have to code it!

psycho · June 10, 2020, 3:24pm

Actually it won't be necessary ! I was browsing the code you quoted to see if I would be able to code it myself in spite of not knowing Go and I stumbled upon maxSizeForCopy setting :

github.com

rclone/rclone/blob/2ea15a72bc76f07eb3ccc0d62f0f19a3e4152a18/backend/s3/s3.go#L778


large file of known size to stay below the 10,000 chunks limit.

Files of unknown size are uploaded with the configured
chunk_size. Since the default chunk size is 5MB and there can be at
most 10,000 chunks, this means that by default the maximum size of
file you can stream upload is 48GB.  If you wish to stream upload
larger files then you will need to increase chunk_size.`,
			Default:  minChunkSize,
			Advanced: true,
		}, {
			Name: "max_upload_parts",
			Help: `Maximum number of parts in a multipart upload.

This option defines the maximum number of multipart chunks to use
when doing a multipart upload.

This can be useful if a service does not support the AWS S3
specification of 10,000 chunks.

Rclone will automatically increase the chunk size when uploading a
large file of a known size to stay below this number of chunks limit.

... which seems to be exactly what I need... because it have been added to solve this exact problem, raised in issue #4159. So I'll just wait v1.53 !

Animosity022 · June 10, 2020, 3:36pm

So you are right and wrong

It's not in the docs as it's not pushed yet so that's a good thing.

Was the pull for that so you should be good using a beta or wait for the next release.

psycho · June 10, 2020, 3:41pm

Ha, seems we cross-posted/edited.
Yes, I thought it was strange to see it in the code but not elsewhere so I looked the history, found the issue and the beta status. Again, thanks for the quick response !

Animosity022 · June 10, 2020, 3:45pm

I think you are good timing too as a point release just dropped so look now

ncw · June 10, 2020, 4:46pm

You should find the max_upload_parts support in the latest beta.

psycho · June 10, 2020, 6:59pm

I just tested it and that works perfectly, it automatically went from 32M to 186M to upload a ~180G file, as expected. I seem to have another problem though, but that will be another post if I can't solve it by myself.

Thanks again !

system · June 13, 2020, 6:59pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.