413 Request Entity Too Large - Chunker, Crypt, OneDrive

Help and Support
1

What is the problem you are having with rclone?

Hi, I'm trying to upload many files to OneDrive using Crypt and Chunker.
My config:
Chunker -> Crypt -> OneDrive
I'm getting an error:

Failed to copy: 413 Request Entity Too Large:

which is weird, I'm using chunker to avoid such problems and it seems to appear only for files >= 2G. OneDrive support files up to 250G.

Can you please look into my setup?

Run the command 'rclone version' and share the full output of the command.

rclone v1.57.0
- os/version: qts 5.0.0 (20220324) (64 bit)
- os/kernel: 5.10.60-qnap (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.17.2
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

OneDrive for Business

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync /share/homes/ OneDrive-Crypt-Chunker:qnap/backups/homes/ --transfers=15 --checkers=15 --bwlimit 80M --progress --stats=1s --config=/share/homes...  --min-size 2G

The rclone config contents with secrets removed.

[OneDrive]
type = onedrive
client_id = <secret>
client_secret = <secret>
token = {"access_token":"<secret>"}
drive_id = <secret>
drive_type = business
chunk_size = 144834560

[OneDrive-Crypt]
type = crypt
remote = OneDrive:ENC-C/
password = <secret>
password2 = <secret>
filename_encryption = obfuscate

[OneDrive-Crypt-Chunker]
type = chunker
remote = OneDrive-Crypt:
#chunk_size = 96Mi
chunk_size = 64Gi
hash_type = md5quick

A log from the command with the -vv flag

2022-05-14 11:25:21 ERROR : admin: failed to open directory "admin": open /share/homes/admin: permission denied
2022-05-14 11:25:21 NOTICE: <secret>/@Recycle: Can't follow symlink without -L/--copy-links
2022-05-14 11:25:21 NOTICE: kamil/@Recycle: Can't follow symlink without -L/--copy-links
2022-05-14 11:25:28 ERROR : kamil/<secret>: Failed to copy: 413 Request Entity Too Large:
2022-05-14 11:25:28 ERROR : kamil/<secret>: Failed to copy: 413 Request Entity Too Large:
2022-05-14 11:25:28 ERROR : kamil/<secret>: Failed to copy: 413 Request Entity Too Large:
2022-05-14 11:25:29 ERROR : kamil/<secret>: Failed to copy: 413 Request Entity Too Large:
2022-05-14 11:25:29 ERROR : kamil/<secret>: Failed to copy: 413 Request Entity Too Large:
2022-05-14 11:25:30 ERROR : kamil/<secret>: Failed to copy: 413 Request Entity Too Large:
2022-05-14 11:25:30 ERROR : kamil/<secret>: Failed to copy: 413 Request Entity Too Large:
2022-05-14 11:25:30 ERROR : kamil/<secret>: Failed to copy: 413 Request Entity Too Large:
2

You have your chunk size at 64G which. I'd assume you want your chunk size less than 2G.

3

Not really, I want to use 64G or even 128G because my Internet Connection is 1Gpbs. I just want to get over OneDrive 250G file limit.
What's interesting is that, even if I have default 2G chunk_size I still get that error.

4

2nd bit would be to update to the latest and validate it still persists as you are a few revs behind.

5

And any info here might be applicable as well:

OneDrive + WebDAV: 413 Request Entity Too Large - Help and Support - rclone forum

6

I try latest stable version and latest beta release. Problem still persist.

rclone --version
rclone v1.58.1
- os/version: qts 5.0.0 (20220324) (64 bit)
- os/kernel: 5.10.60-qnap (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.17.9
- go/linking: static
- go/tags: none

rclone --version
rclone v1.59.0-beta.6142.542c1616b
- os/version: qts 5.0.0 (20220324) (64 bit)
- os/kernel: 5.10.60-qnap (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.18.2
- go/linking: static
- go/tags: none
7

OneDrive limits:

8

Ok, I found the problem in OneDrive config.

chunk_size = 144834560

This option causes problems with any file bigger than 2G.
Do you have any recommendations on this option for best performance? Or just leave it by default?

9

Sorry I personally do not use OneDrive so I would not want to comment.

Perhaps @Ole might have some recommendations.

10

I use the default chunk_size and doubt an increase would improve performance (until I have seen a test case showing so).

I generally use defaults for OneDrive except --checkers, which I increase to 16 in some of my jobs. This gives optimum performance in most situations.

I therefore recommend you remove the settings of chunk_size, --transfers and --bwlimit, unless you have tested and verified that each of them significantly improves performance in your situation.

11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.