Error when uploading larger files to OneDrive

[linuxrecon]

Hi,

it seems like that uploading files that are lager than round about 210MB to OneDrive is resulting in an error like this:

2019/05/07 14:36:19 DEBUG : vojr7qh51lcc4sed43pnf1jop0/6e2js5jslmk60fvh0u2ad36sj8/0a2u13l1vvgvorj0egi0chalq3s8u8rmh9gbekvl9ekqnuetpohg: Uploading segment 199229440/5109523566 size 10485760
2019/05/07 14:36:22 DEBUG : vojr7qh51lcc4sed43pnf1jop0/6e2js5jslmk60fvh0u2ad36sj8/0a2u13l1vvgvorj0egi0chalq3s8u8rmh9gbekvl9ekqnuetpohg: Uploading segment 209715200/5109523566 size 10485760
2019/05/07 14:36:22 DEBUG : Couldn't decode error response: invalid character '<' looking for beginning of value
2019/05/07 14:36:22 DEBUG : vojr7qh51lcc4sed43pnf1jop0/6e2js5jslmk60fvh0u2ad36sj8/0a2u13l1vvgvorj0egi0chalq3s8u8rmh9gbekvl9ekqnuetpohg: Error encountered during upload: 400 Bad Request: 
2019/05/07 14:36:22 DEBUG : vojr7qh51lcc4sed43pnf1jop0/6e2js5jslmk60fvh0u2ad36sj8/0a2u13l1vvgvorj0egi0chalq3s8u8rmh9gbekvl9ekqnuetpohg: Cancelling multipart upload
2019/05/07 14:36:22 ERROR : file1.bin: Failed to copy: 400 Bad Request: 
2019/05/07 14:36:22 ERROR : file1.bin(0xc001482f60): failed to transfer file from cache to remote: 400 Bad Request: 
2019/05/07 14:36:22 DEBUG : file1.bin(0xc001482f60): >close: err=failed to transfer file from cache to remote: 400 Bad Request: 
2019/05/07 14:36:22 ERROR : file1.bin(0xc001482f60): RWFileHandle.Flush error: failed to transfer file from cache to remote: 400 Bad Request: 
2019/05/07 14:36:22 DEBUG : &{file1.bin (rw)}: >Flush: err=failed to transfer file from cache to remote: 400 Bad Request: 
2019/05/07 14:36:22 DEBUG : &{file1.bin (rw)}: Release: 
2019/05/07 14:36:22 DEBUG : file1.bin(0xc001482f60): RWFileHandle.Release nothing to do
2019/05/07 14:36:22 DEBUG : &{file1.bin (rw)}: >Release: err=<nil>
2019/05/07 14:36:52 INFO  : Cleaned the cache: objects 4 (was 4), total size 4.757G (was 4.757G)
2019/05/07 14:37:52 INFO  : Cleaned the cache: objects 4 (was 4), total size 4.757G (was 4.757G)
2019/05/07 14:38:52 INFO  : Cleaned the cache: objects 4 (was 4), total size 4.757G (was 4.757G)
2019/05/07 14:39:52 INFO  : Cleaned the cache: objects 4 (was 4), total size 4.757G (was 4.757G)
2019/05/07 14:40:52 INFO  : Cleaned the cache: objects 4 (was 4), total size 4.757G (was 4.757G)
2019/05/07 14:41:52 INFO  : Cleaned the cache: objects 4 (was 4), total size 4.757G (was 4.757G)
2019/05/07 14:42:52 INFO  : Cleaned the cache: objects 4 (was 4), total size 4.757G (was 4.757G)

I used the “mount” option which comes with rclone. The complete command:
rclone -vv mount --vfs-cache-mode full --vfs-read-chunk-size 100M --vfs-read-chunk-size-limit 0 --buffer-size 128M OneDrive_crypt: onedrive/

I’ve also tested unsetting vfs-read-chunk-size as well as buffer-size 0M and buffer-size 256M. However, the result always is the same.

Thanks in advance.

PS: I’m using the built-in encryption which comes with rclone as you can see in the command above (remote name: OneDrive_crypt).

[Animosity022]

rclone version shows what?

What does your rclone.conf look like without keys / passcodes.

[linuxrecon]

Thanks for your fast respone.

rclone version shows what?

rclone --version
rclone v1.47.0
os/arch: linux/amd64
go version: go1.12.4

What does your rclone.conf look like without keys / passcodes.

[OneDrive]
type = onedrive
token = {"access_token":"..."}
drive_id = MyID
drive_type = personal

[OneDrive_crypt]
type = crypt
remote = OneDrive:vid
filename_encryption = standard
directory_name_encryption = true
password = ...

[Animosity022]

What happens if you just copy the same file up with a rclone copy command, does that work as well?

[linuxrecon]

No, sadly it doesn’t:

rclone copy file1.bin OneDrive_crypt:
2019/05/08 07:59:24 ERROR : file1.bin: Failed to copy: 400 Bad Request:
2019/05/08 07:59:24 ERROR : Attempt 1/3 failed with 2 errors and: 400 Bad Request:
2019/05/08 08:00:48 ERROR : file1.bin: Failed to copy: 400 Bad Request:
2019/05/08 08:00:48 ERROR : Attempt 2/3 failed with 2 errors and: 400 Bad Request:

The parameter -vv show at the end that the copy process stopped at ~366MB:

Transferred: 366.660M / 4.757 GBytes, 8%, 5.917 MBytes/s, ETA 12m41s
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 1, 0%
Elapsed time: 1m1.9s

[ncw (Nick Craig-Wood)]

2019/05/07 14:36:22 ERROR : file1.bin: Failed to copy: 400 Bad Request:

I’m pretty sure this is some sort of limiting at onedrive. Can you ask your administrator if there is a maximum file size set? Or are you using the public onedrive?

[linuxrecon]

I’m using the public OneDrive but paid for it (family account with 1TB of storage).

I’ve searched a lot to find out if there are any limitation. I’ve found out that there is a 15GB limit when uploading files to OneDrive public business instance. However, they never really talked public about the limit that comes with the API nor with the public instance.

I’ve did another test with a 1.9 GB, 4.8 GB and 5 GB file. The 1.9 went through while the 4.8 and 5 GB ran into the same error. So it indeed is a limitation. Thanks for mentioning it. I guess the limit is between 2 and 4 GB then

[ncw (Nick Craig-Wood)]

Good testing! I can’t find any online docs about the file size limits for the public OneDrive… The limit used to be 20GB when I first did the onedrive backend, but it wouldn’t suprise me if they had shrunk it.