OneDrive chunker/union issue: not spreading across multiple remotes

Help and Support
1

What is the problem you are having with rclone?

I have 1 encrypted file that is 1.6TB big (system backup) and want to back it up to OneDrive. Both of the remotes I added work fine seperatly when I upload files to them using rclone copy. But because of the total quota per remote of 1TB and the file limit of 250GB, I need to chunk the file manually or using rclone chunker, and then spread the files across two remotes of OneDrive.

I tried two scenario's:

  1. Use the CLI command split to generate files of 10GB, and then upload those files to a union remote which - as I believe - should spread the files already splitted.
  2. Use the rclone remote overlay chunker to split it into files of 10GB, which - as I believe - should split the huge 1.6TB file and spread the parts over the two remotes.

In both scenarios I get the issue 'HTTP error 507 (507 Insufficient Storage)'. I set up a union and a chunker remote. I tried create_policy lfs and epmfs (which is default). In both scenario's the files are uploaded upto 1TB to 1 remote, and then it stopped. I made sure both remotes had plenty of space (they were both empty before I started):

rclone about remote_a:
Total:   1.005 TiB
Used:    0 B
Free:    1.005 TiB
Trashed: 0 B

rclone about remote_b:
Total:   1.005 TiB
Used:    0 B
Free:    1.005 TiB
Trashed: 0 B

rclone about union_remote_a_remote_b:
Total:   2.010 TiB
Used:    0 B
Free:    2.010 TiB
Trashed: 0 B

rclone about union_chunked_remote_a_remote_b:
Total:   2.010 TiB
Used:    0 B
Free:    2.010 TiB
Trashed: 0 B

How can I upload the 1.6TB backup file to OneDrive? I am aware that the big file in once piece results in a slow upload (because there is 1 upload stream of 8-10MB/s). I can manually split it, and that's fine but still the 'spreading' across two remotes does not work.

Run the command 'rclone version' and share the full output of the command.

rclone v1.68.0
- os/version: darwin 12.7.6 (64 bit)
- os/kernel: 21.6.0 (x86_64)
- os/type: darwin
- os/arch: amd64
- go/version: go1.23.1
- go/linking: dynamic
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Microsoft OneDrive. I have a Family subscribtion and can use 6 seperate 1TB storage locations. For now I have configured two remotes which should be more than enough. They are called remote_a and remote_b.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Scenario 1 (1 big file of 1.6TB):

rclone copy "/Volumes/SSD/Backup" "union_chunked_remote_a_remote_b:" --transfers=4 --checkers=8 --drive-chunk-size=256M --buffer-size=128M -P --stats=1s --low-level-retries=10
Transferring:
Transferred:   	    1.006 TiB / 1.006 TiB, 100%, 1.406 MiB/s, ETA 0s
Errors:                 1 (fatal error encountered)
Elapsed time:  1d7h4m51.1s

Scenario 2 (already splitted the files into pieces of 10GB):

rclone copy "/Volumes/SSD/Backup_Parts/" "union_remote_a_remote_b:" --transfers=4 --checkers=8 --drive-chunk-size=256M --buffer-size=128M -P --stats=1s --low-level-retries=10

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[remote_a]
type = onedrive
client_id = XXX
client_secret = XXX
token = XXX
drive_id = XXX
drive_type = personal

[remote_b]
type = onedrive
client_id = XXX
client_secret = XXX
token = XXX
drive_id = XXX
drive_type = personal

[union_remote_a_remote_b]
type = union
upstreams = remote_a: remote_b:
action_policy = all
create_policy = lfs
search_policy = ff

[union_chunked_remote_a_remote_b]
type = chunker
remote = union_remote_a_remote_b:
chunk_size = 10G
name_format = *.part.###

A log from the command that you were trying to run with the -vv flag

2024/09/11 11:28:19 INFO  : Starting transaction limiter: max 10000 transactions/s with burst 1
2024/09/11 11:28:19 DEBUG : rclone: Version "v1.68.0" starting with parameters ["rclone" "copy" "/Volumes/SSD/Backup" "union_chunked_remote_a_remote_b:" "--transfers=4" "--checkers=8" "--drive-chunk-size=256M" "--buffer-size=128M" "-P" "--stats=1s" "--low-level-retries=10"]
2024/09/11 11:28:19 DEBUG : Creating backend with remote "/Volumes/SSD/Backup"
2024/09/11 11:28:19 DEBUG : Using config file from "/Users/user/.config/rclone/rclone.conf"
2024/09/11 11:28:19 DEBUG : fs cache: adding new entry for parent of "/Volumes/SSD/Backup", "/Volumes/SSD"
2024/09/11 11:28:19 DEBUG : Creating backend with remote "union_chunked_remote_a_remote_b:"
2024/09/11 11:28:19 DEBUG : Creating backend with remote "union_remote_a_remote_b:"
2024/09/11 11:28:19 DEBUG : Creating backend with remote "remote_b:"
2024/09/11 11:28:19 DEBUG : Creating backend with remote "remote_a:"
2024/09/11 11:28:20 DEBUG : union root '': actionPolicy = *policy.All, createPolicy = *policy.Lfs, searchPolicy = *policy.FF
2024/09/11 11:28:20 DEBUG : Reset feature "ListR"
2024/09/11 11:28:20 DEBUG : Backup: Need to transfer - File not found at Destination
2024/09/11 11:28:20 DEBUG : Backup: skip slow MD5 on source file, hashing in-transit
2024/09/11 11:28:21 DEBUG : Backup.part.001_nbuxd4: Starting multipart upload
2024/09/11 11:28:22 DEBUG : Backup.part.001_nbuxd4: Uploading segment 0/10737418240 size 10485760
2024/09/11 11:28:23 DEBUG : Backup.part.001_nbuxd4: Uploading segment 10485760/10737418240 size 10485760
2024/09/11 11:28:25 DEBUG : Backup.part.001_nbuxd4: Uploading segment 20971520/10737418240 size 10485760
...
2024/09/12 18:32:39 DEBUG : Backup.part.103_nbuxd4: Cancelling multipart upload: HTTP error 507 (507 Insufficient Storage) returned body: "{\"error\":{\"code\":\"quotaLimitReached\",\"message\":\"Insufficient Space Available\",\"localizedMessage\":\"The given action cannot be completed because the user has reached their quota limit.\"}}"
2024/09/12 18:32:39 DEBUG : info from Update error:
null
2024/09/12 18:33:10 ERROR : Backup: Failed to copy: HTTP error 507 (507 Insufficient Storage) returned body: "{\"error\":{\"code\":\"quotaLimitReached\",\"message\":\"Insufficient Space Available\",\"localizedMessage\":\"The given action cannot be completed because the user has reached their quota limit.\"}}"
2024/09/12 18:33:10 ERROR : Fatal error received - not attempting retries
2024/09/12 18:33:10 INFO  :
Transferred:   	    1.006 TiB / 1.006 TiB, 100%, 1.406 MiB/s, ETA 0s
Errors:                 1 (fatal error encountered)
Elapsed time:  1d7h4m51.1s

2024/09/12 18:33:10 DEBUG : 11 go routines active
2024/09/12 18:33:10 NOTICE: Failed to copy: HTTP error 507 (507 Insufficient Storage) returned body: "{\"error\":{\"code\":\"quotaLimitReached\",\"message\":\"Insufficient Space Available\",\"localizedMessage\":\"The given action cannot be completed because the user has reached their quota limit.\"}}"
2

welcome to the forum,

currently, you have create_policy = lfs
"Create category: Pick the upstream with the least available free space"

maybe, want create_policy = mfs
"Create category: Pick the upstream with the most available free space"

1 Like
3

Thanks for your quick reply.
If two remotes have 0 bytes used and both 1.0TB available, would this fix the issue that once remote A is full (or B), the data upstream continous on the other remote? Or is that something that Union cannot do? Because the file I try to upload exceeds the total available space on 1 remote, but the union space is enough

4

welcome

i have never used union remote, never used chunker remote.
maybe @jcastill can answer or have suggestions?

5

I have changed this in the config and the upload is running. Might take a while. I now do not use chunker, only union. So I manually splitted the 1.6TB file using split into 10GB files.
I’ll let you know once the upload is done.

6

mfs should be the right answer here but it only works when it returns real values. I have noticed recently that OneDrive used space value provided by API is delayed. I did not test it in details but it can be quite long.

In case of OneDrive I switched to create_policy = rand and it solves the problem. Only downside is that in case when you start with massively unbalanced upstreams it is advisable to balance them before transferring big files.

7

Is there an option to update the quota faster/more often?

8

Thanks. I updated the configuration and the upload is in process right now.

1 Like
9

Not really the delay is on Microsoft side, where their response is outdated.

I use chunk for my needs as I have some 300GB files, but do not use union alongside.

I agree with @kapitainsky approach of using policy rand, should not be a problem unless you get very close to filling your remotes. mfs should work as well, unless you are doing a lot of parallel uploads and movements in OneDrive which is what I noticed delays the quota update. If you are just uploading a couple big of files simultaneously it should be fine.

I also would confirm the chunk size is the best for your application. I try to keep chunks as big as the biggest file that would fit without chunking. This basically minimizes chunk use and simplifies everything.

1 Like
10

This fixed it, thanks! Remote A has 740GB and Remote B has 760GB. So it levelled quite good :slight_smile:

1 Like
11

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.