Backend doesnt support used space or free space

Help and Support
1

What is the problem you are having with rclone?

I'm trying to use a backend which doesnt support used space, free space, or even total size. It's the onedrive backend (sharepoint), of the onedrive for education. In using this backend in a union it fails to use any policy and also just keeps trying to upload after ERROR: 507 INSUFFICIENT STORAGE.

But rclone size backend: works normally for the original backend and its union. It would work ok i guess with an option to manually set the total size of a backend and then keeping track of the quota used either by rclone size or keeping tab of the uploads.

2024/09/27 14:30:32 NOTICE: Free Space is not supported for upstream One1, treating as infinite
2024/09/27 14:21:07 NOTICE: Used Space is not supported for upstream One1, treating as 0

Run the command 'rclone version' and share the full output of the command.

rclone v1.68.1

  • os/version: debian 12.5 (64 bit)
  • os/kernel: 6.8.4-2-pve (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.23.1
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Microsoft Sharepoint/Onedrive for education

The command you were trying to run (eg rclone copy /tmp remote:tmp)

./rclone -vvv -P copy ../we oobo3:

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[One1]
type = webdav
url = https://XXX.sharepoint.com/personal/XXX/Documents
vendor = sharepoint
user = XXX                                                                                                                                     
pass = XXX 

[One2]
type = webdav
url = https://YYY.sharepoint.com/personal/YYY/Documents
vendor = sharepoint
user = XXX                                                                                                                                     
pass = XXX 

[oobo]
type = union
upstreams = One1: One2:

[oobo1]
type = crypt
remote = oobo:
password = XXX                                                                                                                                 password2 = XXX                                                                                                                                

[oobo2]
type = chunker
remote = oobo1:
chunk_size = 240Mi

[oobo3]
type = hasher
remote = oobo2:

A log from the command that you were trying to run with the -vv flag

2024/09/27 14:30:12 DEBUG : FILEFILE: Update will use the normal upload strategy (no chunks)                                                                                                                                         2024/09/27 14:30:13 ERROR : FILEFILE: Failed to copy: 507 INSUFFICIENT STORAGE                          2024/09/27 14:30:13 DEBUG : FILEFILE: Rehash in-fly due to incomplete or slow source set [md5, sha1] (err: <nil>)
2024/09/27 14:30:13 DEBUG : oobo2~hasher.bolt: Opened for writing in 70.474µs
2024/09/27 14:30:13 DEBUG : oobo2~hasher.bolt: released
2024/09/27 14:30:13 ERROR : FILEFILE: Failed to copy: 507 INSUFFICIENT STORAGE
2024/09/27 14:30:13 DEBUG : FILEFILE: Rehash in-fly due to incomplete or slow source set [md5, sha1
] (err: <nil>)
2024/09/27 14:30:13 DEBUG : oobo2~hasher.bolt: Opened for writing in 101.905µs
2024/09/27 14:30:14 DEBUG : oobo2~hasher.bolt: released
2024/09/27 14:30:14 NOTICE: Free Space is not supported for upstream One1, treating as infinite
2

As per rclone docs:

Some policies rely on quota information. These policies should be used only if your upstreams support the respective quota fields.

Policy Required Field
lfs, eplfs Free
mfs, epmfs Free
lus, eplus Used
lno, eplno Objects

So in your situation this is not going to work:

As you try to use default epmfs policy.

What you can do is to use rand:

rand (random) Calls all and then randomizes. Returns only one upstream.

So it will be (other policies also explicitly defined for clarity):

[oobo]
type = union
upstreams = One1: One2:
action_policy = epall
create_policy = rand
search_policy = ff

Make sure that your two union members are less or more equally used before starting using it. You use chunker which is great for keeping all nicely balanced.

As for:

You have to make sure you have enough free space... there is no workaround for that. In case if your one upstream is full you can always mark it as not used for creation of any new items:

[oobo]
type = union
upstreams = One1: One2::nc

but much better IMO is to keep them both balanced. Move some stuff from the full one to one with free space. Use rclone size to check how much data is stored and then move required amount between remotes, for example:

rclone move One1: One2: --max-transfer 250G --cutoff-mode soft
3

Thanks for the reply. Yes I guess if I manage them individually I could upload the quota and use the union for reading. But I think if there could be a flag on each backend of max upload limit, it'd be neat. This Onedrive/Sharepoint used to report in the about command, free space, used space (or so I think) but ms changed something after they lowered the limit.

4

Sure it is possible. When somebody needs it so badly that will implement it and submit PR:)

I have noticed myself that Microsoft is changing thing recently without communicating anything.