How to settings to choose the one with the least free space when copy to the union

Help and Support
1

What is the problem you are having with rclone?

I set up two SharePoint using union mode, named uu:
backend
①f3651-98: 100G remaining
②f3651-98m: 24T remaining

Policy is
union root'': actionPolicy = *policy.EpAll, createPolicy = *policy.EpMfs, searchPolicy = *policy.FF

epmfs (existing path, most free space) Of all the upstreams on which the relative path exists choose the one with the most free space.

But when I run Copy new file to uu:, the new file will be copied to f3651-98:
I hope the file will be copied to the most free space f3651-98m:

How should my policy be set

What is your rclone version (output from rclone version)

rclone v1.54.1

  • os/arch: linux/amd64
  • go version: go1.15.8

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Debian GNU/Linux 9 \n \l

Which cloud storage system are you using? (eg Google Drive)

Union

The command you were trying to run (eg rclone copy /tmp remote:tmp)

root@cnhb55369ds:~/.config/rclone# rclone copy a:zh uu: -vvP
2021/03/19 13:52:08 DEBUG : rclone: Version "v1.54.1" starting with parameters ["rclone" "copy" "a:zh" "uu:" "-vvP"]
2021/03/19 13:52:08 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/03/19 13:52:08 DEBUG : Creating backend with remote "a:zh"
2021/03/19 13:52:08 DEBUG : Creating backend with remote "uu:"
2021/03/19 13:52:08 DEBUG : Creating backend with remote "f3651-98:zh"
2021/03/19 13:52:08 DEBUG : Creating backend with remote "f3651-98m:zh"
2021/03/19 13:52:08 DEBUG : f3651-98: Loaded invalid token from config file - ignoring
2021/03/19 13:52:10 DEBUG : Keeping previous permissions for config file: -rwxrwxrwx
2021/03/19 13:52:10 DEBUG : f3651-98: Saved new token in config file
2021/03/19 13:52:12 DEBUG : union root '': actionPolicy = *policy.EpAll, createPolicy = *policy.EpMfs, searchPolicy = *policy.FF
2021-03-19 13:52:14 DEBUG : .password: Size and modification time the same (differ by -215ms, within tolerance 1s)
2021-03-19 13:52:14 DEBUG : .password: Unchanged skipping

The rclone config contents with secrets removed.

[uu]
type = union
upstreams = f3651-98m:zh f3651-98:zh
create_policy = epmfs
search_policy = ff

A log from the command with the -vv flag

Paste  log here
2

anybody help???

3

is that config file correct, not missing anything?

4

You only posted part of your rclone.conf so we can't answer. If you want to post the full one, that would be superb.

5

What information do you need？

rclone.conf:

[f3651-98]
type = onedrive
token = {"access_token":"xxxxxxxxxxxxxxx
drive_id = b!EbUxxxo0VUqxxxxxxxxxx0
drive_type = documentLibrary
 
[f3651-98m]
type = onedrive
token = {"access_token":"eyJxxxxxx
drive_id = b!0CYexxxxL0-xxxxxxxxxx
drive_type = documentLibrary
chunk_size = 80M

[uu]
type = union
upstreams = f3651-98:test f3651-98m:test
create_policy = epmfs
search_policy = ff

rclone about

root@cnhb55369ds:~# rclone about f3651-98:
Total:   25T
Used:    24.983T
Free:    11.664G
Trashed: 0
root@cnhb55369ds:~# rclone about f3651-98m:
Total:   25T
Used:    162.257G
Free:    24.839T
Trashed: 97.308k

rclone copy

root@cnhb55369ds:~/.config/rclone# rclone copy a:zh uu: -vP
2021-03-22 06:01:03 INFO  : #Num/300mium-679-C/300mium-679-C.mp4: Copied (new)
2021-03-22 06:04:34 INFO  : #Num/300mium-680-C/300mium-680-C.mp4: Copied (new)
2021-03-22 06:29:09 ERROR : J/juy-898-C/juy-898-C.mp4: Failed to copy: quotaLimitReached: Quota limit reached
2021-03-22 06:29:09 ERROR : Cancelling sync due to fatal error: quotaLimitReached: Quota limit reached
2021-03-22 06:29:09 NOTICE: M/miae-027-C/miae-027-C.mp4: Failed to cancel multipart upload: Delete 
2021-03-22 06:29:09 NOTICE: M/miae-051-C/miae-051-C.mp4: Failed to cancel multipart upload: Delete "https://xxx.sharepoint.com/sites/xx/_api/v2.0/drives/bxxx62GSDlM5LrSAa3sIriuFLDN_61gfFQ73LG4Dr-MK0/items/01PAOxxxxx?zJJZVBpVT0": context canceled
2021-03-22 06:29:09 ERROR : M/miae-051-C/miae-051-C.mp4: Failed to copy: Put "https://x?????.sharepoint.com/sites/x?????/_api/v2.0/drives/b!x?????-MK0/items/01PAOx?????pVT0": context canceled
2021-03-22 06:29:09 ERROR : Fatal error received - not attempting retries
Transferred:       17.279G / 5.425 TBytes, 0%, 2.434 MBytes/s, ETA 3w5d23h5m23s
Errors:                 4 (fatal error encountered)
Checks:              5646 / 5646, 100%
Transferred:            2 / 1171, 0%
Elapsed time:    2h1m42.5s
2021/03/22 06:29:09 INFO  : 
Transferred:       17.279G / 5.425 TBytes, 0%, 2.434 MBytes/s, ETA 3w5d23h5m23s
Errors:                 4 (fatal error encountered)
Checks:              5646 / 5646, 100%
Transferred:            2 / 1171, 0%
Elapsed time:    2h1m42.5s

2021/03/22 06:29:09 Failed to copy with 4 errors: last error was: quotaLimitReached: Quota limit reached
root@cnhb55369ds:~/.config/rclone#

rclone copy -vvP

root@cnhb55369ds:~/.config/rclone# rclone copy a:zh uu: -vv
2021/03/22 15:38:28 DEBUG : rclone: Version "v1.54.1" starting with parameters ["rclone" "copy" "a:zh" "uu:" "-vv"]
2021/03/22 15:38:28 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/03/22 15:38:28 DEBUG : Creating backend with remote "a:zh"
2021/03/22 15:38:28 DEBUG : a: Loaded invalid token from config file - ignoring
2021/03/22 15:38:28 DEBUG : Keeping previous permissions for config file: -rwxrwxrwx
2021/03/22 15:38:28 DEBUG : a: Saved new token in config file
2021/03/22 15:38:30 DEBUG : Creating backend with remote "uu:"
2021/03/22 15:38:30 DEBUG : Creating backend with remote "f3651-98m:zh"
2021/03/22 15:38:30 DEBUG : Creating backend with remote "f3651-98:zh"
2021/03/22 15:38:30 DEBUG : f3651-98: Loaded invalid token from config file - ignoring
2021/03/22 15:38:30 DEBUG : One drive root 'zh': Token expired but no uploads in progress - doing nothing
2021/03/22 15:38:30 DEBUG : f3651-98m: Loaded invalid token from config file - ignoring
2021/03/22 15:38:30 DEBUG : Keeping previous permissions for config file: -rwxrwxrwx
2021/03/22 15:38:30 DEBUG : f3651-98m: Saved new token in config file
2021/03/22 15:38:31 DEBUG : Keeping previous permissions for config file: -rwxrwxrwx
2021/03/22 15:38:31 DEBUG : f3651-98: Saved new token in config file
2021/03/22 15:38:33 DEBUG : union root '': actionPolicy = *policy.EpAll, createPolicy = *policy.EpMfs, searchPolicy = *policy.FF

after rclone copy

root@cnhb55369ds:~/.config/rclone# rclone about f3651-98:
Total:   25T
Used:    24.983T
Free:    1.316G
Trashed: 0
root@cnhb55369ds:~/.config/rclone# rclone about f3651-98m:
Total:   25T
Used:    169.760G
Free:    24.832T
Trashed: 97.308k

The new file is not stored in the most free space (f3651-98m: 25T free space)

f3651-98: Only 500G is free, but almost all files are sent here, until the storage usage exceeds, only a small part is stored in f3651-98m:

6

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.