Union remote doesn't switch when first remote is full

What is the problem you are having with rclone?

I set up a union remote to do backups, between 1 gdrive and 1 scaleway S3, my idea while setting up was to fill up gdrive up to 15Gb then use the scaleway S3 for the rest of the data.
So I used the eplfs policy for create and action.

Run the command 'rclone version' and share the full output of the command.

rclone v1.53.3-DEV
- os/arch: linux/amd64
- go version: go1.17.5

Which cloud storage system are you using? (eg Google Drive)

gdrive & scaleway s3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync --transfers 8 --copy-links -P --exclude-from=./exclude /home/darato/ union_backup_1:thinkpad"

The rclone config contents with secrets removed.

[scway_standard_1]
type = s3
provider = Scaleway
env_auth = false
access_key_id =
secret_access_key =
region = fr-par
endpoint = s3.fr-par.scw.cloud
acl = private
storage_class = STANDARD

[scway_standard_1_crypt]
type = crypt
remote = scway_standard_1:rclone-backup/rclone
filename_encryption = standard
directory_name_encryption = true
password = 
password2 = 

[gdrive]
type = drive
client_id = ****.apps.googleusercontent.com
client_secret =
scope = drive
token = {"access_token":"*","token_type":"Bearer","refresh_token":"*","expiry":"2022-01-24T15:58:36.550014595+01:00"}

[gcrypt_gd]
type = crypt
remote = gdrive:rc
filename_encryption = standard
directory_name_encryption = true
password =
password2 =

[union_backup_1]
type = union
upstreams = scway_standard_1_crypt: gcrypt_gd:
action_policy = eplfs
create_policy = eplfs
search_policy = ff

A log from the command with the -vv flag

2022-01-24 15:14:34 ERROR : .virtualenvs/ansible-2.10/lib/python3.8/site-packages/ansible_test/_internal/coverage/html.py: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2022-01-24 15:14:34 ERROR : .virtualenvs/yt-dl/lib/python3.8/site-packages/youtube_dl/extractor/__pycache__/mdr.cpython-38.pyc: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2022-01-24 15:14:35 ERROR : .vscode/extensions/ms-python.vscode-pylance-2021.12.0/dist/typeshed-fallback/stdlib/spwd.pyi: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
Transferred:       11.322G / 13.114 GBytes, 86%, 856.433 kBytes/s, ETA 36m34s
Errors:             11576 (retrying may help)
Transferred:        17091 / 27107, 63%
Elapsed time:    3h51m5.1s
Transferring:
 * .arduino15/packages/es…fi_slot/analogRead.cpp:  0% /1.521k, 0/s, -
 * .vscode/extensions/ms-…llback/stdlib/spwd.pyi:115% /305, 176/s, -
 * .virtualenvs/django_re…oject_name/asgi.py-tpl:  0% /428, 0/s, -
 * .local/lib/python3.8/s…/LC_MESSAGES/django.mo:  0% /7.253k, 0/s, -
 * .local/lib/python3.8/s…lookups.cpython-38.pyc:  0% /5.669k, 0/s, -
 * .arduino15/packages/es…fi_slot/pins_arduino.h:  0% /2.613k, 0/s, -
 * .virtualenvs/yt-dl/lib…dialaan.cpython-38.pyc:  0% /7.013k, 0/s, -
 * .virtualenvs/terraform…ormation/yamlhelper.py:  0% /3.271k, 0/s, -^C


Indeed, the gdrive is full, and there is no single file in the scaleway bucket.

I feel I didn't use the policies the right way :slight_smile:

Please test on the latest version as that's very old.

Please include a full debug log.

Hello,

Thanks for your reply.

There is the full log from the first command, without -vvv (command I stopped with CTRL+C).
If I try to relaunch the same command, I get the following error.

2022-01-24 16:59:59 ERROR : snap/anbox-installer/current: Listing error: symlink: stat /home/darato/snap/anbox-installer/current: no such file or directory
2022-01-24 16:59:59 ERROR : tmp-olo/root/.#fdisk-u-l: Listing error: symlink: stat /home/darato/tmp-olo/root/.#fdisk-u-l: no such file or directory
2022-01-24 16:59:59 ERROR : tmp-olo/root/.#fdisk-l: Listing error: symlink: stat /home/darato/tmp-olo/root/.#fdisk-l: no such file or directory
2022-01-24 17:00:00 ERROR : tmp-olo/etc/os-release: Listing error: symlink: stat /home/darato/tmp-olo/etc/os-release: no such file or directory
2022-01-24 17:00:00 ERROR : tmp-olo/etc/mtab: Listing error: symlink: stat /home/darato/tmp-olo/etc/mtab: no such file or directory
Transferred:              0 B / 354.858 MiB, 0%, 0 B/s, ETA -
Errors:                 4 (no need to retry)
Checks:                35 / 35, 100%
Transferred:            0 / 70, 0%
Elapsed time:         4.4s
Transferring:
 *                                 .bash_history:  0% /480.261Ki, 0/s, -
 *                                      .viminfo:  0% /36.130Ki, 0/s, -
 *                    scripts/SSDSCRIPT-Linux.sh:  0% /1.851Ki, 0/s, -
 *                                 sidenoder/7za:  0% /1.390Mi, 0/s, -
 *                         sidenoder/config.json:  0% /593, 0/s, -
 *                      sidenoder/debug_last.log:  0% /6.938Ki, 0/s, -
 *                              sidenoder/rclone:  0% /41.148Mi, 0/s, -
 *                         sidenoder/rclone.conf:  0% /3.895Ki, 0/s, -panic: interface conversion: upstream.Entry is nil, not *upstream.Object

goroutine 106 [running]:
github.com/rclone/rclone/backend/union.(*Object).Update(0xc00014b260, {0x1e6bfb0, 0xc0005ce540}, {0x1e30b00, 0xc00013e900}, {0x7f812183e388, 0xc0006c2b40}, {0xc0001f2fe0, 0x1, 0x1})
	github.com/rclone/rclone/backend/union/entry.go:76 +0x598
github.com/rclone/rclone/fs/operations.Copy({0x1e6bfb0, 0xc0005ce540}, {0x1ea0ce8, 0xc0028b07e0}, {0x1ea0d58, 0xc00014b260}, {0xc0007e7060, 0x3}, {0x1e9fd98, 0xc0006c2b40})
	github.com/rclone/rclone/fs/operations/operations.go:469 +0x1a16
github.com/rclone/rclone/fs/sync.(*syncCopyMove).pairCopyOrMove(0xc0005d3600, {0x1e6bfb0, 0xc0005ce540}, 0x0, {0x1ea0ce8, 0xc0028b07e0}, 0x3, 0x0)
	github.com/rclone/rclone/fs/sync/sync.go:402 +0x1bb
created by github.com/rclone/rclone/fs/sync.(*syncCopyMove).startTransfers
	github.com/rclone/rclone/fs/sync/sync.go:429 +0x47

To get throught his error I had before, I had to delete all the remote content and upload it again.
I'll do this tomorrow as I don't have enough bandwitch to upload 15GB at home :slight_smile:

I tried the first command with a newer rclone release :

rclone v1.57.0
- os/version: debian bookworm/sid (64 bit)
- os/kernel: 5.15.0-1-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.17.2
- go/linking: static
- go/tags: none

That's just a snippet of a log. It's missing the top part.

Sorry you're right, I forgot the -vvv !!

rclone sync --transfers 8 --copy-links -P --exclude-from=./exclude /home/darato/ union_backup_1:thinkpad/ -vvv
2022/01/24 17:05:36 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "sync" "--transfers" "8" "--copy-links" "-P" "--exclude-from=./exclude" "/home/darato/" "union_backup_1:thinkpad/" "-vvv"]
2022/01/24 17:05:36 DEBUG : Creating backend with remote "/home/darato/"
2022/01/24 17:05:36 DEBUG : local: detected overridden config - adding "{12rtk}" suffix to name
2022/01/24 17:05:36 DEBUG : Using config file from "/home/darato/.config/rclone/rclone.conf"
2022/01/24 17:05:36 DEBUG : fs cache: renaming cache item "/home/darato/" to be canonical "local{12rtk}:/home/darato/"
2022/01/24 17:05:36 DEBUG : Creating backend with remote "union_backup_1:thinkpad/"
2022/01/24 17:05:36 DEBUG : Creating backend with remote "gcrypt_gd"
2022/01/24 17:05:36 DEBUG : Creating backend with remote "scway_standard_1_crypt:"
2022/01/24 17:05:36 DEBUG : Creating backend with remote "gdrive:rc"
2022/01/24 17:05:36 DEBUG : Creating backend with remote "scway_standard_1:rclone-backup/rclone"
2022/01/24 17:05:36 DEBUG : Creating backend with remote "scway_standard_1_crypt:/thinkpad"
2022/01/24 17:05:36 DEBUG : Google drive root 'rc': 'root_folder_id = 0POue6G_eoPkiUk5FTU' - save this in the config to speed up startup
2022/01/24 17:05:36 DEBUG : Creating backend with remote "scway_standard_1:r-backup-1/rclone/jr158nuc9im6pjc7sb7ehse97k"
2022/01/24 17:05:37 DEBUG : Creating backend with remote "gcrypt_gd_:/thinkpad"
2022/01/24 17:05:37 DEBUG : Creating backend with remote "gdrive:rc/026zkazejoaut5su7ajoolzaoekjg"
2022/01/24 17:05:37 DEBUG : Google drive root 'rc/026zkazejoaut5su7ajoolzaoekjg': 'root_folder_id = 0POue6G_eoPkiUk5FTU' - save this in the config to speed up startup
2022/01/24 17:05:38 DEBUG : union root 'thinkpad/': actionPolicy = *policy.EpLfs, createPolicy = *policy.EpLfs, searchPolicy = *policy.FF
2022-01-24 17:05:38 DEBUG : .arduino: Excluded
2022-01-24 17:05:38 DEBUG : .wine: Excluded
2022-01-24 17:05:38 DEBUG : .ssh: Excluded
2022-01-24 17:05:38 DEBUG : .gnupg: Excluded
2022-01-24 17:05:38 DEBUG : .cache: Excluded
2022-01-24 17:05:38 DEBUG : .aws: Excluded
2022-01-24 17:05:38 DEBUG : VirtualBox VMs: Excluded
2022-01-24 17:05:38 DEBUG : .terraform.d: Excluded
2022-01-24 17:05:38 DEBUG : .gocode: Excluded
2022-01-24 17:05:38 DEBUG : Vidéos: Excluded
2022-01-24 17:05:39 DEBUG : .bash_history: Sizes differ (src 491909 vs dst 491472)
2022-01-24 17:05:39 DEBUG : .ICEauthority: Size and modification time the same (differ by -587.912µs, within tolerance 1ms)
2022-01-24 17:05:39 DEBUG : .ICEauthority: Unchanged skipping
2022-01-24 17:05:39 DEBUG : .bash_logout: Size and modification time the same (differ by -816.71µs, within tolerance 1ms)
2022-01-24 17:05:39 DEBUG : .bash_logout: Unchanged skipping
2022-01-24 17:05:39 DEBUG : .bashdb_hist: Size and modification time the same (differ by -233.181µs, within tolerance 1ms)
2022-01-24 17:05:39 DEBUG : .bashdb_hist: Unchanged skipping
2022-01-24 17:05:39 DEBUG : .bashrc: Size and modification time the same (differ by -255.359µs, within tolerance 1ms)
2022-01-24 17:05:39 DEBUG : .bashrc: Unchanged skipping
2022-01-24 17:05:39 DEBUG : .bashrc_olorin: Size and modification time the same (differ by -405.896µs, within tolerance 1ms)
2022-01-24 17:05:39 DEBUG : .bashrc_olorin: Unchanged skipping
2022-01-24 17:05:39 DEBUG : .boto: Size and modification time the same (differ by -436.079µs, within tolerance 1ms)
2022-01-24 17:05:39 DEBUG : .boto: Unchanged skipping
2022-01-24 17:05:39 DEBUG : .emacs: Size and modification time the same (differ by -538.402µs, within tolerance 1ms)
2022-01-24 17:05:39 DEBUG : .emacs: Unchanged skipping
2022-01-24 17:05:39 DEBUG : .gitconfig: Size and modification time the same (differ by -458.208µs, within tolerance 1ms)
2022-01-24 17:05:39 DEBUG : .gitconfig: Unchanged skipping
2022-01-24 17:05:39 DEBUG : .gtk-bookmarks: Size and modification time the same (differ by -455.77µs, within tolerance 1ms)
2022-01-24 17:05:39 DEBUG : .gtk-bookmarks: Unchanged skipping
2022-01-24 17:05:39 DEBUG : .lesshst: Size and modification time the same (differ by -125.697µs, within tolerance 1ms)
2022-01-24 17:05:39 DEBUG : .lesshst: Unchanged skipping
2022-01-24 17:05:39 DEBUG : .node_repl_history: Size and modification time the same (differ by -724.448µs, within tolerance 1ms)
2022-01-24 17:05:39 DEBUG : .node_repl_history: Unchanged skipping
2022-01-24 17:05:39 DEBUG : .profile: Size and modification time the same (differ by -816.71µs, within tolerance 1ms)
2022-01-24 17:05:39 DEBUG : .profile: Unchanged skipping
2022-01-24 17:05:39 DEBUG : .python_history: Size and modification time the same (differ by -578.115µs, within tolerance 1ms)
2022-01-24 17:05:39 DEBUG : .python_history: Unchanged skipping
2022-01-24 17:05:39 DEBUG : .recently-used: Size and modification time the same (differ by -999.006µs, within tolerance 1ms)
2022-01-24 17:05:39 DEBUG : .recently-used: Unchanged skipping
2022-01-24 17:05:39 DEBUG : .terraformrc: Size and modification time the same (differ by -223.724µs, within tolerance 1ms)
2022-01-24 17:05:39 DEBUG : .terraformrc: Unchanged skipping
2022-01-24 17:05:39 DEBUG : .vault-token: Size and modification time the same (differ by -46.079µs, within tolerance 1ms)
2022-01-24 17:05:39 DEBUG : .vault-token: Unchanged skipping
2022-01-24 17:05:39 DEBUG : .viminfo: Sizes differ (src 36997 vs dst 29007)
2022-01-24 17:05:39 DEBUG : .wget-hsts: Size and modification time the same (differ by -956.292µs, within tolerance 1ms)
2022-01-24 17:05:39 DEBUG : .wget-hsts: Unchanged skipping
2022-01-24 17:05:39 DEBUG : 2018.pdf: Size and modification time the same (differ by -934µs, within tolerance 1ms)
2022-01-24 17:05:39 DEBUG : 2018.pdf: Unchanged skipping
2022-01-24 17:05:39 DEBUG : bash_history-2020-02-23T19:05:24+01:00: Size and modification time the same (differ by -152.431µs, within tolerance 1ms)
2022-01-24 17:05:39 DEBUG : bash_history-2020-02-23T19:05:24+01:00: Unchanged skipping
2022-01-24 17:05:39 DEBUG : core: Size and modification time the same (differ by -6.508µs, within tolerance 1ms)
2022-01-24 17:05:39 DEBUG : core: Unchanged skipping
2022-01-24 17:05:39 DEBUG : hs_err_pid16675.log: Size and modification time the same (differ by -847.632µs, within tolerance 1ms)
2022-01-24 17:05:39 DEBUG : hs_err_pid16675.log: Unchanged skipping
2022-01-24 17:05:39 DEBUG : goip.go: Size and modification time the same (differ by -626.419µs, within tolerance 1ms)
2022-01-24 17:05:39 ERROR : snap/anbox-installer/current: Listing error: symlink: stat /home/darato/snap/anbox-installer/current: no such file or directory
2022-01-24 17:05:39 DEBUG : .config/Bitwarden: Excluded
2022-01-24 17:05:39 DEBUG : kube.tar.gz: Size and modification time the same (differ by -730.654µs, within tolerance 1ms)
2022-01-24 17:05:39 DEBUG : kube.tar.gz: Unchanged skipping
2022-01-24 17:05:39 DEBUG : kubectl: Size and modification time the same (differ by -297.68µs, within tolerance 1ms)
2022-01-24 17:05:39 DEBUG : mbox: Size and modification time the same (differ by -260.603µs, within tolerance 1ms)
2022-01-24 17:05:39 DEBUG : mozilla.pdf: Size and modification time the same (differ by -91.122µs, within tolerance 1ms)
2022-01-24 17:05:39 DEBUG : msg_jeedom: Size and modification time the same (differ by -30.738µs, within tolerance 1ms)
2022-01-24 17:05:39 DEBUG : test: Size and modification time the same (differ by -94.664µs, within tolerance 1ms)
2022-01-24 17:05:39 DEBUG : goip.go: Unchanged skipping
2022-01-24 17:05:39 DEBUG : kubectl: Unchanged skipping
2022-01-24 17:05:39 DEBUG : mbox: Unchanged skipping
2022-01-24 17:05:39 DEBUG : testsops.yml: Size and modification time the same (differ by -971.069µs, within tolerance 1ms)
2022-01-24 17:05:39 DEBUG : mozilla.pdf: Unchanged skipping
2022-01-24 17:05:39 DEBUG : msg_jeedom: Unchanged skipping
2022-01-24 17:05:39 DEBUG : test: Unchanged skipping
2022-01-24 17:05:39 DEBUG : testsops.yml: Unchanged skipping
Transferred:              0 B / 319.895 MiB, 0%, 0 B/s, ETA -
Errors:                 1 (no need to retry)
Checks:                34 / 35, 97%
Transferred:            0 / 13, 0%
Elapsed time:         2.7s
Checking:
 *                                  testsops.yml: checking
Transferring:
 *                                 .bash_history:  0% /480.380Ki, 0/s, -
 *                    scripts/SSDSCRIPT-Linux.sh:  0% /1.851Ki, 0/s, -
 *                                 sidenoder/7za:  0% /1.390Mi, 0/s, -
 *                         sidenoder/config.json:  0% /593, 0/s, -
 *                      sidenoder/debug_last.log:  0% /6.938Ki, 0/s, -
 *                              sidenoder/rclone:  0% /41.148Mi, 0/s, -
 *                         sidenoder/rclone.conf:  0% /3.895Ki, 0/s, -
 * temp_teams/google-…able_current_amd64.deb:  0% /65.497Mi, 0/s, -panic: interface conversion: upstream.Entry is nil, not *upstream.Object

goroutine 111 [running]:
github.com/rclone/rclone/backend/union.(*Object).Update(0xc0017aeba0, {0x1e6bfb0, 0xc000654700}, {0x1e30b00, 0xc000708500}, {0x7f6333d944d8, 0xc0001198c0}, {0xc0001e1600, 0x1, 0x1})
        github.com/rclone/rclone/backend/union/entry.go:76 +0x598
github.com/rclone/rclone/fs/operations.Copy({0x1e6bfb0, 0xc000654700}, {0x1ea0ce8, 0xc00036a0e0}, {0x1ea0d58, 0xc0017aeba0}, {0xc00052b090, 0x0}, {0x1e9fd98, 0xc0001198c0})
	github.com/rclone/rclone/fs/operations/operations.go:469 +0x1a16
github.com/rclone/rclone/fs/sync.(*syncCopyMove).pairCopyOrMove(0xc0003b4000, {0x1e6bfb0, 0xc000654700}, 0x0, {0x1ea0ce8, 0xc00036a0e0}, 0x0, 0x0)
	github.com/rclone/rclone/fs/sync/sync.go:402 +0x1bb
created by github.com/rclone/rclone/fs/sync.(*syncCopyMove).startTransfers
	github.com/rclone/rclone/fs/sync/sync.go:429 +0x47

Can you share the rclone about for that remote?

rclone about gcrypt_gd:

Hello,

rclone about gcrypt_gd:
Total:   15 GiB
Used:    14.990 GiB
Free:    11 B
Trashed: 4.414 GiB
Other:   10.038 MiB

I think @Ole wrestled with this one before from some searching.

The policy eplfs does exactly what it says, but for the life of me, I can't see any real world application for this as the remote it is trying to write to has the last free but not enough space to write the files out.

eplfs (existing path, least free space)	Of all the upstreams on which the relative path exists choose the one with the least free space.

I wanted to validate the free drive showed space and it does.

So the remote isn't actually 'full' but has some space left so the policy writes to that.

I think there were some feature requests out there to make it move to the next remote but I'd have to see if I can find it.

Hi Darato,

I fully agree with @Animosity022, lfs/eplfs (least free space) does exactly what it says. It wasn’t designed to save cost on free Drives, but inspired by the policies of trapexit/mergerfs.

The policy you are looking for is: “Create category: Pick the (first) upstream with enough free space to create the file.”, but this policy doesn’t exist. I also guess it would be difficult to implement given the high parallelism of rclone.

I therefore see no possibility to fulfil your wish for sequentially filled upstreams, sorry.

I think (ep)lfs in rclone isn't working as designed.

If you read the mergerfs docs then I think it can work exactly as the @Darato wanted.

However it has two very important options minfreespace which sets the minimum free space a drive can have and moveonenospc which moves the file from a full disk to an empty one if it gets an out of space error (ENOSPC)

  • minfreespace=SIZE: The minimum space value used for creation policies. Can be overridden by branch specific option. Understands 'K', 'M', and 'G' to represent kilobyte, megabyte, and gigabyte respectively. (default: 4G)
  • moveonenospc=BOOL|POLICY: When enabled if a write fails with ENOSPC (no space left on device) or EDQUOT (disk quota exceeded) the policy selected will run to find a new location for the file. An attempt to move the file to that branch will occur (keeping all metadata possible) and if successful the original is unlinked and the write retried. (default: false, true = mfs)

Unfortunately we don't have a standardised error code for disk full (maybe we should!) and reading partially written files is generally impossible in cloud storage so we can't implement moveonnospc easily without buffering the file locally.

However I think the minfreespace options is very implementable and provided it was set to bigger than the largest file you wanted to upload it would fix the problem.

What do you think?

2 Likes

That doesn't sound bad at all. I'd probably suggest to log a feature request for it (I think it's neat but low priority) as if it gets traction and lots of want on the request, it can get moved up.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.