Clarification on union policies for automatic directory creation

Matt812g · August 29, 2023, 2:46pm

What is the problem you are having with rclone?

I created a union between 2 new remotes. I did not specify any policies, so I assume they got the defaults, which I assume are (per current website docs):

action_policy = epall
create_policy = epmfs
search_policy = ff

I then tried to sync between my local drive and the union. I had 5 directories in my local drive. The first three were called (for example):

local:/large1
local:/large2
local:/small

When the sync occurred, these directories were created:

gcrypt5:/large1
gcrypt6:/small
gcrypt7:/large2

The sync went fine for a while. gcrypt6: ended up with all files from /small, which was fine. Then gcrypt5: and gcrypt7: filled up. I then started getting quota errors and nothing else would sync. Of course, this happened because /large1 files were only being created in the gcrypt5: remote. Once I realized that the policy options were the issue, I changed the policy options to what you can see in the below snippet of my rclone.conf file. Basically, for the union:

action-policy = all
create-policy = mfs
search-policy = ff

I assumed this would solve the problem. However, I still got the same errors when I tried to sync.

However, if I manually create the destination directories in the not-full remotes, then syncing continues fine, and the new files are created in the not-full remotes in the directories I manually created.

As I'm sure you can imagine, this is not ideal.

My questions:

What is the proper configuration for a union to automatically create directories when they are needed on remotes that have room for more files?
What did I do wrong?
Why did I have to manually create the directories?
Why wasn't switching policies enough?

Thanks!

Run the command 'rclone version' and share the full output of the command.

$ rclone version
rclone v1.62.2-DEV

os/version: alpine 3.18.2 (64 bit)
os/kernel: 6.3.5 (aarch64)
os/type: linux
os/arch: arm64 (ARMv8 compatible)
go/version: go1.20.7
go/linking: static
go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive(s)

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone sync /some/path videounion:/ -v --links --local-zero-size-links --fast-list --drive-use-trash=false --check-first --delete-before --delete-excluded --transfers 1 --create-empty-src-dirs

The rclone config contents with secrets removed.

[gdrive5]
type = drive
client_id = REDACTED
client_secret = REDACTED
scope = drive
token = REDACTED
team_drive = 
use_trash = false

[gcrypt5]
type = crypt
remote = gdrive5:/
filename_encryption = standard
password = REDACTED
password2 = REDACTED
links = true

[gdrive6]
type = drive
client_id = REDACTED
client_secret = REDACTED
scope = drive
team_drive = 
token = REDACTED
use_trash = false

[gcrypt6]
type = crypt
remote = gdrive6:/
filename_encryption = standard
password = REDACTED
password2 = REDACTED
links = true

[gdrive7]
type = drive
client_id = REDACTED
client_secret = REDACTED
scope = drive
token = REDACTED
team_drive = 
use_trash = false

[gcrypt7]
type = crypt
remote = gdrive7:/
filename_encryption = standard
password = REDACTED
password2 = REDACTED
links = true

[videounion]
type = union
upstreams = gcrypt5:/ gcrypt6:/ gcrypt7:/
action-policy = all
create-policy = mfs
search-policy = ff

A log from the command with the `-vv` flag

Unfortunately, I do not have a -vv log because I manually fixed the issue.
Is it possible to answer this without it? If not, I will undo my most recent sync and try again.

Matt812g · August 29, 2023, 3:44pm

Also, note that the above only worked for the upper-level directories. Every new subdirectory under that has to manually be added, as well. This seems contrary to design, so I wonder if I there is a bug, or I am just doing something wrong.

Animosity022 · August 29, 2023, 4:24pm

Missing your rclone config with the information redacted.

Log file would be super helpful, but the config is 100% needed.

Matt812g · August 29, 2023, 4:29pm

@Animosity022 Config is there.

Matt812g · August 29, 2023, 4:30pm

Let me try removing the greater/lessthan symbols. Maybe that is messing with your browser?

Matt812g · August 29, 2023, 4:31pm

Okay, I removed them. Hopefully it will show for you now @Animosity022 .

Animosity022 · August 29, 2023, 4:32pm

Are they team drives?

Matt812g · August 29, 2023, 4:44pm

They are NOT team drives. Weird that the config tool puts the blank entries in there.

kapitainsky · August 29, 2023, 5:02pm

could you please post:

rclone about gcrypt5:
rclone about gcrypt6:
rclone about gcrypt7:

Matt812g · August 29, 2023, 5:07pm

$ rclone about gcrypt5:
Total:   15 GiB
Used:    14.965 GiB
Free:    35.388 MiB
Trashed: 0 B
Other:   0 B

$ rclone about gcrypt6:
Total:   15 GiB
Used:    9.510 GiB
Free:    5.490 GiB
Trashed: 0 B
Other:   0 B

$ rclone about gcrypt7:
Total:   15 GiB
Used:    14.998 GiB
Free:    1.856 MiB
Trashed: 0 B
Other:   0 B

Matt812g · August 29, 2023, 5:28pm

I kind of wonder if there is an issue with union--crypts--drives.
Might work better do crypt--union--drives.
Might be a cleaner config, too.

kapitainsky · August 29, 2023, 5:31pm

I am not sure myself.

Your policy should work - I use it myself.

    action-policy = epall
    create-policy = mfs
    search-policy = ff

for action epall or all should behave the same.

But indeed I use crypt--union--drives.

So more questions than answers:)

Matt812g · August 29, 2023, 5:32pm

Here is my shell script to synchronize the directories of all union members for when it starts throwing:

for REMOTEOUT in gcrypt5 gcrypt6 gcrypt7; do 
    rclone lsf $REMOTEOUT:/ -R --dirs-only; 
done | sort -u | while read DIR; do 
    printf "$DIR: \n"; 
    for REMOTEIN in gcrypt5 gcrypt6 gcrypt7; do 
        printf "$REMOTEIN:\n"; 
        rclone mkdir "$REMOTEIN:/$DIR"; 
    done; 
    printf "\n"; 
done

Matt812g · August 29, 2023, 5:33pm

Interesting! I will try to change to crypt-union-drives and see if that fixes it.

The other additional question is, by starting out with different policies, did I somehow "lock them in" to that policy config and that is why I have this issue with the new policy config?

kapitainsky · August 29, 2023, 5:36pm

hahah - yes you locked them in - they are always default:

OK - this is where problem is - your policies are ignored...

it should be:

action_policy = all
create_policy = mfs
search_policy = ff

Matt812g · August 29, 2023, 5:38pm

You got it!

I feel like a total idiot!

Thanks for your help, @kapitainsky !

kapitainsky · August 29, 2023, 5:39pm

actually your comment about "lock them in" made me think about it from different angle:)

Animosity022 · August 29, 2023, 5:40pm

Also, that is why we ask for a log file as with the log file, it would have been easy to spot that wrong policy rather than being an eye test.

Matt812g · August 29, 2023, 5:44pm

I copied the options from the commandline after testing and just took the first part off:
--union-action-policy -> action-policy

So, the commandline uses - and the conf file uses _, which may be a problem for anyone if they aren't very careful.

Thanks, all!

system · September 1, 2023, 5:44pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.