Can't move inside union rclone mount

What is the problem you are having with rclone?

cannot rename or move files inside an Union rclone mount.
mv a b
mv: cannot move 'a' to 'b': Input/output error

Is there a way move work inside a Union mount?

What is your rclone version (output from rclone version)

v1.56.1
ubuntu 20.04 lts server

Which cloud storage system are you using? (eg Google Drive)

union of google cloud storage and local

The command you were trying to run (eg rclone copy /tmp remote:tmp)

 mv a b

The rclone config contents with secrets removed.

[remote]
type = google cloud storage
service_account_file = ***

[union]
type = union
action_policy = all
create_policy = all
search_policy = ff

A log from the command with the -vv flag

2021/09/22 13:05:13 DEBUG : /: Attr:
2021/09/22 13:05:13 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2021/09/22 13:05:13 DEBUG : /: Lookup: name="a"
2021/09/22 13:05:13 DEBUG : /: >Lookup: node=a, err=<nil>
2021/09/22 13:05:13 DEBUG : a: Attr:
2021/09/22 13:05:13 DEBUG : a: >Attr: a=valid=1s ino=0 size=0 mode=-rw-rw-r--, err=<nil>
2021/09/22 13:05:13 DEBUG : /: Lookup: name="b"
2021/09/22 13:05:13 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2021/09/22 13:05:13 DEBUG : /: Lookup: name="b"
2021/09/22 13:05:13 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2021/09/22 13:05:13 DEBUG : /: Lookup: name="b"
2021/09/22 13:05:13 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2021/09/22 13:05:13 DEBUG : /: Rename: oldName="a", newName="b", newDir=/
2021/09/22 13:05:13 ERROR : a: Dir.Rename error: Fs "union root ''" can't rename files (no server-side Move or Copy)
2021/09/22 13:05:13 ERROR : a: Dir.Rename error: Fs "union root ''" can't rename files (no server-side Move or Copy)
2021/09/22 13:05:13 DEBUG : /: >Rename: err=Fs "union root ''" can't rename files (no server-side Move or Copy)
2021/09/22 13:05:13 DEBUG : /: Lookup: name="b"
2021/09/22 13:05:13 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2021/09/22 13:05:13 DEBUG : /: Lookup: name="b"
2021/09/22 13:05:13 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory

Does it work without the union?

yes. without union (only remote with gcs) it works.

Can you share a log as I can't see your screen? Thanks.

the error log is on the first post.
or need the log without union?

You said:

Can you share that log.

the log without union. (only remote with gcs)

2021/09/22 14:10:45 DEBUG : /: Attr:
2021/09/22 14:10:45 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2021/09/22 14:10:45 DEBUG : /: Lookup: name="b"
2021/09/22 14:10:45 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2021/09/22 14:10:45 DEBUG : /: Lookup: name="b"
2021/09/22 14:10:45 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2021/09/22 14:10:45 DEBUG : /: Lookup: name="b"
2021/09/22 14:10:45 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2021/09/22 14:10:45 DEBUG : /: Rename: oldName="a", newName="b", newDir=/
2021/09/22 14:10:45 DEBUG : a: md5 = d41d8cd98f00b204e9800998ecf8427e OK
2021/09/22 14:10:45 INFO  : a: Copied (server-side copy) to: b
2021/09/22 14:10:46 INFO  : a: Deleted
2021/09/22 14:10:46 INFO  : a: vfs cache: renamed in cache to "b"
2021/09/22 14:10:46 DEBUG : b: Updating file with b 0xc0002bc480
2021/09/22 14:10:46 DEBUG : : Added virtual directory entry vDel: "a"
2021/09/22 14:10:46 DEBUG : : Added virtual directory entry vAddFile: "b"
2021/09/22 14:10:46 DEBUG : /: >Rename: err=<nil>
2021/09/22 14:10:46 DEBUG : /: Invalidating "b"

This is the problem...

What upstreams have you got in your union? You seem to have left that out of your config.

For server side move / copy all the upstreams need to support server side move or copy.

1 Like
rclone mount union: /union --union-upstreams "/local remote:BUCKET" --umask=0002 --allow-other --default-permissions

have just /local and remote:BUCKET upstreams.

A mount with just remote:BUCKET and no union works fine.

And local is local ubuntu 20.04 lts.

Both support server side move or copy.

Can you report the results of

rclone backend features xxxx:

On all the parts of the union and on the union itself please? So

rclone backend features /local
rclone backend features remote:BUCKET
rclone backend features union:

sure...

rclone backend features /local

{
        "Name": "local",
        "Root": "/local",
        "String": "Local file system at /local",
        "Precision": 1,
        "Hashes": [
                "md5",
                "sha1",
                "whirlpool",
                "crc32",
                "dropbox",
                "mailru",
                "quickxor"
        ],
        "Features": {
                "About": true,
                "BucketBased": false,
                "BucketBasedRootOK": false,
                "CanHaveEmptyDirectories": true,
                "CaseInsensitive": false,
                "ChangeNotify": false,
                "CleanUp": false,
                "Command": true,
                "Copy": false,
                "DirCacheFlush": false,
                "DirMove": true,
                "Disconnect": false,
                "DuplicateFiles": false,
                "GetTier": false,
                "IsLocal": true,
                "ListR": false,
                "MergeDirs": false,
                "Move": true,
                "OpenWriterAt": true,
                "PublicLink": false,
                "Purge": true,
                "PutStream": true,
                "PutUnchecked": false,
                "ReadMimeType": false,
                "ServerSideAcrossConfigs": false,
                "SetTier": false,
                "SetWrapper": false,
                "Shutdown": false,
                "SlowHash": true,
                "SlowModTime": false,
                "UnWrap": false,
                "UserInfo": false,
                "WrapFs": false,
                "WriteMimeType": false
        }
}

rclone backend features remote:BUCKET

{
        "Name": "remote",
        "Root": "BUCKET",
        "String": "GCS bucket BUCKET",
        "Precision": 1,
        "Hashes": [
                "md5"
        ],
        "Features": {
                "About": false,
                "BucketBased": true,
                "BucketBasedRootOK": true,
                "CanHaveEmptyDirectories": false,
                "CaseInsensitive": false,
                "ChangeNotify": false,
                "CleanUp": false,
                "Command": false,
                "Copy": true,
                "DirCacheFlush": false,
                "DirMove": false,
                "Disconnect": false,
                "DuplicateFiles": false,
                "GetTier": false,
                "IsLocal": false,
                "ListR": true,
                "MergeDirs": false,
                "Move": false,
                "OpenWriterAt": false,
                "PublicLink": false,
                "Purge": false,
                "PutStream": true,
                "PutUnchecked": false,
                "ReadMimeType": true,
                "ServerSideAcrossConfigs": false,
                "SetTier": false,
                "SetWrapper": false,
                "Shutdown": false,
                "SlowHash": false,
                "SlowModTime": false,
                "UnWrap": false,
                "UserInfo": false,
                "WrapFs": false,
                "WriteMimeType": true
        }
}

rclone backend features union: --union-upstreams "/local remote:BUCKET"

{
        "Name": "union",
        "Root": "",
        "String": "union root ''",
        "Precision": 1,
        "Hashes": [
                "md5"
        ],
        "Features": {
                "About": false,
                "BucketBased": false,
                "BucketBasedRootOK": false,
                "CanHaveEmptyDirectories": false,
                "CaseInsensitive": false,
                "ChangeNotify": false,
                "CleanUp": false,
                "Command": false,
                "Copy": false,
                "DirCacheFlush": false,
                "DirMove": false,
                "Disconnect": false,
                "DuplicateFiles": false,
                "GetTier": false,
                "IsLocal": false,
                "ListR": true,
                "MergeDirs": false,
                "Move": false,
                "OpenWriterAt": false,
                "PublicLink": false,
                "Purge": false,
                "PutStream": true,
                "PutUnchecked": false,
                "ReadMimeType": false,
                "ServerSideAcrossConfigs": false,
                "SetTier": false,
                "SetWrapper": false,
                "Shutdown": false,
                "SlowHash": false,
                "SlowModTime": false,
                "UnWrap": false,
                "UserInfo": false,
                "WrapFs": false,
                "WriteMimeType": false
        }
}

I see what is going on...

In order to rename a file you need to be able to Move it or Copy it then delete it (used in s3/gcs/swift/b2/etc as you can't rename an object)

The local backend can Move but not Copy

The bucket can Copy but not Move

The union's features are the AND of the underlying remotes so the union can't Move or Copy

So when rclone tries to move a file in the union it can't Move or Copy so it gives up.

The best solution that I can think of is for the union to advertise Move if all the underlying remotes support Move or Copy. Then in its own implementation of Move it should be turning Move into Copy+Delete on remotes which only support Copy.

I don't this would be too tricky to implement.

Can you open a new issue on Github about this and put a link to this forum post in please?

1 Like

done!

thanks a lot!

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.