Rclone rc backend/command not working as expected

Iguana9999 · August 31, 2020, 9:56am

What is the problem you are having with rclone?

Trying to switch over a teamdrive mounted by rclone with a service account. While the output shows it switches over, the config file gets updated, the mount is still using the original SA which it started with.
At the moment SA1.json has errors on files, running the command seems to switch it over to SA2.json (it does in the output and in the config) but mount will only start using SA2.json on a remount.

What is your rclone version (output from `rclone version`)

v1.52.3 (tested with v1.50 to latest beta, all the same for me)

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Debian 10

Which cloud storage system are you using? (eg Google Drive)

Google team drive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

 rclone -q rc backend/command command=set fs=clouddrive: -o service_account_file=SA.json

The rclone config contents with secrets removed.

[clouddrive]
type = drive
scope = drive
service_account_file = SA1.json
team_drive = *
chunk_size = 64M

[cloudcrypt]
type = crypt
remote = clouddrive:Crypt2
filename_encryption = standard
password = *
password2 = *

A log from the command with the `-vv` flag

2020/08/31 17:53:14 DEBUG : rclone: Version "v1.52.3" starting with parameters ["rclone" "-vv" "rc" "backend/command" "command=set" "fs=clouddrive:" "-o" "service_account_file=/SA1.json"]
{
        "result": {
                "service_account_file": {
                        "current": "SA1.json",
                        "previous": "SA2.json"
                }
        }
}
2020/08/31 17:53:14 DEBUG : 4 go routines active

Animosity022 · August 31, 2020, 10:46am

Check out:

Iguana9999 · August 31, 2020, 10:48am

Thank, I did (multiple times) and re-read it multiple times today. As stated above I use the rc command as stated there. If there is something obvious that I'm missing in that post let me know

Animosity022 · August 31, 2020, 10:49am

This line:

Iguana9999 · August 31, 2020, 11:09am

Thanks but that is for updating the config.

As stated there the rclone rc backend/command command=set fs=clouddrive: should be used to change the service account of a running mount. It SHOULD re-authenticate after running that command but for me it doesn't. I'm doing exactly as stated there and it looks like it works except for the auth part.

darthShadow · August 31, 2020, 11:32am

What's your mount command?

Iguana9999 · August 31, 2020, 11:41am

See below, going to test it later with minimal mount command.

[Unit]
Description=
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
User=X
Group=X
ExecStart=/usr/bin/rclone mount \
        --config /home/X/.config/rclone/rclone.conf \
        --read-only \
        --allow-other \
        --allow-non-empty \
        --dir-cache-time=96h \
        --drive-chunk-size 32M \
        --buffer-size 256M \
        --vfs-read-chunk-size=128M \
        --vfs-read-chunk-size-limit off \
        --timeout 1h \
        --rc \
        --rc-no-auth \
        --log-level=DEBUG \
        --log-file=/home/X/logs/clouddrive.log \
        cloudcrypt: /home/X/.cloudcrypt
ExecStop=/bin/fusermount -uz /home/X/.cloudcrypt
Restart=on-abort

[Install]
WantedBy=default.target

I've complete a minimal test setup and the issue is still the same.
Mounted a test using a new setup config with an SA that is known to have issues.

rclone mount testcrypt: /home/X/.test --rc --rc-no-auth --rc-addr=localhost:6000 --allow-other -vvv

ran rsync and monitored logs of the mount, Errors on the file in sync.

ran:

rclone rc --rc-addr=localhost:6000 backend/command command=set fs=test: -o service_account_file=/home/X/.config/rclone/SA/SA2.json

{
        "result": {
                "service_account_file": {
                        "current": "/home/X/.config/rclone/SA/SA2.json",
                        "previous": "/home/X/.config/rclone/SA/SA1.json"
                }
        }
}

Monitored logs of the mount and errors are still there, after stopping the mount and starting it back up again the rsync ran fine without any errors.

darthShadow · August 31, 2020, 12:38pm

The problem here seems to be that you are mounting the crypt remote and running the command on the drive remote. I don't think the backend command infra. traverses wrapped remotes. @ncw Thoughts?

Just to rule it out, can you try the following too:

This is with the folder specified too

rclone rc --rc-addr=localhost:6000 backend/command command=set fs=clouddrive:Crypt2 -o service_account_file=/home/X/.config/rclone/SA/SA2.json

This is with the crypt remote

rclone rc --rc-addr=localhost:6000 backend/command command=set fs=cloudcrypt: -o service_account_file=/home/X/.config/rclone/SA/SA2.json

I don't think either of them will work however, tbh.

Iguana9999 · August 31, 2020, 12:48pm

rclone rc --rc-addr=localhost:6000 backend/command command=set fs=clouddrive:Crypt2 -o service_account_file=/home/X/.config/rclone/SA/SA2.json

command works, issue still there.

rclone rc --rc-addr=localhost:6000 backend/command command=set fs=cloudcrypt: -o service_account_file=/home/X/.config/rclone/SA/SA2.json

        "error": "command \"set\" failed: command not found",
        "input": {
                "command": "set",
                "fs": "testcrypt:",
                "opt": {
                        "service_account_file": "/home/X/.config/rclone/SA/SA2.json"
                }
        },
        "path": "backend/command",
        "status": 500
}

Doesn't work as RC doesn't run on crypt.

darthShadow · August 31, 2020, 12:50pm

Yep, that's what I expected. Will need to wait for ncw to chime in if there is something that can be done in the existing infra for this or something needs to be added to the infra.

ncw · August 31, 2020, 1:54pm

If you want to run backend commands on the clouddrive remote then you will have to use clouddrive:Crypt2 as the target. If you use clouddrive: then rclone will create a new remote and do the command on that, not do the command on the existing clouddrive:Crypt2 remote.

So I would have expected @darthShadow s suggestion to work.

If it isn't working then it means that the right backend isn't being found.

@Iguana9999 Can you try

v1.53.0-beta.4717.c31defbbd on branch master (uploaded in 15-30 mins)

with

rclone -q rc backend/command command=set fs=clouddrive:Crypt2 -o service_account_file=SA.json

I added a debug to show when backends are being created Creating backend with remote %q which should help debug what is going on.

Ideally we'd get the crypt backend to run translated commands on the backend it is wrapping, but I haven't figured out a sensible way of doing that yet.

Iguana9999 · August 31, 2020, 2:30pm

I needed to change the mount names so test: is clouddrive and testcrypt is cloudcrypt.

rclone mount testcrypt: /home/X/.test --rc --rc-no-auth --rc-addr=localhost:6000 --allow-other -vvv

2020/08/31 22:19:17 DEBUG : rclone: Version "v1.53.0-beta.4717.c31defbbd" starting with parameters ["rclone" "mount" "testcrypt:" "/home/ernesto/.test" "--rc" "--rc-no-auth" "--rc-addr=localhost:6000" "--allow-other" "-vvv"]
2020/08/31 22:19:17 NOTICE: Serving remote control on http://127.0.0.1:6000/
2020/08/31 22:19:17 DEBUG : Creating backend with remote "testcrypt:"
2020/08/31 22:19:17 DEBUG : Using config file from "/home/X/.config/rclone/rclone.conf"
2020/08/31 22:19:18 DEBUG : Encrypted drive 'testcrypt:': Mounting on "/home/X/.test"

rclone rc backend/command command=set fs=test:Crypt2 -o service_account_file=SA.json

rc: "backend/command": with parameters map[command:set fs:test:Crypt2 opt:map[service_account_file:/home/X/.config/rclone/SA/SA1.json]]
2020/08/31 22:24:30 DEBUG : Creating backend with remote "test:Crypt2"
2020/08/31 22:24:31 DEBUG : Changing Service Account File from /home/X/.config/rclone/SA/SA1.json to /home/X/.config/rclone/SA/SA2.json
2020/08/31 22:24:31 DEBUG : Saving config "service_account_file" = "/home/X/.config/rclone/SA/SA2.json" in section "test" of the config file

rclone rc backend/command command=set fs=test: -o service_account_file=SA.json

2020/08/31 22:33:54 DEBUG : Creating backend with remote "test:"

Also tested with the none encrypted mount and that doesn't show a creating backend upon the rc command.

2020/08/31 22:45:27 DEBUG : Creating backend with remote "test:"
2020/08/31 22:45:27 DEBUG : Using config file from "/home/X/.config/rclone/rclone.conf"
2020/08/31 22:45:27 DEBUG : Google drive root '': Mounting on "/home/X/.test"
2020/08/31 22:45:27 DEBUG : : Root:
2020/08/31 22:45:27 DEBUG : : >Root: node=/, err=<nil>
2020/08/31 22:45:30 DEBUG : : Statfs:
2020/08/31 22:45:30 DEBUG : Google drive root '': read info from team drive "X"
2020/08/31 22:45:30 DEBUG : : >Statfs: stat={Blocks:274877906944 Bfree:274877906944 Bavail:274877906944 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>
2020/08/31 22:45:33 DEBUG : rc: "backend/command": with parameters map[command:set fs:test: opt:map[service_account_file:/home/X/.config/rclone/SA/SA1.json]]
2020/08/31 22:45:33 DEBUG : Changing Service Account File from /home/X/.config/rclone/SA/SA1.json to /home/X/.config/rclone/SA/SA2.json
2020/08/31 22:45:33 DEBUG : Saving config "service_account_file" = "/home/X/.config/rclone/SA/SA2.json" in section "test" of the config file
2020/08/31 22:45:33 DEBUG : Keeping previous permissions for config file: -rwxr-xr-x

ncw · August 31, 2020, 4:51pm

OK, have a go with this - I think I worked out what the problem is - namely that crypt wasn't putting the drive backend into the cache so the rc command was never working.

I think I've fixed that and the backend lifecyle so hopefully this will work!

v1.53.0-beta.4719.32dcc9cd4.fix-wrap-backends on branch fix-wrap-backends (uploaded in 15-30 mins)

Iguana9999 · August 31, 2020, 11:54pm

Testing gives me this, I assume I need to send it to test:Crypt2 as that one is not making an new backend so it uses the correct mount. For completion both to test: and test:Crypt2 added.

I don't have an SA with issues at the moment but I assume it works

rclone rc --rc-addr=localhost:6000 backend/command command=set fs=test: -o service_account_file=/home/X/.config/rclone/SA/SA.json

2020/09/01 07:49:29 DEBUG : rc: "backend/command": with parameters map[command:set fs:test: opt:map[service_account_file:/home/X/.config/rclone/SA/SA]]
2020/09/01 07:49:29 DEBUG : Creating backend with remote "test:"

rclone rc --rc-addr=localhost:6000 backend/command command=set fs=test:Crypt2 -o service_account_file=/home/X/.config/rclone/SA/SA.json

2020/09/01 07:50:06 DEBUG : rc: "backend/command": with parameters map[command:set fs:test:Crypt2 opt:map[service_account_file:/home/ernesto/.config/rclone/SA/SA.json]]

ncw · September 1, 2020, 4:58pm

Yes that looks correct

I'll merge this into master shortly so it will be in v1.53 when it is released (this week )

Iguana9999 · September 2, 2020, 1:03am

Perfect! Should add a note to the example that when using an crypt the underlying FS:/path should be used instead of just the main FS.

Thanks for the always quick and on the point replies and fixes!

ncw · September 2, 2020, 5:15pm

Where is that example?

You could send a PR if you wanted!

PS v1.53 is released with this fix

Iguana9999 · September 2, 2020, 11:53pm

Thought I've seen it somewhere in the help of rclone while debugging but can't find it that fast.
This is in the docs so seems to be clear anyway.

Note to run these commands on a running backend then see backend/command in the rc docs.

rclone backend <command> remote:path [opts] <args> [flags]

system · September 5, 2020, 11:53pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.