Local rclone moves data as expected, but remote command does not

Help and Support
1

What is the problem you are having with rclone?

I am running rclone in a container with docker compose. My command is

rcd --rc-web-gui --rc-addr 0.0.0.0:5572 --rc-web-fetch-url=https://api.github.com/repos/rclone/rclone-webui-react/releases/latest --rc-web-gui-update --rc-user=redacted --rc-pass=redacted -vv --checksum --transfers=4 --checkers=4 --contimeout=60s --timeout=300s --retries=3 --low-level-retries=10 --stats=1s --stats-file-name-length=0

I have a google drive remote that is encrypted and a volume mounted into the container at /data. If I exec into the container run

rclone sync --progress /data googleDrive-encrypted-restic:/ --transfers=10 --checkers=10

I can see all the directories and files contained in /data in google drive.

I would like to trigger this command remotely. The command I've come up with to do this is.

rclone rc sync/copy --progress srcFs="/data" dstFs="googleDrive-encrypted-restic:/" --transfers="10" --checkers="10" --rc-user="redacted" --rc-pass="redacted" --rc-addr="http://10.10.10.71:5572" -vv _async=true

I have tried a number of iterations of this command, and when run it produces an output which makes me believe that all the directories and files are being transferred, however the only file that gets transferred is the sole file at the root of /data

/data is a restic snapshot repository. So the data structure looks like this.

drwxr-xr-x    7 nobody   nobody           8 Feb 19 05:07 .
drwxr-xr-x    1 root     root          4096 Feb 19 07:41 ..
-r--------    1 nobody   nobody         155 Feb 19 05:07 config
drwx------  258 nobody   nobody         258 Feb 19 05:07 data
drwx------    2 nobody   nobody           6 Feb 20 17:00 index
drwx------    2 nobody   nobody           3 Feb 19 05:07 keys
drwx------    2 nobody   nobody           2 Feb 20 17:00 locks
drwx------    2 nobody   nobody           6 Feb 20 17:00 snapshots

Note config is the sole file at the root of /data. None of the directories or their contents get moved.

Run the command 'rclone version' and share the full output of the command.

/data # rclone version
rclone v1.65.2
- os/version: alpine 3.19.0 (64 bit)
- os/kernel: 6.5.0-17-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.21.6
- go/linking: static
- go/tags: none

At the time of this post, rclone is the latest linux version available.

Which cloud storage system are you using? (eg Google Drive)

Google Drive and Local File System

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone rc sync/copy --progress srcFs="/data" dstFs="googleDrive-encrypted-restic:/" --transfers="10" --checkers="10" --rc-user="redacted" --rc-pass="redacted" --rc-addr="http://10.10.10.71:5572" -vv _async=true

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

/data # rclone config redacted
[gdrive-work]
type = drive
client_id = XXX
client_secret = XXX
scope = drive
token = XXX
team_drive = 

[googleDrive]
type = drive
client_id = XXX
client_secret = XXX
scope = drive
token = XXX
team_drive = 

[googleDrive-encrypted]
type = crypt
remote = googleDrive:/rclone
password = XXX
password2 = XXX

[googleDrive-encrypted-restic]
type = crypt
remote = googleDrive:/rclone/restic
password = XXX
password2 = XXX

[local-container]
type = local
### Double check the config for sensitive info before posting publicly

A log from the command that you were trying to run with the -vv flag

2024/02/20 17:27:39 DEBUG : rclone: Version "v1.65.2" starting with parameters ["rclone" "rc" "sync/copy" "--prog
ress" "srcFs=/data" "dstFs=googleDrive-encrypted-restic:/" "--transfers=10" "--checkers=10" "--rc-user=redacted
" "--rc-pass=redacted" "--rc-addr=http://10.10.10.71:5572" "-vv"]                                
Transferred:              0 B / 0 B, -, 0 B/s, ETA -                                                             
Transferred:              0 B / 0 B, -, 0 B/s, ETA -                                                             
Elapsed time:        15.4s                                                                                       
2024/02/20 17:27:54 DEBUG : 7 go routines active

Also of note, that the two commands take roughly the same amount of time. In the remote command the first file is added to google drive nearly instantly, then continues to run for about 14-15 seconds, finishes without error, and exits. This is consistent with running the command locally, for what that's worth.

2

If you include _async=true, it is expected that the command will return right away while the actual sync job works invisibly in the background. The --progress that you included in this case does not show the progress of the async job, but rather the progress of the client sending the request. To see the progress of the async job, you can use job/status.

That said, despite seeing _async=true in your command, I don't actually see it in your debug log. Maybe this was from a different run where you omitted it?

What happens if you run rclone rc core/stats right after the sync/copy command? Do the stats match what you'd expect?

Have you verified that the other directories contain files which differ from the destination? Rclone only transfers files if it detects that they are different (so, running the same sync command twice should usually produce nothing the second time, unless the source has changed in between.) Also, sync does not copy empty directories by default -- you would need to pass createEmptySrcDirs=true if you want it to copy them.

Another thing to check: what happens if you operations/list these remotes through the rc? Does it find both remotes and list the expected contents?

3

Sorry I forgot to remove the _async=true from the command in my initial post. The output for the remote command is posted above at the bottom of the inital post and you can see that _async is not a part of that output.

The directories do contain content at the source, and the commands are being run after everything has been removed from the destination. So the destination is empty as far as I can see. Running from inside the container does bring the directories over and I can delete them from the destination and rerun it and get the same results.

There might be some kind of weird cache issue or something going on. It seems that even though I deleted these files after initially syncing them from within the container. Google or rclone may be smart enough to realize that the file still exists in the trash. So i deleted all of the trash. Now I am getting a new error when running the remote sync.

❯ rclone rc sync/copy --progress srcFs="/data" dstFs="googleDrive-encrypted-restic:/" --transfe
rs="10" --checkers="10" --rc-user="redacted" --rc-pass="redacted" --rc-addr="
http://10.10.10.71:5572" -vv                                                                   
2024/02/20 19:40:47 DEBUG : rclone: Version "v1.65.2" starting with parameters ["rclone" "rc" "
sync/copy" "--progress" "srcFs=/data" "dstFs=googleDrive-encrypted-restic:/" "--transfers=10" "
--checkers=10" "--rc-user=redacted" "--rc-pass=redacted" "--rc-addr=http://10
.10.10.71:5572" "-vv"]                                                                         
Transferred:              0 B / 0 B, -, 0 B/s, ETA -                                           
Elapsed time:         2.5s{                                                                    
        "error": "googleapi: Error 404: File not found: 1qyoIWcQAPr9halglfxuHszX0ElW9tRWa., not
Found",                                                                                        
        "input": {                                                                             
                "dstFs": "googleDrive-encrypted-restic:/",                                     
                "srcFs": "/data"                                                               
        },                                                                                     
        "path": "sync/copy",                                                                   
        "status": 500                                                                          
Transferred:              0 B / 0 B, -, 0 B/s, ETA -                                           
Errors:                 1 (retrying may help)                                                  
Elapsed time:         2.7s                                                                     
2024/02/20 19:40:49 DEBUG : 7 go routines active                                               
2024/02/20 19:40:49 Failed to rc: operation "sync/copy" failed: googleapi: Error 404: File not 
found: 1qyoIWcQAPr9halglfxuHszX0ElW9tRWa., notFound
4

There seems to be an issue, for me, with this config. It seems to work as expected if the remote attribute in my config remote = googleDrive:/restic is in the root path eg: mydrive:/restic, but doesn't work if I try to configure a sub folder like remote = googleDrive:/rclone/restic.

Non Working Config

[googleDrive-encrypted-restic]
type = crypt
remote = googleDrive:/rclone/restic
password = xxx
password2 = xxx

Working Conifg

[googleDrive-encrypted-restic]
type = crypt
remote = googleDrive:/restic
password = xxx
password2 = xxx
5

I've definitely noticed this too with Google Drive. It is sometimes aware of trashed files when it shouldn't be, and I haven't yet pinned down an exact sequence to reliably reproduce it (any help would be welcome!)

You may need to run fscache/clear after changing the config. Does it behave differently if you do that?

Also, while this shouldn't really matter, it would be worth trying without the leading slash (eg. googleDrive:rclone/restic). Rclone usually prefers no slash (but in most cases it will strip the slash itself, so the result should be the same...)

Also, double-check that the remote param is the encrypted (not decrypted) name. In your first post, you had remote = googleDrive:/rclone, meaning that anything under rclone should look like scrambled nonsense to Google... and therefore googleDrive:/rclone/restic looks wrong (unless you reset stuff since then).

6

I worded that confusingly...to be clear: what I mean is that the remote param should be the name that Google would see, not the name that only you can see through crypt.

7

I apologize, for being a bad troubleshooter, but I combined your advice regarding the removal of the beginning slash in my config and fscache/clear into a single step and it does work as expected now.

I will play around with it some more to see if I can reliably reproduce the google trash behavior. What i can say initially, is that the files were in the trash, after the first local command I ran, because I wanted to test rclone and reliability myself, but running subsequent local commands from the container never resulted in the odd trash behavior. This is what ultimately caused my confusion and the opening of this topic. Only when I tried to run the command remotely did the trash issue occur. Maybe that might help narrow down the scope of the issue.

8

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.