Remote command sync/sync syncs new file into trashed folder on google drive

Suspected Bug
1

What is the problem you are having with rclone?

Sync with remote command seems to sync into trashed folders when re-uploading files in the same folder again.

The strange thing is ... I could not reproduce it without using remote commands.

I started with a rclone in docker with webgui and everything and also some drive tweaking options. But I could reduce it to a two shell (one server, one client) step by step with minimal parameters and a fresh downloaded 1.57.0 used directly from the working directory.

Run the command 'rclone version' and share the full output of the command.

rclone v1.57.0
- os/version: slackware 15.0 (64 bit)
- os/kernel: 5.14.15-Unraid (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.17.2
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

  1. prepare local folder:

    mkdir -p /tmp/googletest/testfolder

  2. create new shared (team) drive in google or delete everything in an existing and empty the trash afterwards.

  3. starting server in a separate shell:

    ./rclone -vv rcd --rc-user=test --rc-pass=test --config=./rclone.conf --rc-addr 127.0.0.1:1234

  4. sync

    ./rclone rc --user test --pass test --rc-addr 127.0.0.1:1234 sync/sync createEmptySrcDirs=true srcFs=/tmp/googletest dstFs=googledrive_test:/

  5. verify folder created testfolder on drive.google.com

  6. delete (move to trash) testfolder on drive.google.com

  7. create new local testfile in testfolder

    touch /tmp/googletest/testfolder/testfile

  8. sync again

    ./rclone rc --user test --pass test --rc-addr 127.0.0.1:1234 sync/sync createEmptySrcDirs=true srcFs=/tmp/googletest dstFs=googledrive_test:/

  9. check drive on drive.google.com:

    • file shows as recently uploaded, but not visible
    • restoring testfolder and the file testfile is in the restored testfolder

The rclone config contents with secrets removed.

[googledrive_test]
type = drive
client_id = <myclientid>
client_secret = <myclientsecret>
token = {"access_token":"<myaccesstoken>","token_type":"Bearer","refresh_token":"<myrefreshtoken>","expiry":"2022-01-21T17:54:08.681084866+01:00"}
team_drive = <myteamdriveid>
root_folder_id =

A log from the command with the -vv flag

output of the server from the separate shell:

2022/01/21 22:56:49 DEBUG : rclone: Version "v1.57.0" starting with parameters ["./rclone" "-vv" "rcd" "--rc-user=test" "--rc-pass=test" "--config=./rclone.conf" "--rc-addr" "127.0.0.1:1234"]
2022/01/21 22:56:49 INFO  : Using --user test --pass XXXX as authenticated user
2022/01/21 22:56:49 NOTICE: Serving remote control on http://127.0.0.1:1234/
2022/01/21 22:58:25 DEBUG : rc: "sync/sync": with parameters map[createEmptySrcDirs:true dstFs:googledrive_test:/ srcFs:/tmp/googletest]
2022/01/21 22:58:25 DEBUG : Creating backend with remote "/tmp/googletest"
2022/01/21 22:58:25 DEBUG : Using config file from "/root/rclone.conf"
2022/01/21 22:58:25 DEBUG : Creating backend with remote "googledrive_test:/"
2022/01/21 22:58:25 DEBUG : fs cache: renaming cache item "googledrive_test:/" to be canonical "googledrive_test:"
2022/01/21 22:58:25 DEBUG : Google drive root '': read info from Shared Drive "test"
2022/01/21 22:58:25 DEBUG : Google drive root '': Waiting for checks to finish
2022/01/21 22:58:25 DEBUG : Google drive root '': Waiting for transfers to finish
2022/01/21 22:58:25 DEBUG : testfolder: Making directory
2022/01/21 22:58:26 DEBUG : Google drive root '': copied 1 directories
2022/01/21 22:58:26 DEBUG : Waiting for deletions to finish
2022/01/21 22:58:26 INFO  : There was nothing to transfer
2022/01/21 22:58:26 DEBUG : rc: "sync/sync": reply map[]: <nil>
2022/01/21 22:58:53 DEBUG : rc: "sync/sync": with parameters map[createEmptySrcDirs:true dstFs:googledrive_test:/ srcFs:/tmp/googletest]
2022/01/21 22:58:53 DEBUG : fs cache: switching user supplied name "googledrive_test:/" for canonical name "googledrive_test:"
2022/01/21 22:58:53 DEBUG : Google drive root '': read info from Shared Drive "test"
2022/01/21 22:58:53 DEBUG : Google drive root '': Waiting for checks to finish
2022/01/21 22:58:53 DEBUG : Google drive root '': Waiting for transfers to finish
2022/01/21 22:58:55 DEBUG : testfolder/testfile: md5 = d41d8cd98f00b204e9800998ecf8427e OK
2022/01/21 22:58:55 INFO  : testfolder/testfile: Copied (new)
2022/01/21 22:58:55 DEBUG : Waiting for deletions to finish
2022/01/21 22:58:55 DEBUG : rc: "sync/sync": reply map[]: <nil>
2

hello and welcome to the forum,

but in the log snippet, no files were deleted?

and not following the sequence,
step 6, you manually delete the testfolder or are you saying rclone deletes the folder?

step 9. file shows as recently uploaded, but not visible
what does that mean; how can the file uploaded but not visible? where is it visible and where is it not visible?

3

step 6:
I delete respectively moved the folder to trash manually on drive.google.com page. so the drive looks empty but the folder is visible in the trash.

step 9: on drive.google.com, you have a "Activities" sidebar, where you see recent file changes. Sometimes there is also suggested files (mostly also new uploaded ones). But the "main" view of the drive is empty. If I click on the file link in the "Activities" it shows a popup, that the file was recently moved to trash.

As I know a bit the Google Drive API, you call for all files and folders in a flat view and have to structure them by yourself into a folder hierarchy with the metadata. You can get a folder independently where it is or was. So my guess would be, that the folder was found from rclone, but not checked that the folder was in the trash. So the files was uploaded to this trashed folder instead of creating a new folder and put the file in there. This als explains why the file magically appear on restoring the folder.

I don't know the internals of rclone, but I don't understand why this should behave differently for remote commands. Because without remote commands I could not reproduce it. The testfolder was newly created (even there was the old one in the trash) and the file was "attached" into this new folder.
Due I strongly assume backend code is the same for remote and "local" commands.... I cant get it into my head, that I get two different results.

4

i will admit i am not a gdrive and remote command expert, maybe there is some specific knowledge i am lacking.

nevertheless, imho, if you want to claim a rclone bug, need to have hard documented proof.
if this is a bug, we need to try to find it so the rclone developers can fix it.

for example,
--- Waiting for deletions to finish, even if the dest is empty and therefore no files to delete, rclone sync will always print that.
--- in the log snippet, nothing was deleted.
--- the trash folder, nothing to prove what was in the trash, then rclone did something, then dirs/files ended up in the trash?
use --drive-trashed-only to show the state of the trash before and after rclone sync is run.
--- the commands that were not executed remotely, they were not posted, nor was the full rclone debug log.

5

Good point, I will edit that away, misinterpretation from my side, so no value on this bug report.

That's a question of interpretation. But technically right. At least until googles auto trash delete occurs after 30 days. Till then the file is uploaded and can be restored, only with the whole folder from the trash. And also only if known that it is there.

Nice hint, didn't know this flag and worse... didn't thought about using rclone to show the state on google drive.

Same as above... I could have used rclone for the remote actions on google.

I really appreciate your feedback. Will take the time to update the post with that. Just not sure if I manage it today (already late night here), but let you know when I updated the post.

6

quick comment, please do not re-edit old posts, ok, get very confusing very quickly.
just let it be, no problem.

7

sure, just trying to be a good rclone bug hunter :upside_down_face:

here, that is what matters. the narrowest test case that will prove your bug claim.

8

What is the problem you are having with rclone?

Sync with remote command seems to sync into trashed folders when re-uploading files in the same folder again.

The strange thing is ... I could not reproduce it without using remote commands.

Run the command 'rclone version' and share the full output of the command.

rclone v1.57.0
- os/version: slackware 15.0 (64 bit)
- os/kernel: 5.14.15-Unraid (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.17.2
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

  1. prepare local folder:

    rm -rf /tmp/googletest
mkdir -p /tmp/googletest/testfolder

  2. create new shared (team) drive in google or delete everything in an existing and empty the trash afterwards.
    Here everything is empty on the remote:

    ./rclone --config=./rclone.conf -vv lsf -R googledrive_test:/
2022/01/22 10:34:55 DEBUG : rclone: Version "v1.57.0" starting with parameters ["./rclone" "--config=./rclone.conf" "-vv" "lsf" "-R" "googledrive_test:/"]
2022/01/22 10:34:55 DEBUG : Creating backend with remote "googledrive_test:/"
2022/01/22 10:34:55 DEBUG : Using config file from "/root/rclone.conf"
2022/01/22 10:34:55 DEBUG : fs cache: renaming cache item "googledrive_test:/" to be canonical "googledrive_test:"
2022/01/22 10:34:55 DEBUG : Google drive root '': read info from Shared Drive "test"
2022/01/22 10:34:55 DEBUG : 4 go routines active

./rclone --config=./rclone.conf -vv lsf -R googledrive_test:/ --drive-trashed-only=true
2022/01/22 10:35:09 DEBUG : rclone: Version "v1.57.0" starting with parameters ["./rclone" "--config=./rclone.conf" "-vv" "lsf" "-R" "googledrive_test:/" "--drive-trashed-only=true"]
2022/01/22 10:35:09 DEBUG : Creating backend with remote "googledrive_test:/"
2022/01/22 10:35:09 DEBUG : Using config file from "/root/rclone.conf"
2022/01/22 10:35:09 DEBUG : googledrive_test: detected overridden config - adding "{txBp-}" suffix to name
2022/01/22 10:35:09 DEBUG : fs cache: renaming cache item "googledrive_test:/" to be canonical "googledrive_test{txBp-}:"
2022/01/22 10:35:10 DEBUG : Google drive root '': read info from Shared Drive "test"
2022/01/22 10:35:10 DEBUG : 4 go routines active

  3. starting server in a separate shell:

    ./rclone -vv rcd --rc-user=test --rc-pass=test --config=./rclone.conf --rc-addr 127.0.0.1:1234

  4. sync

    ./rclone rc --user test --pass test --rc-addr 127.0.0.1:1234 sync/sync createEmptySrcDirs=true srcFs=/tmp/googletest dstFs=googledrive_test:/
{}

  5. verify folder created testfolder on googledrive_test:

    ./rclone --config=./rclone.conf -vv lsf -R googledrive_test:/
2022/01/22 10:36:02 DEBUG : rclone: Version "v1.57.0" starting with parameters ["./rclone" "--config=./rclone.conf" "-vv" "lsf" "-R" "googledrive_test:/"]
2022/01/22 10:36:02 DEBUG : Creating backend with remote "googledrive_test:/"
2022/01/22 10:36:02 DEBUG : Using config file from "/root/rclone.conf"
2022/01/22 10:36:02 DEBUG : fs cache: renaming cache item "googledrive_test:/" to be canonical "googledrive_test:"
testfolder/
2022/01/22 10:36:02 DEBUG : 4 go routines active

  6. move testfolder on remote to trash

    ./rclone --config=./rclone.conf -vv rmdir googledrive_test:/testfolder --drive-use-trash=true
2022/01/22 10:36:21 DEBUG : rclone: Version "v1.57.0" starting with parameters ["./rclone" "--config=./rclone.conf" "-vv" "rmdir" "googledrive_test:/testfolder" "--drive-use-trash=true"]
2022/01/22 10:36:21 DEBUG : Creating backend with remote "googledrive_test:/testfolder"
2022/01/22 10:36:21 DEBUG : Using config file from "/root/rclone.conf"
2022/01/22 10:36:21 DEBUG : googledrive_test: detected overridden config - adding "{6Cm8i}" suffix to name
2022/01/22 10:36:22 DEBUG : fs cache: renaming cache item "googledrive_test:/testfolder" to be canonical "googledrive_test{6Cm8i}:testfolder"
2022/01/22 10:36:22 INFO  : Google drive root 'testfolder': Removing directory
2022/01/22 10:36:23 DEBUG : 4 go routines active

  7. verify testfolder is in trash

    ./rclone --config=./rclone.conf -vv lsf -R googledrive_test:/ --drive-trashed-only=true
2022/01/22 10:36:36 DEBUG : rclone: Version "v1.57.0" starting with parameters ["./rclone" "--config=./rclone.conf" "-vv" "lsf" "-R" "googledrive_test:/" "--drive-trashed-only=true"]
2022/01/22 10:36:36 DEBUG : Creating backend with remote "googledrive_test:/"
2022/01/22 10:36:36 DEBUG : Using config file from "/root/rclone.conf"
2022/01/22 10:36:36 DEBUG : googledrive_test: detected overridden config - adding "{txBp-}" suffix to name
2022/01/22 10:36:36 DEBUG : fs cache: renaming cache item "googledrive_test:/" to be canonical "googledrive_test{txBp-}:"
testfolder/
2022/01/22 10:36:37 DEBUG : 4 go routines active

  8. verify empty drive (without trash)

    ./rclone --config=./rclone.conf -vv lsf -R googledrive_test:/
2022/01/22 10:36:51 DEBUG : rclone: Version "v1.57.0" starting with parameters ["./rclone" "--config=./rclone.conf" "-vv" "lsf" "-R" "googledrive_test:/"]
2022/01/22 10:36:51 DEBUG : Creating backend with remote "googledrive_test:/"
2022/01/22 10:36:51 DEBUG : Using config file from "/root/rclone.conf"
2022/01/22 10:36:51 DEBUG : fs cache: renaming cache item "googledrive_test:/" to be canonical "googledrive_test:"
2022/01/22 10:36:51 DEBUG : Google drive root '': read info from Shared Drive "test"
2022/01/22 10:36:51 DEBUG : 4 go routines active

  9. create new local testfile in testfolder

    echo "test" >/tmp/googletest/testfolder/testfile

  10. sync again

    ./rclone rc --user test --pass test --rc-addr 127.0.0.1:1234 sync/sync createEmptySrcDirs=true srcFs=/tmp/googletest dstFs=googledrive_test:/
{}

  11. check content on remote:

    ./rclone --config=./rclone.conf -vv lsf -R googledrive_test:/
2022/01/22 10:37:30 DEBUG : rclone: Version "v1.57.0" starting with parameters ["./rclone" "--config=./rclone.conf" "-vv" "lsf" "-R" "googledrive_test:/"]
2022/01/22 10:37:30 DEBUG : Creating backend with remote "googledrive_test:/"
2022/01/22 10:37:30 DEBUG : Using config file from "/root/rclone.conf"
2022/01/22 10:37:30 DEBUG : fs cache: renaming cache item "googledrive_test:/" to be canonical "googledrive_test:"
2022/01/22 10:37:31 DEBUG : Google drive root '': read info from Shared Drive "test"
2022/01/22 10:37:31 DEBUG : 4 go routines active

./rclone --config=./rclone.conf -vv lsf -R googledrive_test:/ --drive-trashed-only=true
2022/01/22 10:37:43 DEBUG : rclone: Version "v1.57.0" starting with parameters ["./rclone" "--config=./rclone.conf" "-vv" "lsf" "-R" "googledrive_test:/" "--drive-trashed-only=true"]
2022/01/22 10:37:43 DEBUG : Creating backend with remote "googledrive_test:/"
2022/01/22 10:37:43 DEBUG : Using config file from "/root/rclone.conf"
2022/01/22 10:37:43 DEBUG : googledrive_test: detected overridden config - adding "{txBp-}" suffix to name
2022/01/22 10:37:43 DEBUG : fs cache: renaming cache item "googledrive_test:/" to be canonical "googledrive_test{txBp-}:"
testfolder/
2022/01/22 10:37:43 DEBUG : 4 go routines active

  12. untrash and show drive content (file is untrashed with the folder)

    ./rclone --config=./rclone.conf -vv backend untrash googledrive_test:/ --drive-trashed-only=true
2022/01/22 10:41:20 DEBUG : rclone: Version "v1.57.0" starting with parameters ["./rclone" "--config=./rclone.conf" "-vv" "backend" "untrash" "googledrive_test:/" "--drive-trashed-only=true"]
2022/01/22 10:41:20 DEBUG : Using config file from "/root/rclone.conf"
2022/01/22 10:41:20 DEBUG : : finding trash to restore in directory "0AGeunfztK11sUk9PVA"
2022/01/22 10:41:21 INFO  : testfolder: restoring "1WQUC2ToLzFDk4h-Lt1rqc4WAHiWDNqH7"
2022/01/22 10:41:21 DEBUG : testfolder: finding trash to restore in directory "1WQUC2ToLzFDk4h-Lt1rqc4WAHiWDNqH7"
{
        "Untrashed": 1,
        "Errors": 0
}
2022/01/22 10:41:22 DEBUG : 4 go routines active

./rclone --config=./rclone.conf -vv lsf -R googledrive_test:/
2022/01/22 10:41:48 DEBUG : rclone: Version "v1.57.0" starting with parameters ["./rclone" "--config=./rclone.conf" "-vv" "lsf" "-R" "googledrive_test:/"]
2022/01/22 10:41:48 DEBUG : Creating backend with remote "googledrive_test:/"
2022/01/22 10:41:48 DEBUG : Using config file from "/root/rclone.conf"
2022/01/22 10:41:48 DEBUG : fs cache: renaming cache item "googledrive_test:/" to be canonical "googledrive_test:"
testfolder/
testfolder/testfile
2022/01/22 10:41:48 DEBUG : 4 go routines active

The rclone config contents with secrets removed.

[googledrive_test]
type = drive
client_id = <myclientid>
client_secret = <myclientsecret>
token = {"access_token":"<myaccesstoken>","token_type":"Bearer","refresh_token":"<myrefreshtoken>","expiry":"2022-01-21T17:54:08.681084866+01:00"}
team_drive = <myteamdriveid>
root_folder_id =

A log from the command with the -vv flag

output of the server from the separate shell:

./rclone -vv rcd --rc-user=test --rc-pass=test --config=./rclone.conf --rc-addr 127.0.0.1:1234
2022/01/22 10:35:25 DEBUG : rclone: Version "v1.57.0" starting with parameters ["./rclone" "-vv" "rcd" "--rc-user=test" "--rc-pass=test" "--config=./rclone.conf" "--rc-addr" "127.0.0.1:1234"]
2022/01/22 10:35:25 INFO  : Using --user test --pass XXXX as authenticated user
2022/01/22 10:35:25 NOTICE: Serving remote control on http://127.0.0.1:1234/
2022/01/22 10:35:38 DEBUG : rc: "sync/sync": with parameters map[createEmptySrcDirs:true dstFs:googledrive_test:/ srcFs:/tmp/googletest]
2022/01/22 10:35:38 DEBUG : Creating backend with remote "/tmp/googletest"
2022/01/22 10:35:38 DEBUG : Using config file from "/root/rclone.conf"
2022/01/22 10:35:38 DEBUG : Creating backend with remote "googledrive_test:/"
2022/01/22 10:35:38 DEBUG : fs cache: renaming cache item "googledrive_test:/" to be canonical "googledrive_test:"
2022/01/22 10:35:38 DEBUG : Google drive root '': read info from Shared Drive "test"
2022/01/22 10:35:38 DEBUG : Google drive root '': Waiting for checks to finish
2022/01/22 10:35:38 DEBUG : Google drive root '': Waiting for transfers to finish
2022/01/22 10:35:38 DEBUG : testfolder: Making directory
2022/01/22 10:35:39 DEBUG : Google drive root '': copied 1 directories
2022/01/22 10:35:39 DEBUG : Waiting for deletions to finish
2022/01/22 10:35:39 INFO  : There was nothing to transfer
2022/01/22 10:35:39 DEBUG : rc: "sync/sync": reply map[]: <nil>
2022/01/22 10:37:13 DEBUG : rc: "sync/sync": with parameters map[createEmptySrcDirs:true dstFs:googledrive_test:/ srcFs:/tmp/googletest]
2022/01/22 10:37:13 DEBUG : fs cache: switching user supplied name "googledrive_test:/" for canonical name "googledrive_test:"
2022/01/22 10:37:14 DEBUG : Google drive root '': read info from Shared Drive "test"
2022/01/22 10:37:14 DEBUG : Google drive root '': Waiting for checks to finish
2022/01/22 10:37:14 DEBUG : Google drive root '': Waiting for transfers to finish
2022/01/22 10:37:15 DEBUG : testfolder/testfile: md5 = d8e8fca2dc0f896fd7cb4cb0031ba249 OK
2022/01/22 10:37:15 INFO  : testfolder/testfile: Copied (new)
2022/01/22 10:37:15 DEBUG : Waiting for deletions to finish
2022/01/22 10:37:15 DEBUG : rc: "sync/sync": reply map[]: <nil>
9

here a new version.
I reproduced it with only rclone commands including showing the result at the end.

10

i did not do a deep dive into that yet.

might be an issue with shared drives, perhaps test without using that.

11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.