Files deleted and not moved

What is the problem you are having with rclone?

I am attempting to do a server-side move between two My Drives in different domains. In Source:, I grant access to a file to the impersonation user in Target:. In the Target: config, I enable shared_with_me so I can see the shared file. I do the move, and the file is deleted in Source: but not created in Target:. The end result is that the file is simply gone.

What is your rclone version (output from rclone version)

rclone v1.53.3

  • os/arch: windows/amd64
  • go version: go1.15.5

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows 10

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone move Source: Target: --drive-server-side-across-configs --drive-root-folder-id "root" -vv

The rclone config contents with secrets removed.

[Source]
type = drive
scope = drive
service_account_file = C:\Users\Chad\.config\rclone\source.json

[Target]
type = drive
scope = drive
service_account_file = C:\Users\Chad\.config\rclone\target.json
shared_with_me = true

A log from the command with the -vv flag

λ rclone move Source: Target: --drive-server-side-across-configs --drive-root-folder-id "root" -vv
2021/01/22 10:57:05 DEBUG : rclone: Version "v1.53.3" starting with parameters ["rclone" "move" "Source:" "Target:" "--drive-server-side-across-configs" "--drive-root-folder-id" "root" "-vv"]
2021/01/22 10:57:05 DEBUG : Using config file from "C:\\Users\\Chad\\.config\\rclone\\rclone.conf"
2021/01/22 10:57:05 DEBUG : Creating backend with remote "Source:"
2021/01/22 10:57:05 DEBUG : Creating backend with remote "Target:"
2021/01/22 10:57:06 DEBUG : Doc 1.docx: Size and modification time the same (differ by 0s, within tolerance 1ms)
2021/01/22 10:57:06 DEBUG : Doc 1.docx: Unchanged skipping
2021/01/22 10:57:06 DEBUG : Google drive root '': Waiting for checks to finish
2021/01/22 10:57:06 INFO  : Doc 1.docx: Deleted
2021/01/22 10:57:06 DEBUG : Google drive root '': Waiting for transfers to finish
2021/01/22 10:57:06 INFO  : There was nothing to transfer
2021/01/22 10:57:06 INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 2 / 2, 100%
Deleted:                1
Elapsed time:         1.1s

2021/01/22 10:57:06 DEBUG : 10 go routines active

In the Drive UI, I can see the file is gone in Source: and does not exist in Target:. It is simply gone.

What happened is that here

Rclone decided that the source and destination were the same since they had the same modtime and size.

So no need to copy the file, deleting the source is all that is required.

But in this case Source: and Target: seem to be pointing to the same drive so deleting stuff from the Source: deletes it from the Target:.

So I suppose the question is why did the file show up in the Target: when you have shared_with_me = true.

Can you set up the test again, but before you run the rclone move do an rclone lsf Source: --drive-root-folder-id "root" and rclone lsf Target: --drive-root-folder-id "root"

It is possible that --drive-root-folder-id "root" is confusing things - that shouldn't be necessary should it?

So I suppose the question is why did the file show up in the Target: when you have shared_with_me = true .

Isn't that expected? That should show me the files in Target: that are shared with me (versus files I own). And that's exactly what it shows. In this example, the Source: user has created a Google Doc named Shared and has shared it with the Target: user.

C:\Temp\console
λ rclone lsf Source: --drive-root-folder-id "root"
Shared.docx

C:\Temp\console
λ rclone lsf Target: --drive-root-folder-id "root"

C:\Temp\console
λ rclone lsf Target: --drive-root-folder-id "root" --drive-shared-with-me
Shared.docx

Shouldn't Rclone refuse to perform a Move operation if the Target: remote has --drive-shared-with-me enabled? Because that seems like it will always produce this behavior of simply deleting the file in Source:.

Something definitely needs fixing, yes. I think maybe an additional safety check here - don't delete the source if src fs == dst fs and src ID == dst ID.

The issue is caused by the the fact that somehow rclone "saw" the source item in the target. We don't see it in the listing above though. If rclone hadn't seen the target item then it would have done the move as requested.

I tried to replicate this, but I haven't managed to - rclone doesn't see the file in Target:

Is it possible you had it set up differently for your first test?

What do you mean when you say that we don't see it in the listing above? I believe that listing does show Target: seeing the file in question, but now I'm wondering if I'm misunderstanding what it means to see the file. Here's that listing again, with some comments:

Showing the file in Source

C:\Temp\console
λ rclone lsf Source: --drive-root-folder-id "root"
Shared.docx

Not showing the file in Target because we're looking at files owned by Target

C:\Temp\console
λ rclone lsf Target: --drive-root-folder-id "root"

Showing the file in Target because we're now looking at files shared with Target

C:\Temp\console
λ rclone lsf Target: --drive-root-folder-id "root" --drive-shared-with-me
Shared.docx

I was expecting this to act the same as adding the --drive-shared-with-me flag as you have shared_with_me = true in your config

If you use the --drive-shared-with-me flag on an rclone sync/copy/move then it applies to both the source and the destination which will cause the problem above.

I think maybe what you want is a config like this

[TargetShared]
type = drive
scope = drive
service_account_file = C:\Users\Chad\.config\rclone\target.json
shared_with_me = true

[Target]
type = drive
scope = drive
service_account_file = C:\Users\Chad\.config\rclone\target.json

And do your copy from TargetShared to Target?

Ah. I did remove that at some point after starting this thread because I was wanting to illustrate what rclone sees with and without the --drive-shared-with-me option, using lsf.

Would it? In that case, I should see files shared with Source: (Impersonation User 1) and files shared with Target: (Impersonation User 2). Those should be two different sets of files.

Conceptually, I think your idea of Target: and TargetShared: should work, but I think it's hard for a new user to arrive at that configuration straightaway. It seems too easy to have a configuration that will result in just deleting data without moving it anywhere.

Can you try this to see if it fixes the data loss issue?

v1.54.0-beta.5097.797825d46.fix-move-check on branch fix-move-check (uploaded in 15-30 mins)

I'm getting a 404 at that link.

Sorry, I messed up there..

Try this

v1.54.0-beta.5103.3d32b1929.fix-move-check on branch fix-move-check (uploaded in 15-30 mins)

The file was still deleted for me.

λ rclone_fix.exe move Source: Target: --drive-server-side-across-configs -vv
2021/01/27 11:33:48 DEBUG : rclone: Version "v1.54.0-beta.5103.3d32b1929.fix-move-check" starting with parameters ["rclone_fix.exe" "move" "Source:" "Target:" "--drive-server-side-across-configs" "-vv"]
2021/01/27 11:33:48 DEBUG : Using config file from "C:\\Users\\Chad\\.config\\rclone\\rclone.conf"
2021/01/27 11:33:48 DEBUG : Creating backend with remote "Source:"
2021/01/27 11:33:48 DEBUG : Google drive root '': root_folder_id = "0ANK1xd3sAx84Uk9PVA" - save this in the config to speed up startup
2021/01/27 11:33:48 DEBUG : Creating backend with remote "Target:"
2021/01/27 11:33:49 DEBUG : Google drive root '': root_folder_id = "0ALcRX0-WmpNVUk9PVA" - save this in the config to speed up startup
2021/01/27 11:33:49 DEBUG : fixtest1.txt: Size and modification time the same (differ by 0s, within tolerance 1ms)
2021/01/27 11:33:49 DEBUG : fixtest1.txt: Unchanged skipping
2021/01/27 11:33:49 INFO  : fixtest1.txt: Deleted
2021/01/27 11:33:49 DEBUG : Google drive root '': Waiting for checks to finish
2021/01/27 11:33:49 DEBUG : Google drive root '': Waiting for transfers to finish
2021/01/27 11:33:49 INFO  : There was nothing to transfer
2021/01/27 11:33:49 INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 2 / 2, 100%
Deleted:                1 (files), 0 (dirs)
Elapsed time:         1.4s

2021/01/27 11:33:49 DEBUG : 12 go routines active

Foo, yes I see where my approach went wrong.

Try this

v1.54.0-beta.5103.d3f6212e5.fix-move-check on branch fix-move-check (uploaded in 15-30 mins)

This will emit an NOTICE level log when it doesn't delete a file

That worked great. Thanks!

λ rclone_fix.exe move Source: Target: --drive-server-side-across-configs -vv
2021/01/27 12:58:32 DEBUG : rclone: Version "v1.54.0-beta.5103.d3f6212e5.fix-move-check" starting with parameters ["rclone_fix.exe" "move" "Source:" "Target:" "--drive-server-side-across-configs" "-vv"]
2021/01/27 12:58:32 DEBUG : Using config file from "C:\\Users\\Chad\\.config\\rclone\\rclone.conf"
2021/01/27 12:58:32 DEBUG : Creating backend with remote "Source:"
2021/01/27 12:58:33 DEBUG : Google drive root '': root_folder_id = "0ANK1xd3sAx84Uk9PVA" - save this in the config to speed up startup
2021/01/27 12:58:33 DEBUG : Creating backend with remote "Target:"
2021/01/27 12:58:33 DEBUG : Google drive root '': root_folder_id = "0ALcRX0-WmpNVUk9PVA" - save this in the config to speed up startup
2021/01/27 12:58:33 DEBUG : fixtest2.txt: Size and modification time the same (differ by 0s, within tolerance 1ms)
2021/01/27 12:58:33 DEBUG : fixtest2.txt: Unchanged skipping
2021/01/27 12:58:33 NOTICE: fixtest2.txt: Not removing source file as it is the same file as the destination
2021/01/27 12:58:34 DEBUG : Google drive root '': Waiting for checks to finish
2021/01/27 12:58:34 DEBUG : Google drive root '': Waiting for transfers to finish
2021/01/27 12:58:34 INFO  : There was nothing to transfer
2021/01/27 12:58:34 INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         1.6s

2021/01/27 12:58:34 DEBUG : 10 go routines active

Thank you for testing.

I've merged this fix to master now which means it will be in the latest beta in 15-30 mins and released in v1.50

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.