Rclone move will automatically dedupe if file with same name is found in destination?

kacaukacau · November 26, 2020, 4:54pm

What is the problem you are having with rclone?

With rclone move, if a file in the destination has the same name as a file in source, one of the file will appear in trash after the command.

What is your rclone version (output from `rclone version`)

rclone v1.53.0-beta.4731.725ae9138
- os/arch: linux/amd64
- go version: go1.15

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu 20.04 64-bit

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone move -P gdrive:unencrypted/recycle-bin/backups/pc01 gdrive:unencrypted/backups/pc01

The rclone config contents with secrets removed.

[gdrive]
type = drive
client_id = xxxxxxxxxx
client_secret = xxxxxxxxxx
scope = drive
root_folder_id = xxxxxxxxxx
token = xxxxxxxxx
team_drive = xxxxxxxxxx
chunk_size = 16M
server_side_across_configs = true

It seems like one of the file will be deleted, sometimes file in destination sometimes file from source. They have the same name but different size and hash.

Animosity022 · November 26, 2020, 5:46pm

Do you have any debug logs?

move copies from the source to destination so it would get deleted from the source all the time.

felix@gemini:~$ rclone move hosts GD: -vv
2020/11/26 12:45:43 DEBUG : rclone: Version "v1.53.3" starting with parameters ["rclone" "move" "hosts" "GD:" "-vv"]
2020/11/26 12:45:43 DEBUG : Creating backend with remote "hosts"
2020/11/26 12:45:43 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2020/11/26 12:45:43 DEBUG : fs cache: adding new entry for parent of "hosts", "/home/felix"
2020/11/26 12:45:43 DEBUG : Creating backend with remote "GD:"
2020/11/26 12:45:44 DEBUG : hosts: Sizes differ (src 224 vs dst 222)
2020/11/26 12:45:45 DEBUG : hosts: MD5 = 7ef0186629f7630906293a6835028ec0 OK
2020/11/26 12:45:45 INFO  : hosts: Copied (replaced existing)
2020/11/26 12:45:45 INFO  : hosts: Deleted
2020/11/26 12:45:45 INFO  :
Transferred:           224 / 224 Bytes, 100%, 296 Bytes/s, ETA 0s
Checks:                 2 / 2, 100%
Deleted:                1
Renamed:                1
Transferred:            1 / 1, 100%
Elapsed time:         1.8s

2020/11/26 12:45:45 DEBUG : 4 go routines active

kacaukacau · November 26, 2020, 5:53pm

$ rclone move -P gdrive:unencrypted/tester/recycle-bin/backups/pc01 gdrive:unencrypted/tester/backups/pc01 -vv
2020/11/26 18:52:39 DEBUG : rclone: Version "v1.53.0-beta.4731.725ae9138" starting with parameters ["rclone" "move" "-P" "gdrive:unencrypted/tester/recycle-bin/backups/pc01" "gdrive:unencrypted/tester/backups/pc01" "-vv"]
2020/11/26 18:52:39 DEBUG : Using config file from "/home/ubuntu/.config/rclone/rclone.conf"
2020/11/26 18:52:39 DEBUG : Creating backend with remote "gdrive:unencrypted/tester/recycle-bin/backups/pc01"
2020/11/26 18:52:40 DEBUG : Creating backend with remote "gdrive:unencrypted/tester/backups/pc01"
2020-11-26 18:52:41 DEBUG : Google drive root 'unencrypted/tester/backups/pc01': Using server side directory move
2020-11-26 18:52:41 INFO  : Google drive root 'unencrypted/tester/backups/pc01': Server side directory move failed - fallback to file moves: can't copy directory - destination already exists
2020-11-26 18:52:41 DEBUG : Google drive root 'unencrypted/tester/backups/pc01': Waiting for checks to finish
2020-11-26 18:52:41 DEBUG : file1: Sizes differ (src 21 vs dst 15)
2020-11-26 18:52:41 DEBUG : Google drive root 'unencrypted/tester/backups/pc01': Waiting for transfers to finish
2020-11-26 18:52:42 INFO  : file1: Deleted
2020-11-26 18:52:43 INFO  : file1: Moved (server side)
2020-11-26 18:52:43 INFO  : There was nothing to transfer
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 3 / 3, 100%
Deleted:                1
Renamed:                1
Elapsed time:         4.3s
2020/11/26 18:52:43 INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 3 / 3, 100%
Deleted:                1
Renamed:                1
Elapsed time:         4.3s

2020/11/26 18:52:43 DEBUG : 6 go routines active

As you can see it says sizes differ, but it deletes one. Since Google Drive supports multiple files with same name in the same folder, shouldn't it be better if rclone just moves it to the destination and leave it there even if there will be two file with the same name?

Animosity022 · November 26, 2020, 5:55pm

So what's the problem there? It looks like it did a server move at the end.

(Didn't misread as you had a ninja edit which makes following things tough)

Animosity022 · November 26, 2020, 5:57pm

Sorry as I misread your question.

Google Drive supports multiple file names.

Rclone does not.

kacaukacau · November 26, 2020, 5:59pm

Didn't misread as you had a ninja edit which makes following things tough

Oops, sorry, I should've just posted another reply instead of ninja-editting

kacaukacau · November 26, 2020, 6:02pm

So essentially it means it's functioning as it's supposed to, which is it'll delete one of the file if the names of two files are identical?

Animosity022 · November 26, 2020, 6:03pm

Rclone works with a plethora of backends and Google Drive is the only one (maybe one more) that supports duplicate file names which no OS really supports either.

You are correct as it's functioning exactly as it is supposed to as if you move a file, it deletes from the source and copies to the destination and in this case replaces the existing file if it's newer.

kacaukacau · November 26, 2020, 6:07pm

I see. It would be great if rclone could support a flag to rename it instead of replacing the file though.

Animosity022 · November 26, 2020, 6:26pm

You could use something like:

   --backup-dir string                    Make backups into hierarchy based in DIR.

or you can look at suffix.

felix@gemini:~$ rclone copy /etc/hosts GD: -vv --suffix .old
2020/11/26 13:24:41 DEBUG : rclone: Version "v1.53.3" starting with parameters ["rclone" "copy" "/etc/hosts" "GD:" "-vv" "--suffix" ".old"]
2020/11/26 13:24:41 DEBUG : Creating backend with remote "/etc/hosts"
2020/11/26 13:24:41 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2020/11/26 13:24:41 DEBUG : fs cache: adding new entry for parent of "/etc/hosts", "/etc"
2020/11/26 13:24:41 DEBUG : Creating backend with remote "GD:"
2020/11/26 13:24:42 DEBUG : hosts: Sizes differ (src 221 vs dst 224)
2020/11/26 13:24:42 INFO  : hosts: Moved (server side)
2020/11/26 13:24:43 DEBUG : hosts: MD5 = 574584fa3f744ea629b956464b687c40 OK
2020/11/26 13:24:43 INFO  : hosts: Copied (new)
2020/11/26 13:24:43 INFO  :
Transferred:           221 / 221 Bytes, 100%, 160 Bytes/s, ETA 0s
Checks:                 1 / 1, 100%
Renamed:                1
Transferred:            1 / 1, 100%
Elapsed time:         2.0s

and it makes a .old of the orig.

felix@gemini:~$ rclone lsl GD:hosts
rclone       221 2020-11-26 13:24:26.390000000 hosts
felix@gemini:~$ rclone lsl GD:hosts.old
      224 2020-11-26 12:45:16.134000000 hosts.old

system · November 29, 2020, 6:26pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.