Rclone copy overwrites existing files

kaplannn · May 22, 2019, 5:44pm

Hey guys,

Im moving big files (60GB) from MD to my TD. Somehow rclone copy will NOT ignore existing files and continue to copy the same files over and over.

I use encryption, MD and TD they have different encryption keys.

My Command:

rclone copy -vv --ignore-existing --tpslimit 7 -c --checkers=20 --transfers=5 --drive-chunk-size 256M --fast-list --max-transfer 650G --stats 5s --drive-service-account-file=SA.json --log-file=/opt/scripts/MDtoTD.log --drive-shared-with-me --disable copy $SOURCE $DESTINATION

Files are the same on source and destination:

total 13739437
-rw-rw-r-- 1 plex plex 14069182988 Dec 27 2017 Annabelle 2 - 2017 - HDTV-2160p.mkv

rclone v1.47.0-074-g81fad0f0-beta

os/arch: linux/amd64
go version: go1.12.5

Any ideas?

Thanks

calisro · May 22, 2019, 6:05pm

Not without a log file... Can you create a debug (-vv) log file? Also you may want to run rclone dedupe against your dest.

kaplannn · May 22, 2019, 6:11pm

https://pastebin.com/STidbekj

So there are no entries about skipping or errors.

I have no dupes in this library

Cheers

calisro · May 22, 2019, 6:19pm

Hm. Can you try

Rclone check $SOURCE $DESTINATION -vv

For that same directory/files

Btw, if you're duplicating your crypt to a different drive, you may want to copy the base crypt rather than within the crypt so that it can use hashes and the nonce doesn't change making syncs harder later....

kaplannn · May 22, 2019, 6:32pm

2019/05/22 20:30:30 DEBUG : rclone: Version "v1.47.0-074-g81fad0f0-beta" starting with parameters ["rclone" "check" "mdcrypt:4k/Gladiator" "tdm4kcrypt:Gladiator" "-vv"]

2019/05/22 20:30:30 DEBUG : Using config file from "/home/plex/.config/rclone/rclone.conf"

2019/05/22 20:30:33 INFO : Encrypted drive 'tdm4kcrypt:Gladiator': Waiting for checks to finish

2019/05/22 20:30:33 ERROR : Gladiator - 2000 - Remux-2160p.mkv: File not in Encrypted drive 'tdm4kcrypt:Gladiator'

2019/05/22 20:30:33 NOTICE: Encrypted drive 'tdm4kcrypt:Gladiator': 1 files missing

2019/05/22 20:30:33 NOTICE: Encrypted drive 'tdm4kcrypt:Gladiator': 1 differences found

2019/05/22 20:30:33 Failed to check with 2 errors: last error was: 1 differences found

Hmm is there are problem cause 2 different crypts?

kaplannn · May 22, 2019, 6:40pm

~/.config/rclone$ rclone ls mdcrypt:4k/Gladiator

65979051567 Gladiator - 2000 - Remux-2160p.mkv

~/.config/rclone$ rclone ls tdm4kcrypt:Gladiator

~/.config/rclone$ ll /home/plex/tdm4k/Gladiator

total 64432668

-rw-rw-r-- 1 plex plex 65979051567 May 9 2018 Gladiator - 2000 - Remux-2160p.mkv

Hmm interesting so rclone doesnt seem to list it but the file is actually there

If i do:
„~/.config/rclone$ rclone ls tdm4kcrypt:“

It lists everything, including „Gladiator“

kaplannn · May 22, 2019, 6:51pm

Ok things are weird:

There are dupes if i do rclone ls:

But not in the rclone mount if i check

Animosity022 · May 22, 2019, 6:56pm

The mount would flag an error I believe. You should dedupe that

kaplannn · May 22, 2019, 6:58pm

Doing this. Still, why is it doing this? Maybe cause of the „disable copy“?

calisro · May 22, 2019, 7:14pm

I don't think it's the disable copy but maybe a combo of having a dup and using ignore-existing. I use disable copy at the moment because of that server side move bug.

Best thing to do right now is dedupe and then sync again.

kaplannn · May 22, 2019, 7:30pm

I mean i didnt have dupes before i run this command
Trying it without the strings. Will report back. Thanks for the help

calisro · May 22, 2019, 7:34pm

You may have had duplicates. You wouldn't have necessarily "seen" them unless you ran a dedupe or checked debug logs that may have reported it.

kaplannn · May 22, 2019, 9:12pm

Its still not working. Everytime i run this command, it creates a dupe. I also tried it without —ignore-existing and —disable copy.

I dont know what else i could do. I mean rclone copy works fine if i upload from sonarr/radarr direct to the td.

So either its because of My Drive vs Team Drive or Crypt1 vs Crypt2
Or something with the command

calisro · May 22, 2019, 11:06pm

The one thing about team drives is... There is a delay from when you upload to it till when they are 'visable'. So if you're uploading a file and not giving it time to 'appear' before you run it a second time, I'll bet it will create a duplicate.

It isn't crypt to crypt if you're copying them at the level you are. My comment above about crypt was about the nonce. If you upload to each one at the decrypted layer then you can never sync them at the lower encrypted level because the nonce will be different. This isn't you're case right now.

To help debug, i'd dedupe. Then run a copy with a debug log. Then wait 5 minutes. Then run a copy again with a debug. if you get a duplicate, post both logs.

kaplannn · May 23, 2019, 1:36am

Thanks. So i deleted all dupes and started again with one file. I did it 2 times with a paus of 10mins inbetween:

Log 1:https://pastebin.com/bgf6Nwzs
Log 2:https://pastebin.com/qEn9tWzc

So weird, no errors or nothing. Just copies it again

Thanks for your help

EDIT: here my rclone.conf entries:

[tdm4kcrypt]

type = crypt
remote = tdm4k:
filename_encryption = standard
directory_name_encryption = true
password = Blabla
password2 = Blabla

[tdm4k]

type = drive
scope = drive
service_account_file = /sa/sa.json
team_drive = Blabla

[mdcrypt]

type = crypt
remote = md:Dataacd
filename_encryption = standard
password = Blabla
password2 = Blabla

[md]

type = drive
client_id = blabla
client_secret = blabla
scope = drive
token = {"access_token":"blabla“)

So there is also no Access token and client id for the TD. Could this be a problem?

calisro · May 23, 2019, 1:12pm

I'm not sure. Seems like a bug. @ncw any thoughts?

ncw · May 23, 2019, 3:00pm

--drive-shared-with-me is prone to making duplicates because in shared with me mode it can't see the existing directories so it creates new ones.

So I think this is probably this issue

calisro · May 23, 2019, 3:15pm

I just saw this also. in the docs:

https://rclone.org/drive/#rclone-appears-to-be-re-copying-files-it-shouldn-t

Rclone appears to be re-copying files it shouldn’t

The most likely cause of this is the duplicated file issue above - run rclone dedupe and check your logs for duplicate object or directory messages.

This can also be caused by a delay/caching on google drive’s end when comparing directory listings. Specifically with team drives used in combination with --fast-list. Files that were uploaded recently may not appear on the directory list sent to rclone when using --fast-list.

Waiting a moderate period of time between attempts (estimated to be approximately 1 hour) and/or not using --fast-list both seem to be effective in preventing the problem.

maybe try without fast-list. Maybe it will help. Maybe not?

kaplannn · May 23, 2019, 3:22pm

Ahh yeah, you mentioned this would work in 1.48?

If i remove this string, it will throw errors:

2019/05/23 17:18:56 INFO : Encrypted drive 'tdm4kcrypt:': Waiting for checks to finish
2019/05/23 17:18:56 INFO : Encrypted drive 'tdm4kcrypt:': Waiting for transfers to finish
2019/05/23 17:18:56 ERROR : Attempt 1/3 failed with 1 errors and: directory not found
2019/05/23 17:18:57 ERROR : : error reading source directory: directory not found
2019/05/23 17:18:57 INFO : Encrypted drive 'tdm4kcrypt:': Waiting for checks to finish
2019/05/23 17:18:57 INFO : Encrypted drive 'tdm4kcrypt:': Waiting for transfers to finish
2019/05/23 17:18:57 ERROR : Attempt 2/3 failed with 1 errors and: directory not found
2019/05/23 17:18:58 ERROR : : error reading source directory: directory not found
2019/05/23 17:18:58 INFO : Encrypted drive 'tdm4kcrypt:': Waiting for checks to finish
2019/05/23 17:18:58 INFO : Encrypted drive 'tdm4kcrypt:': Waiting for transfers to finish
2019/05/23 17:18:58 ERROR : Attempt 3/3 failed with 1 errors and: directory not found
2019/05/23 17:18:58 Failed to copy: directory not found

So i kinda need this string in order to work. Or is there maybe another way?

Thanks

kaplannn · May 23, 2019, 3:24pm

I tried it without, same behaviour. And yes, no dupes

Also if i wait 8hours, doesnt matter