Duplicate md5sum to match two separate files?

Help and Support
1

What is the problem you are having with rclone?

I have the same file in my unencrypted and encrypted drives. I moved both files to a new directory and renamed both files to match each other in the hopes of not having to reupload the same file to my encrypted drive. Now, whenever I run the copy command, Rclone still tries to copy over the same file. I'm guessing the md5sum changed so it's not registering as the same file?

Is there any way to duplicate the md5sum of the original file to match the file in the encrypted drive so Rclone stops trying to copy over the exact copy?

What is your rclone version (output from rclone version)

v1.57.0

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy gdrive:dir1 gdrivecrypt: --transfers=5 --progress
2

hi, trying to understand your example?

so in that directory, you have a crypted file and a non-crypted file and you are trying to compare them?

would help to see a debug log.

3

When you copy a file to a crypted remote, the md5sum will always change as that's part of the encryption process.

There is no way to get them to ever match or even duplicate in reality.

You can use:

https://rclone.org/commands/rclone_cryptcheck/

If you want to compare as an example and see them change:

felix@gemini:~$ rclone copy /etc/hosts gcrypt:
felix@gemini:~$ rclone lsjson --hash gcrypt:hosts
[
{"Path":"hosts","Name":"hosts","Size":221,"MimeType":"application/octet-stream","ModTime":"2021-11-28T16:25:53.161Z","IsDir":false,"ID":"1Du8e3PSA-6XgbEQabGZWL2_M-IDQUoDw"}
]
felix@gemini:~$ rclone delete gcrypt:hosts
felix@gemini:~$ rclone copy /etc/hosts gcrypt:
felix@gemini:~$ rclone lsjson --hash gcrypt:hosts
[
{"Path":"hosts","Name":"hosts","Size":221,"MimeType":"application/octet-stream","ModTime":"2021-11-28T16:25:53.161Z","IsDir":false,"ID":"1EhEx9G-GddZHPP7Yrbb7OWyleJgPIb1O"}
]

4

I have:

gdrive:directory1/file1

gdrivecrypt:directory2/file1

They are the same file, but in 2 separate directories. I essentially want to sync the directories but rclone recognizes them as 2 different files now.

5

So there is no way for me to 'trick' rclone into thinking they are the same file then?

6

note: cannot mix crypted and non-crypted files in the same dir.

when you run the command, rclone will crypt the source files from gdrive: and copy them to gdrivecrypt:
rclone will save the source file md5 hash as metadata in the dest file.
rclone will not re-sync a source file unless it has changed.

if you would use a debug log, you can see what rclone is doing
add -vv to the command. would see something like

rclone sync gdrive:source gcrypt:dest -vv 
DEBUG : file.txt: md5 = f0d751816d3dc5a6675e1ba22e68d82e OK
INFO  : file.txt: Copied (new)

rclone cryptcheck gdrive:source gcrypt:dest -vv 
INFO  : Using md5 for hash comparisons
DEBUG : file.txt: OK
NOTICE: Encrypted drive 'gcrypt:dest': 0 differences found
NOTICE: Encrypted drive 'gcrypt:dest': 1 matching files
7

If they are the same file, they wouldn't reupload though.

felix@gemini:~$ rclone copy /etc/hosts gcrypt: -vv
2021/11/28 18:14:54 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2021/11/28 18:14:54 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "copy" "/etc/hosts" "gcrypt:" "-vv"]
2021/11/28 18:14:54 DEBUG : Creating backend with remote "/etc/hosts"
2021/11/28 18:14:54 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2021/11/28 18:14:54 DEBUG : fs cache: adding new entry for parent of "/etc/hosts", "/etc"
2021/11/28 18:14:54 DEBUG : Creating backend with remote "gcrypt:"
2021/11/28 18:14:54 DEBUG : Creating backend with remote "GD:crypt"
2021/11/28 18:14:54 DEBUG : hosts: Size and modification time the same (differ by -909.132µs, within tolerance 1ms)
2021/11/28 18:14:54 DEBUG : hosts: Unchanged skipping
2021/11/28 18:14:54 INFO  :
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         0.5s

2021/11/28 18:14:54 DEBUG : 4 go routines active

You'd have to run the command and share some debug output to see why they are uploading if they are the same file.

8

Okay so I ran the code with -vv and it said:

Modification times differ by -12.741s: 2018-06-16 22:23:59.741 +0000 UTC, 2018-06-16 22:23:47 +0000 UTC

Not even sure how that happened, as it was 3 years ago. I then ran the code with --checksum and that lets me run the copy command without overwriting the existing file. To fully fix the issue, I'm trying to change the metadata of the files so they match up. Thanks guys.

9

Depending on how lucky you feel, you can change the modify window to something bigger to work around that:

   --modify-window duration               Max time diff to be considered the same (default 1ns)
1 Like
10

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.