Files no longer identical after move on gdrive crypt remote and locally

Help and Support
#1

Note: I am using my OS repo version of rclone, not latest. Is this okay?

What is the problem you are having with rclone?

My local directory tree is getting cumbersome so I moved some directories. e.g.

mkdir funky-new-directory
mv a b c funky-new-directory

I made the same changes on my gdrive crypt remote to keep them in sync without uploading everything again.

rclone mkdir remote:top-level/funky-new-directory
rclone mkdir remote:top-level/funky-new-directory/a
rclone mkdir remote:top-level/funky-new-directory/b
rclone mkdir remote:top-level/funky-new-directory/c
rclone move remote:top-level/a remote:top-level/funky-new-directory/a
rclone move remote:top-level/b remote:top-level/funky-new-directory/b
rclone move remote:top-level/c remote:top-level/funky-new-directory/c

Now, when I do a dry run sync on one of the directories, rclone wants to upload some of the local files. These files looked identical before the move.

$ rclone sync funky-new-directory/a remote:top-level/funky-new-directory/a --dry-run
2022/09/09 18:56:26 NOTICE: foo: Skipped copy as --dry-run is set (size 13.344Ki)
2022/09/09 18:56:27 NOTICE: bar: Skipped copy as --dry-run is set (size 99.046Ki)
2022/09/09 18:56:27 NOTICE: baz: Skipped copy as --dry-run is set (size 99.136Ki)
2022/09/09 18:56:27 NOTICE: bop: Skipped copy as --dry-run is set (size 19.267Ki)
2022/09/09 18:56:27 NOTICE: 
Transferred:   	  230.792Ki / 230.792 KiByte, 100%, 0 Byte/s, ETA -
Checks:                38 / 38, 100%
Transferred:            4 / 4, 100%
Elapsed time:         2.6s

When I run the check command, it shows more differences than the dry run sync indicates.

$ rclone check funky-new-directory/a remote:top-level/funky-new-directory/a
2022/09/09 18:59:39 ERROR : No common hash found - not using a hash for checks
2022/09/09 18:59:39 ERROR : foo: File not in Encrypted drive 'remote:top-level/funky-new-directory/a'
etc...

2022/09/09 18:59:39 NOTICE: Encrypted drive 'remote:top-level/funky-new-directory/a': 38 files missing
2022/09/09 18:59:39 NOTICE: Encrypted drive 'remote:top-level/funky-new-directory/a': 38 differences found
2022/09/09 18:59:39 NOTICE: Encrypted drive 'remote:top-level/funky-new-directory/a': 38 errors while checking
2022/09/09 18:59:39 Failed to check with 38 errors: last error was: 38 differences found

Run the command 'rclone version' and share the full output of the command.

$ rclone version
rclone v1.56.0
- os/version: debian 10.12 (64 bit)
- os/kernel: 5.9.1-rt20avl1 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.16.5
- go/linking: static
- go/tags: none

$ apt show rclone
Package: rclone
Version: 1.45-3

Which cloud storage system are you using? (eg Google Drive)

google drive

[primary]
type = drive
scope = drive.file
team_drive = 

[remote]
type = crypt
remote = primary:encrypted

The command you were trying to run (eg rclone copy /tmp remote:tmp)

See above.

The rclone config contents with secrets removed.

See above

A log from the command with the -vv flag

Not sure which command to run with this flag.

#2

I realise I didn't express a question here, which given that I'm asking for help seems silly.

How can I avoid this loss of identical-file status when moving directories in the future? (Some of the directories are a lot bigger than the one I showed in my examples.)

#5

I'm guessing that a couple of days with no reply means that no one has experience of anything similar.

Am I correct in thinking that I would have to replicate this with the latest version of rclone if I'm going to submit a bug report?

#6

Just for kicks, I installed the latest rclone to see what that would say.

$ rclone version
rclone v1.59.1
- os/version: debian 10.12 (64 bit)
- os/kernel: 5.9.1-rt20avl1 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.18.5
- go/linking: static
- go/tags: none

A dry run sync said exactly the same as above. (4 files out of 38 would be copied.) In contrast, the check command gave the opposite result from above. (Zero differences rather than 38 differences.)

$ rclone check psychotherapy/mmcyp paranoid:Documents/psychotherapy/mmcyp
2022/09/13 11:07:46 ERROR : No common hash found - not using a hash for checks
2022/09/13 11:07:47 NOTICE: Encrypted drive 'remote:top-level/funky-new-directory/a': 0 differences found
2022/09/13 11:07:47 NOTICE: Encrypted drive 'remote:top-level/funky-new-directory/a': 38 hashes could not be checked
2022/09/13 11:07:47 NOTICE: Encrypted drive 'remote:top-level/funky-new-directory/a': 38 matching files
#7

Hi crantok,

I had a look at your post and found it difficult to establish exactly what had happened and therefore suspected a mistake on your side. I may well be wrong, but that's why I didn't post initially.

My analysis was that the situation could have occurred if you didn't perform this:

or made a typo in it or something similar. I simply miss information to rule out many of the mistakes that could have happened on your side.

I would therefore love to see the (sequential non-broken) output from a command sequence something like this:

mkdir testfolder
mkdir testfolder/a
echo "Hello World" > testfolder/a/foo
rclone mkdir  remote:testfolder
rclone sync ./testfolder remote:testfolder -v
mkdir testfolder/funky-new-directory
mv testfolder/a testfolder/funky-new-directory
rclone lsl remote:testfolder
rclone mkdir remote:testfolder/funky-new-directory
rclone move remote:testfolder/a remote:remote:testfolder/funky-new-directory/a -v
rclone lsl remote:testfolder
rclone sync ./testfolder remote:testfolder --dry-run -v
rclone check ./testfolder remote:testfolder -v

performed on the latest version of rclone.

Then my next step might be to ask for a repetition of the sequence where debug info (-vv) is output from the command(s) suspected to cause the issue.

But before trying that, perhaps we should investigate if there is a simpler way to move your folders around. Did you consider/investigate using rclone sync --track-renames?

https://rclone.org/docs/#track-renames
https://rclone.org/docs/#track-renames-strategy-hash-modtime-leaf-size

#8

Thanks @Ole . I should try replicating with a simpler example like the one you gave. Only a quarter of files lost their identical-to-local status so it might be something about specific files types/names/etc.

I had not noticed track-renames so thank you for pointing that out. The docs say that track renames can use the modtime and leaf strategies on encrypted remotes so I will definitely experiment with that if I run into the same thing again.

The docs also say that the hash strategy is the default and that the hash strategy is not available on encrypted remotes. That made me wonder about what check was doing. Turns out, I should have been using cryptcheck instead. Oops.

#9

You're welcome!

Hope you find an easy way to fix/move/upload the remaining files. It is hard to say what happened. I doubt it is related to rclone or the files types/name, but sure ready to investigate if you manage to reproduce.

You can also use check --download, which I use on a crypt based on a remote without hash sums.