Couldn't move into backup dir: corrupted on transfer Chunker/Crypt/Jottacloud

jdr85 · September 9, 2021, 11:45am

What is the problem you are having with rclone?

Hi, I sync local files like this: files --> chunker --> crypt --> jottacloud. I do not want to delete modified/deleted files in my sync target but move them to a backup directory (with --backup-dir). Unfortunately this does not work, the backup dir is created but it is empty. The log is full of error messages like "Couldn't move into backup dir: corrupted on transfer" (see log below).

For the log I created an empty file testfile.txt and synced the whole folder containing some more files and one symlink. Then I modified the file and synced again.

What is your rclone version (output from `rclone version`)

- os/version: debian 10.10 (64 bit)
- os/kernel: 4.19.0-17-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.16.5
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Jottacloud Personal

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone -vv --delete-during --delete-excluded --backup-dir=Chunker-Crypt-Jotta:/daten_backup/eingehend/2021-09-09_133342 --exclude=.sync/Archive/** sync /daten/eingehend Chunker-Crypt-Jotta:/daten/eingehend

The rclone config contents with secrets removed.

[Jotta]
type = jottacloud
configVersion = 1
client_id = jottacli
client_secret = 
tokenURL = https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/token
token = {...}
device = 
mountpoint = 
hard_delete = false
no_versions = true

[Crypt-Jotta]
type = crypt
remote = Jotta:Sync_Crypt
password = ...
password2 = ...

[Chunker-Crypt-Jotta]
type = chunker
remote = Crypt-Jotta:
chunk_size = 512Mi
hash_type = md5all
name_format = *.rc##

A log from the command with the `-vv` flag

2021/09/09 13:33:42 DEBUG : Setting --config "/path/rclone.conf" from environment variable RCLONE_CONFIG="/path/rclone.conf"
2021/09/09 13:33:42 DEBUG : rclone: Version "v1.56.0" starting with parameters ["rclone" "-vv" "--delete-during" "--delete-excluded" "--backup-dir=Chunker-Crypt-Jotta:/daten_backup/eingehend/2021-09-09_133342" "--exclude=.sync/Archive/**" "sync" "/daten/eingehend" "Chunker-Crypt-Jotta:/daten/eingehend"]
2021/09/09 13:33:42 DEBUG : Creating backend with remote "/daten/eingehend"
2021/09/09 13:33:42 DEBUG : Using config file from "/path/rclone.conf"
2021/09/09 13:33:42 DEBUG : Creating backend with remote "Chunker-Crypt-Jotta:/daten/eingehend"
2021/09/09 13:33:42 DEBUG : Creating backend with remote "Crypt-Jotta:/daten/eingehend"
2021/09/09 13:33:43 DEBUG : Creating backend with remote "Jotta:Sync_Crypt/6fjn533fam0j33igmppc8f9bj4/5ada1ichrcqhulf1b44uk416jg"
2021/09/09 13:33:43 DEBUG : Reset feature "ListR"
2021/09/09 13:33:43 DEBUG : Creating backend with remote "Chunker-Crypt-Jotta:/daten_backup/eingehend/2021-09-09_133342"
2021/09/09 13:33:43 DEBUG : Creating backend with remote "Crypt-Jotta:/daten_backup/eingehend/2021-09-09_133342"
2021/09/09 13:33:43 DEBUG : Creating backend with remote "Jotta:Sync_Crypt/sm6oclu0un2as35drelteaogdg/5ada1ichrcqhulf1b44uk416jg/fjfbaul82opujbosun14et5qj527fd7omjdka0fsm748s4r7nm9g"
2021/09/09 13:33:43 DEBUG : Reset feature "ListR"
2021/09/09 13:33:43 DEBUG : Waiting for deletions to finish
2021/09/09 13:33:43 NOTICE: _external_: Can't follow symlink without -L/--copy-links
2021/09/09 13:33:43 DEBUG : .gocryptfs.reverse.conf: Size and modification time the same (differ by -218.456357ms, within tolerance 1s)
2021/09/09 13:33:43 DEBUG : .gocryptfs.reverse.conf: Unchanged skipping
2021/09/09 13:33:43 DEBUG : .backup: Size and modification time the same (differ by -72.894ms, within tolerance 1s)
2021/09/09 13:33:43 DEBUG : .backup: Unchanged skipping
2021/09/09 13:33:43 DEBUG : .gocryptfs.reverse_fne.conf: Size and modification time the same (differ by -408.96679ms, within tolerance 1s)
2021/09/09 13:33:43 DEBUG : .gocryptfs.reverse_fne.conf: Unchanged skipping
2021/09/09 13:33:43 DEBUG : Chunked 'Chunker-Crypt-Jotta:/daten/eingehend': Waiting for checks to finish
2021/09/09 13:33:43 DEBUG : testfile.txt: Sizes differ (src 9 vs dst 0)
2021/09/09 13:33:43 DEBUG : .gocryptfs.reverse_fnp.conf: Size and modification time the same (differ by 0s, within tolerance 1s)
2021/09/09 13:33:43 DEBUG : .gocryptfs.reverse_fnp.conf: Unchanged skipping
2021/09/09 13:33:44 DEBUG : testfile.txt: move 1 data chunks...
2021/09/09 13:33:44 INFO  : testfile.txt.rc01: Moved (server-side)
2021/09/09 13:33:44 INFO  : testfile.txt: Moved (server-side)
2021/09/09 13:33:44 ERROR : iutks9upgad69fcahdukc2d7tg: Failed to remove corrupted object: error 404: no.jotta.backup.errors.NoSuchFileException: File /60d6d0f30ded4d0134723b1d/Jotta/Archive/Sync_Crypt/sm6oclu0un2as35drelteaogdg/5ada1ichrcqhulf1b44uk416jg/fjfbaul82opujbosun14et5qj527fd7omjdka0fsm748s4r7nm9g/iutks9upgad69fcahdukc2d7tg does not exist (Not Found)
2021/09/09 13:33:45 ERROR : testfile.txt: Couldn't move: corrupted on transfer: md5 crypted hash differ "d41d8cd98f00b204e9800998ecf8427e" vs "869b514ffe43f635b17a1613076992a6"
2021/09/09 13:33:45 DEBUG : Chunked 'Chunker-Crypt-Jotta:/daten/eingehend': Waiting for transfers to finish
2021/09/09 13:33:45 ERROR : Chunked 'Chunker-Crypt-Jotta:/daten/eingehend': not deleting directories as there were IO errors
2021/09/09 13:33:45 ERROR : Attempt 1/3 failed with 1 errors and: corrupted on transfer: md5 crypted hash differ "d41d8cd98f00b204e9800998ecf8427e" vs "869b514ffe43f635b17a1613076992a6"
2021/09/09 13:33:45 DEBUG : Waiting for deletions to finish
2021/09/09 13:33:45 NOTICE: _external_: Can't follow symlink without -L/--copy-links
2021/09/09 13:33:45 DEBUG : .backup: Size and modification time the same (differ by -72.894ms, within tolerance 1s)
2021/09/09 13:33:45 DEBUG : .backup: Unchanged skipping
2021/09/09 13:33:45 DEBUG : .gocryptfs.reverse.conf: Size and modification time the same (differ by -218.456357ms, within tolerance 1s)
2021/09/09 13:33:45 DEBUG : .gocryptfs.reverse.conf: Unchanged skipping
2021/09/09 13:33:45 DEBUG : .gocryptfs.reverse_fne.conf: Size and modification time the same (differ by -408.96679ms, within tolerance 1s)
2021/09/09 13:33:45 DEBUG : .gocryptfs.reverse_fne.conf: Unchanged skipping
2021/09/09 13:33:45 DEBUG : .gocryptfs.reverse_fnp.conf: Size and modification time the same (differ by 0s, within tolerance 1s)
2021/09/09 13:33:45 DEBUG : .gocryptfs.reverse_fnp.conf: Unchanged skipping
2021/09/09 13:33:45 DEBUG : Chunked 'Chunker-Crypt-Jotta:/daten/eingehend': Waiting for checks to finish
2021/09/09 13:33:45 DEBUG : Chunked 'Chunker-Crypt-Jotta:/daten/eingehend': Waiting for transfers to finish
2021/09/09 13:33:45 DEBUG : testfile.txt: skip slow MD5 on source file, hashing in-transit
2021/09/09 13:33:46 DEBUG : testfile.txt: md5 = 31e1fc3526fd97d36b737b4ead8c9edc OK
2021/09/09 13:33:46 INFO  : testfile.txt.rc01_65d27d: Moved (server-side) to: testfile.txt.rc01
2021/09/09 13:33:47 DEBUG : testfile.txt: md5 = 47f5aad2bf636bb9d08286839a409b6d OK
2021/09/09 13:33:47 DEBUG : testfile.txt: md5 = d5829ed0f72673ea9e8f71079faeccee OK
2021/09/09 13:33:47 INFO  : testfile.txt: Copied (new)
2021/09/09 13:33:47 ERROR : Attempt 2/3 succeeded
2021/09/09 13:33:47 INFO  : 
Transferred:   	         57 / 57 Byte, 100%, 56 Byte/s, ETA 0s
Checks:                13 / 13, 100%
Renamed:                3
Transferred:            1 / 1, 100%
Elapsed time:         4.3s

2021/09/09 13:33:47 DEBUG : 10 go routines active

jdr85 · September 10, 2021, 6:46am

I did some more experiments and found out that other things such as server-side copy or --suffix instead of --backup-dir do not work either. However, everything including --backup-dir seems to work fine if I set hash_type = none in Chunker config. Why?

jdr85 · September 22, 2021, 5:56am

Sorry for pushing this...does nobody have any idea why this happens or any hint how to isolate the reason for this error?

albertony · September 22, 2021, 9:04am

Perhaps it is a file name length issue?

Chunker encodes chunk number in file name, so with default name_format setting it adds 17 characters. Also chunker adds 7 characters of temporary suffix during operations. Many file systems limit base file name without path by 255 characters. Using rclone's crypt remote as a base file system limits file name by 143 characters. Thus, maximum name length is 231 for most files and 119 for chunker-over-crypt. A user in need can change name format to e.g. *.rcc## and save 10 characters (provided at most 99 chunks per file).

(from "Caveats and Limitations" in chunker docs).

ivandeex · September 22, 2021, 10:11am

tl;dr

disabling chunker hash checks for crypt is the right thing to do
we miss a runtime warning and a documented caveat for chunker+hash configured over crypt

crypt is a file transformation layer (a wrapping backend in rclone parlance).
crypt is special.

whenever you upload the same content into crypt, it gets encrypted with a new random IV (initialization vector). I don't know a way to configure crypt with a fixed/constant IV.
@ncw am i right or is there one?

in other words, the same unencrypted content (with fixed hash) has a large number of encrypted representations (with many hashes)

sometimes however the encrypted content is preserved. for example, when filename encryption is disabled and the file is renamed within the same directory. chunker will query the underlying crypt whether it can use a fast "server-side move" method and crypt will accept that. in the end the encrypted file will just change its name but keep encrypted content and hash.

similar caveat was reported in subthread below Errors uploading files when using -crypt-no-data-encryption=true · Issue #5498 · rclone/rclone · GitHub

jdr85 · September 23, 2021, 9:45am

Thanks a lot, this really sounds like the root cause for this strange behavior - although I am just a regular user and do not understand everything in detail.
I fully agree with you that this caveat should be documented to avoid sleepless nights for other people.

I have now set Chunker hash_type=none and it seems to work fine! Am I right in my assumption that checksums and therefore data consistency will still be handled by the underlying remotes?

ivandeex · September 23, 2021, 10:01am

crypt maintains data consistency internally. you can confirm it by applying rclone cryptcheck Crypt-Jotta:path/to/file.rcNN on every chunk.

Applying checksums on crypt files directly can be tricky due to reasons explained above - they might change encrypted content when copied/moved. Dedicated cryptcheck with two outcomes (OK vs corrupted) is what we have now.

You could however change the order of layers and run crypt on top of chunker to have both chunker hashsums and crypt consistency checks together.

@ncw
What do you think about idea to extend the cryptcheck command so it will also accept chunker and consequently work on the underlying chunks?

ivandeex · September 23, 2021, 10:12am

@ncw Again, is there a configuration setting to fix (set constant) IV for crypt files? Is it possible in theory?

system · September 26, 2021, 10:13am

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.