Possible issue with encrypted chunker (chunker -> crypt -> remote)

What is the problem you are having with rclone?

I was testing out encrypted chunking, and it seems that chunker -> crypt -> remote fails, while crypt -> chunker -> remote succeeds. I'd like to use the former, since it hides the chunked filenames too, as mentioned in this forum post: Encrypted chunker.

What is your rclone version (output from rclone version)

rclone v1.53.0-DEV
- os/arch: darwin/amd64
- go version: go1.15

EDIT: I tried it with the latest beta as well:

rclone v1.54.0-beta.4751.4a001b8a0
- os/arch: darwin/amd64
- go version: go1.15

Which OS you are using and how many bits (eg Windows 7, 64 bit)

MacOS Catalina - 10.15.6, 64 bit

Which cloud storage system are you using? (eg Google Drive)

Testing with local filesystem.

EDIT: Verified that it has the same issue with Google Drive as a remote.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone -vv sync /tmp/input.4800 test-chunk-chunker:

The rclone config contents with secrets removed.

[test-chunk-crypt]
type = crypt
remote = /tmp/mytestdir/
filename_encryption = standard
directory_name_encryption = true
password = REDACTED

[test-chunk-chunker]
type = chunker
remote = test-chunk-crypt:
chunk_size = 2k
hash_type = sha1
name_format = *.rcc.###
fail_hard = true

[test-chunk-chunklast]
type = chunker
remote = /tmp/mychunklastdir/
chunk_size = 2k
hash_type = sha1
name_format = *.rcc.###
fail_hard = true

[test-chunk-cryptfirst]
type = crypt
remote = test-chunk-chunklast:
password = REDACTED

A log from the command with the -vv flag

# Create a 4800 byte file.
$ ruby -e 'puts("0123456789abcde\n" * 300)' >input.4800

$ rclone -vv sync /tmp/input.4800 test-chunk-chunker:
2020/09/05 10:41:26 DEBUG : rclone: Version "v1.53.0-DEV" starting with parameters ["rclone" "-vvv" "sync" "/tmp/input.4800" "test-chunk-chunker:"]
2020/09/05 10:41:26 DEBUG : Creating backend with remote "/tmp/input.4800"
Enter configuration password:
password:
2020/09/05 10:41:29 DEBUG : Using config file from "/home/.config/rclone/rclone.conf"
2020/09/05 10:41:29 DEBUG : fs cache: adding new entry for parent of "/tmp/input.4800", "/tmp"
2020/09/05 10:41:29 DEBUG : Creating backend with remote "test-chunk-chunker:"
2020/09/05 10:41:29 DEBUG : Creating backend with remote "test-chunk-crypt:"
2020/09/05 10:41:29 DEBUG : Creating backend with remote "/tmp/mytestdir/"
2020/09/05 10:41:29 DEBUG : input.4800: Need to transfer - File not found at Destination
2020/09/05 10:41:29 INFO  : input.4800.rcc.001_6o2986: Moved (server side)
2020/09/05 10:41:29 ERROR : input.4800: corrupted on transfer: sizes differ 4800 vs 4832
2020/09/05 10:41:29 INFO  : input.4800: Removing failed copy
2020/09/05 10:41:29 ERROR : Attempt 1/3 failed with 1 errors and: corrupted on transfer: sizes differ 4800 vs 4832
2020/09/05 10:41:29 DEBUG : input.4800: Need to transfer - File not found at Destination
2020/09/05 10:41:29 INFO  : input.4800.rcc.001_6o297i: Moved (server side)
2020/09/05 10:41:29 ERROR : input.4800: corrupted on transfer: sizes differ 4800 vs 4832
2020/09/05 10:41:29 INFO  : input.4800: Removing failed copy
2020/09/05 10:41:29 ERROR : Attempt 2/3 failed with 1 errors and: corrupted on transfer: sizes differ 4800 vs 4832
2020/09/05 10:41:29 DEBUG : input.4800: Need to transfer - File not found at Destination
2020/09/05 10:41:29 INFO  : input.4800.rcc.001_6o29ff: Moved (server side)
2020/09/05 10:41:29 ERROR : input.4800: corrupted on transfer: sizes differ 4800 vs 4832
2020/09/05 10:41:29 INFO  : input.4800: Removing failed copy
2020/09/05 10:41:29 ERROR : Attempt 3/3 failed with 1 errors and: corrupted on transfer: sizes differ 4800 vs 4832
2020/09/05 10:41:29 INFO  :
Transferred:   	   14.297k / 14.297 kBytes, 100%, 2.012 MBytes/s, ETA 0s
Errors:                 1 (retrying may help)
Checks:                 3 / 3, 100%
Renamed:                3
Elapsed time:         2.6s

2020/09/05 10:41:29 DEBUG : 4 go routines active
2020/09/05 10:41:29 Failed to sync: corrupted on transfer: sizes differ 4800 vs 4832

####### 
# But, crypt -> chunk works fine.

$ rclone -vvv sync /tmp/input.4800 test-chunk-cryptfirst:
2020/09/05 10:45:52 DEBUG : rclone: Version "v1.53.0-DEV" starting with parameters ["rclone" "-vvv" "sync" "/tmp/input.4800" "test-chunk-cryptfirst:"]
2020/09/05 10:45:52 DEBUG : Creating backend with remote "/tmp/input.4800"
Enter configuration password:
password:
2020/09/05 10:45:56 DEBUG : Using config file from "/home/.config/rclone/rclone.conf"
2020/09/05 10:45:56 DEBUG : fs cache: adding new entry for parent of "/tmp/input.4800", "/tmp"
2020/09/05 10:45:56 DEBUG : Creating backend with remote "test-chunk-cryptfirst:"
2020/09/05 10:45:56 DEBUG : Creating backend with remote "test-chunk-chunklast:"
2020/09/05 10:45:56 DEBUG : Creating backend with remote "/tmp/mychunklastdir"
2020/09/05 10:45:56 DEBUG : input.4800: Need to transfer - File not found at Destination
2020/09/05 10:45:56 DEBUG : input.4800: Computing SHA-1 hash of encrypted source
2020/09/05 10:45:56 INFO  : pjbek3f6griff0g0r267njm5k4.rcc.001_6o9ovj: Moved (server side)
2020/09/05 10:45:56 INFO  : pjbek3f6griff0g0r267njm5k4.rcc.002_6o9ovj: Moved (server side)
2020/09/05 10:45:56 INFO  : pjbek3f6griff0g0r267njm5k4.rcc.003_6o9ovj: Moved (server side)
2020/09/05 10:45:56 INFO  : input.4800: Copied (new)
2020/09/05 10:45:56 INFO  :
Transferred:   	    4.734k / 4.734 kBytes, 100%, 1.315 MBytes/s, ETA 0s
Checks:                 3 / 3, 100%
Renamed:                3
Transferred:            1 / 1, 100%
Elapsed time:         4.7s

2020/09/05 10:45:56 DEBUG : 4 go routines active

That looks like a definite bug to me! Can you make a GitHub issue with the above in and we'll take a look. Thank you.

1 Like

@ncw Thank you for confirming that! I'll open a GitHub issue with all this information.

EDIT: For anyone following along, the GitHub issue is: https://github.com/rclone/rclone/issues/4570

1 Like