What is the problem you are having with rclone?
I was testing out encrypted chunking, and it seems that chunker -> crypt -> remote fails, while crypt -> chunker -> remote succeeds. I'd like to use the former, since it hides the chunked filenames too, as mentioned in this forum post: Encrypted chunker.
What is your rclone version (output from rclone version)
rclone v1.53.0-DEV
- os/arch: darwin/amd64
- go version: go1.15
EDIT: I tried it with the latest beta as well:
rclone v1.54.0-beta.4751.4a001b8a0
- os/arch: darwin/amd64
- go version: go1.15
Which OS you are using and how many bits (eg Windows 7, 64 bit)
MacOS Catalina - 10.15.6, 64 bit
Which cloud storage system are you using? (eg Google Drive)
Testing with local filesystem.
EDIT: Verified that it has the same issue with Google Drive as a remote.
The command you were trying to run (eg rclone copy /tmp remote:tmp)
rclone -vv sync /tmp/input.4800 test-chunk-chunker:
The rclone config contents with secrets removed.
[test-chunk-crypt]
type = crypt
remote = /tmp/mytestdir/
filename_encryption = standard
directory_name_encryption = true
password = REDACTED
[test-chunk-chunker]
type = chunker
remote = test-chunk-crypt:
chunk_size = 2k
hash_type = sha1
name_format = *.rcc.###
fail_hard = true
[test-chunk-chunklast]
type = chunker
remote = /tmp/mychunklastdir/
chunk_size = 2k
hash_type = sha1
name_format = *.rcc.###
fail_hard = true
[test-chunk-cryptfirst]
type = crypt
remote = test-chunk-chunklast:
password = REDACTED
A log from the command with the -vv flag
# Create a 4800 byte file.
$ ruby -e 'puts("0123456789abcde\n" * 300)' >input.4800
$ rclone -vv sync /tmp/input.4800 test-chunk-chunker:
2020/09/05 10:41:26 DEBUG : rclone: Version "v1.53.0-DEV" starting with parameters ["rclone" "-vvv" "sync" "/tmp/input.4800" "test-chunk-chunker:"]
2020/09/05 10:41:26 DEBUG : Creating backend with remote "/tmp/input.4800"
Enter configuration password:
password:
2020/09/05 10:41:29 DEBUG : Using config file from "/home/.config/rclone/rclone.conf"
2020/09/05 10:41:29 DEBUG : fs cache: adding new entry for parent of "/tmp/input.4800", "/tmp"
2020/09/05 10:41:29 DEBUG : Creating backend with remote "test-chunk-chunker:"
2020/09/05 10:41:29 DEBUG : Creating backend with remote "test-chunk-crypt:"
2020/09/05 10:41:29 DEBUG : Creating backend with remote "/tmp/mytestdir/"
2020/09/05 10:41:29 DEBUG : input.4800: Need to transfer - File not found at Destination
2020/09/05 10:41:29 INFO : input.4800.rcc.001_6o2986: Moved (server side)
2020/09/05 10:41:29 ERROR : input.4800: corrupted on transfer: sizes differ 4800 vs 4832
2020/09/05 10:41:29 INFO : input.4800: Removing failed copy
2020/09/05 10:41:29 ERROR : Attempt 1/3 failed with 1 errors and: corrupted on transfer: sizes differ 4800 vs 4832
2020/09/05 10:41:29 DEBUG : input.4800: Need to transfer - File not found at Destination
2020/09/05 10:41:29 INFO : input.4800.rcc.001_6o297i: Moved (server side)
2020/09/05 10:41:29 ERROR : input.4800: corrupted on transfer: sizes differ 4800 vs 4832
2020/09/05 10:41:29 INFO : input.4800: Removing failed copy
2020/09/05 10:41:29 ERROR : Attempt 2/3 failed with 1 errors and: corrupted on transfer: sizes differ 4800 vs 4832
2020/09/05 10:41:29 DEBUG : input.4800: Need to transfer - File not found at Destination
2020/09/05 10:41:29 INFO : input.4800.rcc.001_6o29ff: Moved (server side)
2020/09/05 10:41:29 ERROR : input.4800: corrupted on transfer: sizes differ 4800 vs 4832
2020/09/05 10:41:29 INFO : input.4800: Removing failed copy
2020/09/05 10:41:29 ERROR : Attempt 3/3 failed with 1 errors and: corrupted on transfer: sizes differ 4800 vs 4832
2020/09/05 10:41:29 INFO :
Transferred: 14.297k / 14.297 kBytes, 100%, 2.012 MBytes/s, ETA 0s
Errors: 1 (retrying may help)
Checks: 3 / 3, 100%
Renamed: 3
Elapsed time: 2.6s
2020/09/05 10:41:29 DEBUG : 4 go routines active
2020/09/05 10:41:29 Failed to sync: corrupted on transfer: sizes differ 4800 vs 4832
#######
# But, crypt -> chunk works fine.
$ rclone -vvv sync /tmp/input.4800 test-chunk-cryptfirst:
2020/09/05 10:45:52 DEBUG : rclone: Version "v1.53.0-DEV" starting with parameters ["rclone" "-vvv" "sync" "/tmp/input.4800" "test-chunk-cryptfirst:"]
2020/09/05 10:45:52 DEBUG : Creating backend with remote "/tmp/input.4800"
Enter configuration password:
password:
2020/09/05 10:45:56 DEBUG : Using config file from "/home/.config/rclone/rclone.conf"
2020/09/05 10:45:56 DEBUG : fs cache: adding new entry for parent of "/tmp/input.4800", "/tmp"
2020/09/05 10:45:56 DEBUG : Creating backend with remote "test-chunk-cryptfirst:"
2020/09/05 10:45:56 DEBUG : Creating backend with remote "test-chunk-chunklast:"
2020/09/05 10:45:56 DEBUG : Creating backend with remote "/tmp/mychunklastdir"
2020/09/05 10:45:56 DEBUG : input.4800: Need to transfer - File not found at Destination
2020/09/05 10:45:56 DEBUG : input.4800: Computing SHA-1 hash of encrypted source
2020/09/05 10:45:56 INFO : pjbek3f6griff0g0r267njm5k4.rcc.001_6o9ovj: Moved (server side)
2020/09/05 10:45:56 INFO : pjbek3f6griff0g0r267njm5k4.rcc.002_6o9ovj: Moved (server side)
2020/09/05 10:45:56 INFO : pjbek3f6griff0g0r267njm5k4.rcc.003_6o9ovj: Moved (server side)
2020/09/05 10:45:56 INFO : input.4800: Copied (new)
2020/09/05 10:45:56 INFO :
Transferred: 4.734k / 4.734 kBytes, 100%, 1.315 MBytes/s, ETA 0s
Checks: 3 / 3, 100%
Renamed: 3
Transferred: 1 / 1, 100%
Elapsed time: 4.7s
2020/09/05 10:45:56 DEBUG : 4 go routines active