What is the problem you are having with rclone?
Total amount of data transferred is double the source file size when using chunker + crypt + --crypt-no-data-encryption=true. I have also verified on the network that the data is doubled and not just a reporting issue.
What is your rclone version (output from rclone version
)
rclone v1.57.0-beta.5609.22ad1e3be.fix-5498-crypt-no-encrypt
- os/version: ubuntu 18.04 (64 bit)
- os/kernel: 4.15.0-153-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.16.6
- go/linking: static
- go/tags: none
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Tested on Ubuntu 18.04 vm, and FreeBSD 64bit.
Which cloud storage system are you using? (eg Google Drive)
Google Drive
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
rclone copy 200mbtestfile1 chunker:1 -vv --crypt-no-data-encryption=true
rclone copy 200mbtestfile1 chunker:2 -vv --crypt-no-data-encryption=false
The rclone config contents with secrets removed.
[gdrive]
type = drive
scope = drive
token =
[crypt]
type = crypt
remote = gdrive:back
filename_encryption = standard
directory_name_encryption = true
password = abcdefghijklmnopqrstuvwxyz
password2 = abcdefghijklmnopqrstuvwxyz
[chunker]
type = chunker
remote = crypt:
chunk_size = 100M
hash_type = md5
name_format = *.rcc###
meta_format = simplejson
A log from the command with the -vv
flag
Total Transferred data is 400mb with crypt-no-data-encryption=true
root@docker:~/rclone-v1.57.0-beta.5609.22ad1e3be.fix-5498-crypt-no-encrypt-linux-amd64# ./rclone copy 200mbtestfile1 chunker:1 -vv --crypt-no-data-encryption=true
2021/08/18 17:48:11 DEBUG : rclone: Version "v1.57.0-beta.5609.22ad1e3be.fix-5498-crypt-no-encrypt" starting with parameters ["./rclone" "copy" "200mbtestfile1" "chunker:1" "-vv" "--crypt-no-data-encryption=true"]
2021/08/18 17:48:11 DEBUG : Creating backend with remote "200mbtestfile1"
2021/08/18 17:48:11 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/08/18 17:48:11 DEBUG : fs cache: adding new entry for parent of "200mbtestfile1", "/root/rclone-v1.57.0-beta.5609.22ad1e3be.fix-5498-crypt-no-encrypt-linux-amd64"
2021/08/18 17:48:11 DEBUG : Creating backend with remote "chunker:1"
2021/08/18 17:48:11 DEBUG : Creating backend with remote "crypt:1"
2021/08/18 17:48:11 DEBUG : crypt: detected overridden config - adding "{94O0Q}" suffix to name
2021/08/18 17:48:11 DEBUG : Creating backend with remote "gdrive:back/gk4i51h3lcf98821kcdo8d5hb4"
2021/08/18 17:48:11 DEBUG : Google drive root 'back/gk4i51h3lcf98821kcdo8d5hb4': 'root_folder_id = 0AABIkKINH93LUk9PVA' - save this in the config to speed up startup
2021/08/18 17:48:12 DEBUG : fs cache: renaming cache item "crypt:1" to be canonical "crypt{94O0Q}:1"
2021/08/18 17:48:12 DEBUG : Reset feature "ListR"
2021/08/18 17:48:12 DEBUG : 200mbtestfile1: Need to transfer - File not found at Destination
2021/08/18 17:48:12 DEBUG : 200mbtestfile1: skip slow MD5 on source file, hashing in-transit
2021/08/18 17:48:13 DEBUG : juieeg9b8dn681et3u3ca6ghk3im0bi28p996fu62pdmf1pnb8c0: Sending chunk 0 length 8388608
2021/08/18 17:48:14 DEBUG : juieeg9b8dn681et3u3ca6ghk3im0bi28p996fu62pdmf1pnb8c0: Sending chunk 8388608 length 8388608
2021/08/18 17:48:14 DEBUG : juieeg9b8dn681et3u3ca6ghk3im0bi28p996fu62pdmf1pnb8c0: Sending chunk 16777216 length 8388608
2021/08/18 17:48:14 DEBUG : juieeg9b8dn681et3u3ca6ghk3im0bi28p996fu62pdmf1pnb8c0: Sending chunk 25165824 length 8388608
2021/08/18 17:48:14 DEBUG : juieeg9b8dn681et3u3ca6ghk3im0bi28p996fu62pdmf1pnb8c0: Sending chunk 33554432 length 8388608
2021/08/18 17:48:15 DEBUG : juieeg9b8dn681et3u3ca6ghk3im0bi28p996fu62pdmf1pnb8c0: Sending chunk 41943040 length 8388608
2021/08/18 17:48:15 DEBUG : juieeg9b8dn681et3u3ca6ghk3im0bi28p996fu62pdmf1pnb8c0: Sending chunk 50331648 length 8388608
2021/08/18 17:48:15 DEBUG : juieeg9b8dn681et3u3ca6ghk3im0bi28p996fu62pdmf1pnb8c0: Sending chunk 58720256 length 8388608
2021/08/18 17:48:15 DEBUG : juieeg9b8dn681et3u3ca6ghk3im0bi28p996fu62pdmf1pnb8c0: Sending chunk 67108864 length 8388608
2021/08/18 17:48:16 DEBUG : juieeg9b8dn681et3u3ca6ghk3im0bi28p996fu62pdmf1pnb8c0: Sending chunk 75497472 length 8388608
2021/08/18 17:48:16 DEBUG : juieeg9b8dn681et3u3ca6ghk3im0bi28p996fu62pdmf1pnb8c0: Sending chunk 83886080 length 8388608
2021/08/18 17:48:16 DEBUG : juieeg9b8dn681et3u3ca6ghk3im0bi28p996fu62pdmf1pnb8c0: Sending chunk 92274688 length 8388608
2021/08/18 17:48:16 DEBUG : juieeg9b8dn681et3u3ca6ghk3im0bi28p996fu62pdmf1pnb8c0: Sending chunk 100663296 length 4194304
2021/08/18 17:48:18 DEBUG : o2mm978qcuqmokshucn0c22mtcm340n1kar9pqr6epljbpnig2d0: Sending chunk 0 length 8388608
2021/08/18 17:48:18 DEBUG : o2mm978qcuqmokshucn0c22mtcm340n1kar9pqr6epljbpnig2d0: Sending chunk 8388608 length 8388608
2021/08/18 17:48:18 DEBUG : o2mm978qcuqmokshucn0c22mtcm340n1kar9pqr6epljbpnig2d0: Sending chunk 16777216 length 8388608
2021/08/18 17:48:18 DEBUG : o2mm978qcuqmokshucn0c22mtcm340n1kar9pqr6epljbpnig2d0: Sending chunk 25165824 length 8388608
2021/08/18 17:48:19 DEBUG : o2mm978qcuqmokshucn0c22mtcm340n1kar9pqr6epljbpnig2d0: Sending chunk 33554432 length 8388608
2021/08/18 17:48:19 DEBUG : o2mm978qcuqmokshucn0c22mtcm340n1kar9pqr6epljbpnig2d0: Sending chunk 41943040 length 8388608
2021/08/18 17:48:19 DEBUG : o2mm978qcuqmokshucn0c22mtcm340n1kar9pqr6epljbpnig2d0: Sending chunk 50331648 length 8388608
2021/08/18 17:48:19 DEBUG : o2mm978qcuqmokshucn0c22mtcm340n1kar9pqr6epljbpnig2d0: Sending chunk 58720256 length 8388608
2021/08/18 17:48:20 DEBUG : o2mm978qcuqmokshucn0c22mtcm340n1kar9pqr6epljbpnig2d0: Sending chunk 67108864 length 8388608
2021/08/18 17:48:20 DEBUG : o2mm978qcuqmokshucn0c22mtcm340n1kar9pqr6epljbpnig2d0: Sending chunk 75497472 length 8388608
2021/08/18 17:48:20 DEBUG : o2mm978qcuqmokshucn0c22mtcm340n1kar9pqr6epljbpnig2d0: Sending chunk 83886080 length 8388608
2021/08/18 17:48:21 DEBUG : o2mm978qcuqmokshucn0c22mtcm340n1kar9pqr6epljbpnig2d0: Sending chunk 92274688 length 8388608
2021/08/18 17:48:21 DEBUG : o2mm978qcuqmokshucn0c22mtcm340n1kar9pqr6epljbpnig2d0: Sending chunk 100663296 length 4194304
2021/08/18 17:48:26 DEBUG : pg8f9sce6e0tf95ieov9etq43grheofobl2jcspvbbhae82si890: Sending chunk 0 length 8388608
2021/08/18 17:48:26 DEBUG : pg8f9sce6e0tf95ieov9etq43grheofobl2jcspvbbhae82si890: Sending chunk 8388608 length 8388608
2021/08/18 17:48:27 DEBUG : pg8f9sce6e0tf95ieov9etq43grheofobl2jcspvbbhae82si890: Sending chunk 16777216 length 8388608
2021/08/18 17:48:27 DEBUG : pg8f9sce6e0tf95ieov9etq43grheofobl2jcspvbbhae82si890: Sending chunk 25165824 length 8388608
2021/08/18 17:48:27 DEBUG : pg8f9sce6e0tf95ieov9etq43grheofobl2jcspvbbhae82si890: Sending chunk 33554432 length 8388608
2021/08/18 17:48:27 DEBUG : pg8f9sce6e0tf95ieov9etq43grheofobl2jcspvbbhae82si890: Sending chunk 41943040 length 8388608
2021/08/18 17:48:28 DEBUG : pg8f9sce6e0tf95ieov9etq43grheofobl2jcspvbbhae82si890: Sending chunk 50331648 length 8388608
2021/08/18 17:48:28 DEBUG : pg8f9sce6e0tf95ieov9etq43grheofobl2jcspvbbhae82si890: Sending chunk 58720256 length 8388608
2021/08/18 17:48:28 DEBUG : pg8f9sce6e0tf95ieov9etq43grheofobl2jcspvbbhae82si890: Sending chunk 67108864 length 8388608
2021/08/18 17:48:29 DEBUG : pg8f9sce6e0tf95ieov9etq43grheofobl2jcspvbbhae82si890: Sending chunk 75497472 length 8388608
2021/08/18 17:48:29 DEBUG : pg8f9sce6e0tf95ieov9etq43grheofobl2jcspvbbhae82si890: Sending chunk 83886080 length 8388608
2021/08/18 17:48:29 DEBUG : pg8f9sce6e0tf95ieov9etq43grheofobl2jcspvbbhae82si890: Sending chunk 92274688 length 8388608
2021/08/18 17:48:30 DEBUG : pg8f9sce6e0tf95ieov9etq43grheofobl2jcspvbbhae82si890: Sending chunk 100663296 length 4194304
2021/08/18 17:48:30 INFO : juieeg9b8dn681et3u3ca6ghk3im0bi28p996fu62pdmf1pnb8c0: Copied (new) to: pg8f9sce6e0tf95ieov9etq43grheofobl2jcspvbbhae82si890
2021/08/18 17:48:31 INFO : juieeg9b8dn681et3u3ca6ghk3im0bi28p996fu62pdmf1pnb8c0: Deleted
2021/08/18 17:48:33 DEBUG : uemdjqlrc5fg6u5kp050r7alj48qh6q75r5e2qsvierub5dmilt0: Sending chunk 0 length 8388608
2021/08/18 17:48:33 DEBUG : uemdjqlrc5fg6u5kp050r7alj48qh6q75r5e2qsvierub5dmilt0: Sending chunk 8388608 length 8388608
2021/08/18 17:48:33 DEBUG : uemdjqlrc5fg6u5kp050r7alj48qh6q75r5e2qsvierub5dmilt0: Sending chunk 16777216 length 8388608
2021/08/18 17:48:34 DEBUG : uemdjqlrc5fg6u5kp050r7alj48qh6q75r5e2qsvierub5dmilt0: Sending chunk 25165824 length 8388608
2021/08/18 17:48:34 DEBUG : uemdjqlrc5fg6u5kp050r7alj48qh6q75r5e2qsvierub5dmilt0: Sending chunk 33554432 length 8388608
2021/08/18 17:48:34 DEBUG : uemdjqlrc5fg6u5kp050r7alj48qh6q75r5e2qsvierub5dmilt0: Sending chunk 41943040 length 8388608
2021/08/18 17:48:34 DEBUG : uemdjqlrc5fg6u5kp050r7alj48qh6q75r5e2qsvierub5dmilt0: Sending chunk 50331648 length 8388608
2021/08/18 17:48:35 DEBUG : uemdjqlrc5fg6u5kp050r7alj48qh6q75r5e2qsvierub5dmilt0: Sending chunk 58720256 length 8388608
2021/08/18 17:48:35 DEBUG : uemdjqlrc5fg6u5kp050r7alj48qh6q75r5e2qsvierub5dmilt0: Sending chunk 67108864 length 8388608
2021/08/18 17:48:35 DEBUG : uemdjqlrc5fg6u5kp050r7alj48qh6q75r5e2qsvierub5dmilt0: Sending chunk 75497472 length 8388608
2021/08/18 17:48:35 DEBUG : uemdjqlrc5fg6u5kp050r7alj48qh6q75r5e2qsvierub5dmilt0: Sending chunk 83886080 length 8388608
2021/08/18 17:48:36 DEBUG : uemdjqlrc5fg6u5kp050r7alj48qh6q75r5e2qsvierub5dmilt0: Sending chunk 92274688 length 8388608
2021/08/18 17:48:36 DEBUG : uemdjqlrc5fg6u5kp050r7alj48qh6q75r5e2qsvierub5dmilt0: Sending chunk 100663296 length 4194304
2021/08/18 17:48:37 INFO : o2mm978qcuqmokshucn0c22mtcm340n1kar9pqr6epljbpnig2d0: Copied (new) to: uemdjqlrc5fg6u5kp050r7alj48qh6q75r5e2qsvierub5dmilt0
2021/08/18 17:48:37 INFO : o2mm978qcuqmokshucn0c22mtcm340n1kar9pqr6epljbpnig2d0: Deleted
2021/08/18 17:48:38 INFO : 200mbtestfile1: Copied (new) to: cj5rmftkcga3ecog3r53rk3g1o
2021/08/18 17:48:38 INFO :
Transferred: 400Mi / 400 MiByte, 100%, 15.096 MiByte/s, ETA 0s
Checks: 4 / 4, 100%
Deleted: 2 (files), 0 (dirs)
Renamed: 2
Transferred: 3 / 3, 100%
Elapsed time: 27.1s
2021/08/18 17:48:38 DEBUG : 7 go routines active
Total Transferred data is 200mb with crypt-no-data-encryption=false
root@docker:~/rclone-v1.57.0-beta.5609.22ad1e3be.fix-5498-crypt-no-encrypt-linux-amd64# ./rclone copy 200mbtestfile1 chunker:2 -vv --crypt-no-data-encryption=false
2021/08/18 17:48:45 DEBUG : rclone: Version "v1.57.0-beta.5609.22ad1e3be.fix-5498-crypt-no-encrypt" starting with parameters ["./rclone" "copy" "200mbtestfile1" "chunker:2" "-vv" "--crypt-no-data-encryption=false"]
2021/08/18 17:48:45 DEBUG : Creating backend with remote "200mbtestfile1"
2021/08/18 17:48:45 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/08/18 17:48:45 DEBUG : fs cache: adding new entry for parent of "200mbtestfile1", "/root/rclone-v1.57.0-beta.5609.22ad1e3be.fix-5498-crypt-no-encrypt-linux-amd64"
2021/08/18 17:48:45 DEBUG : Creating backend with remote "chunker:2"
2021/08/18 17:48:45 DEBUG : Creating backend with remote "crypt:2"
2021/08/18 17:48:45 DEBUG : crypt: detected overridden config - adding "{ConAo}" suffix to name
2021/08/18 17:48:45 DEBUG : Creating backend with remote "gdrive:back/kta7igk8o9ifbhpbc13ff1b1eg"
2021/08/18 17:48:46 DEBUG : Google drive root 'back/kta7igk8o9ifbhpbc13ff1b1eg': 'root_folder_id = 0AABIkKINH93LUk9PVA' - save this in the config to speed up startup
2021/08/18 17:48:46 DEBUG : fs cache: renaming cache item "crypt:2" to be canonical "crypt{ConAo}:2"
2021/08/18 17:48:46 DEBUG : Reset feature "ListR"
2021/08/18 17:48:46 DEBUG : 200mbtestfile1: Need to transfer - File not found at Destination
2021/08/18 17:48:46 DEBUG : 200mbtestfile1: skip slow MD5 on source file, hashing in-transit
2021/08/18 17:48:48 DEBUG : ib6db9lf0ui5292i2pdsced82s3c5cjkccplcge8insqds9othjg: Sending chunk 0 length 8388608
2021/08/18 17:48:48 DEBUG : ib6db9lf0ui5292i2pdsced82s3c5cjkccplcge8insqds9othjg: Sending chunk 8388608 length 8388608
2021/08/18 17:48:49 DEBUG : ib6db9lf0ui5292i2pdsced82s3c5cjkccplcge8insqds9othjg: Sending chunk 16777216 length 8388608
2021/08/18 17:48:49 DEBUG : ib6db9lf0ui5292i2pdsced82s3c5cjkccplcge8insqds9othjg: Sending chunk 25165824 length 8388608
2021/08/18 17:48:49 DEBUG : ib6db9lf0ui5292i2pdsced82s3c5cjkccplcge8insqds9othjg: Sending chunk 33554432 length 8388608
2021/08/18 17:48:50 DEBUG : ib6db9lf0ui5292i2pdsced82s3c5cjkccplcge8insqds9othjg: Sending chunk 41943040 length 8388608
2021/08/18 17:48:50 DEBUG : ib6db9lf0ui5292i2pdsced82s3c5cjkccplcge8insqds9othjg: Sending chunk 50331648 length 8388608
2021/08/18 17:48:50 DEBUG : ib6db9lf0ui5292i2pdsced82s3c5cjkccplcge8insqds9othjg: Sending chunk 58720256 length 8388608
2021/08/18 17:48:50 DEBUG : ib6db9lf0ui5292i2pdsced82s3c5cjkccplcge8insqds9othjg: Sending chunk 67108864 length 8388608
2021/08/18 17:48:51 DEBUG : ib6db9lf0ui5292i2pdsced82s3c5cjkccplcge8insqds9othjg: Sending chunk 75497472 length 8388608
2021/08/18 17:48:51 DEBUG : ib6db9lf0ui5292i2pdsced82s3c5cjkccplcge8insqds9othjg: Sending chunk 83886080 length 8388608
2021/08/18 17:48:51 DEBUG : ib6db9lf0ui5292i2pdsced82s3c5cjkccplcge8insqds9othjg: Sending chunk 92274688 length 8388608
2021/08/18 17:48:52 DEBUG : ib6db9lf0ui5292i2pdsced82s3c5cjkccplcge8insqds9othjg: Sending chunk 100663296 length 4219936
2021/08/18 17:48:53 DEBUG : 200mbtestfile1: md5 = 7cd6c2f0907c202b0f481d1fd25a2352 OK
2021/08/18 17:48:53 DEBUG : 736vcs4nocn0om165f9bq8h0j3fqqds1huqj2jgvep1bbdr2n4hg: Sending chunk 0 length 8388608
2021/08/18 17:48:53 DEBUG : 736vcs4nocn0om165f9bq8h0j3fqqds1huqj2jgvep1bbdr2n4hg: Sending chunk 8388608 length 8388608
2021/08/18 17:48:54 DEBUG : 736vcs4nocn0om165f9bq8h0j3fqqds1huqj2jgvep1bbdr2n4hg: Sending chunk 16777216 length 8388608
2021/08/18 17:48:54 DEBUG : 736vcs4nocn0om165f9bq8h0j3fqqds1huqj2jgvep1bbdr2n4hg: Sending chunk 25165824 length 8388608
2021/08/18 17:48:54 DEBUG : 736vcs4nocn0om165f9bq8h0j3fqqds1huqj2jgvep1bbdr2n4hg: Sending chunk 33554432 length 8388608
2021/08/18 17:48:55 DEBUG : 736vcs4nocn0om165f9bq8h0j3fqqds1huqj2jgvep1bbdr2n4hg: Sending chunk 41943040 length 8388608
2021/08/18 17:48:55 DEBUG : 736vcs4nocn0om165f9bq8h0j3fqqds1huqj2jgvep1bbdr2n4hg: Sending chunk 50331648 length 8388608
2021/08/18 17:48:55 DEBUG : 736vcs4nocn0om165f9bq8h0j3fqqds1huqj2jgvep1bbdr2n4hg: Sending chunk 58720256 length 8388608
2021/08/18 17:48:56 DEBUG : 736vcs4nocn0om165f9bq8h0j3fqqds1huqj2jgvep1bbdr2n4hg: Sending chunk 67108864 length 8388608
2021/08/18 17:48:56 DEBUG : 736vcs4nocn0om165f9bq8h0j3fqqds1huqj2jgvep1bbdr2n4hg: Sending chunk 75497472 length 8388608
2021/08/18 17:48:56 DEBUG : 736vcs4nocn0om165f9bq8h0j3fqqds1huqj2jgvep1bbdr2n4hg: Sending chunk 83886080 length 8388608
2021/08/18 17:48:57 DEBUG : 736vcs4nocn0om165f9bq8h0j3fqqds1huqj2jgvep1bbdr2n4hg: Sending chunk 92274688 length 8388608
2021/08/18 17:48:57 DEBUG : 736vcs4nocn0om165f9bq8h0j3fqqds1huqj2jgvep1bbdr2n4hg: Sending chunk 100663296 length 4219936
2021/08/18 17:48:58 DEBUG : 200mbtestfile1: md5 = 92209945fa847286140c53aa37bcfddc OK
2021/08/18 17:48:58 INFO : 200mbtestfile1.rcc001_1w1w9x: Moved (server-side) to: 200mbtestfile1.rcc001
2021/08/18 17:48:59 INFO : 200mbtestfile1.rcc002_1w1w9x: Moved (server-side) to: 200mbtestfile1.rcc002
2021/08/18 17:49:00 DEBUG : 200mbtestfile1: md5 = 3be305ed7065461d7eca471153ab4188 OK
2021/08/18 17:49:00 INFO : 200mbtestfile1: Copied (new)
2021/08/18 17:49:00 INFO :
Transferred: 200.049Mi / 200.049 MiByte, 100%, 15.391 MiByte/s, ETA 0s
Checks: 2 / 2, 100%
Renamed: 2
Transferred: 1 / 1, 100%
Elapsed time: 14.3s
2021/08/18 17:49:00 DEBUG : 6 go routines active
In the end, the files are uploaded to gdrive with the same chunked file names and the file contents are encrypted/not encrypted as they should be.
root@docker:~/rclone-v1.57.0-beta.5609.22ad1e3be.fix-5498-crypt-no-encrypt-linux-amd64# rclone lsl crypt:1
104857600 2021-08-18 16:39:08.280000000 200mbtestfile1.rcc002
79 2021-08-18 16:39:08.280000000 200mbtestfile1
104857600 2021-08-18 16:39:08.280000000 200mbtestfile1.rcc001
root@docker:~/rclone-v1.57.0-beta.5609.22ad1e3be.fix-5498-crypt-no-encrypt-linux-amd64# rclone lsl crypt:2
104883232 2021-08-18 16:39:08.280000000 200mbtestfile1.rcc001
104883232 2021-08-18 16:39:08.280000000 200mbtestfile1.rcc002
127 2021-08-18 16:39:08.280000000 200mbtestfile1