Not encrypting or compressing before upload

Help and Support
1

What is the problem you are having with rclone?

I created remote Cloudflare:bucket. Then I created a crypt remote for Cloudflare:bucket. I also created a compress remote for Cloudflare:bucket. I am able to copy to Cloudflare:bucket just fine, but the files are not being compressed or encrypted. I even tried deleting the compress remote and leaving just the crypt remote. No avail.

Run the command 'rclone version' and share the full output of the command.

rclone v1.59.0
- os/version: Microsoft Windows 10 Pro 21H2 (64 bit)
- os/kernel: 10.0.19044.1865 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.18.3
- go/linking: static
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Cloudflare R2

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone --config=/path/to/rclone.conf copy --verbose /path/to/local/folder/1.PNG Cloudflare:bucket

The rclone config contents with secrets removed.

[Cloudflare]
type = s3
provider = Cloudflare
access_key_id = *redacted*
secret_access_key = *redacted*
region = auto
endpoint = *redacted*

[EncryptCloudflare]
type = crypt
remote = Cloudflare:bucket
password = *redacted*
password2 = *redacted*

[CompressCloudflare]
type = compress
remote = Cloudflare:bucket

A log from the command with the -vv flag

2022/08/04 20:13:32 DEBUG : rclone: Version "v1.59.0" starting with parameters ["path\\to\\rclone.exe" "--config=path\\to\\rclone.conf" "copy" "-vv" "path\\to\\1.PNG" "Cloudflare:bucket"]
2022/08/04 20:13:32 DEBUG : Creating backend with remote "path\\to\\1.PNG"
2022/08/04 20:13:32 DEBUG : Using config file from "path\\to\\rclone.conf"
2022/08/04 20:13:32 DEBUG : fs cache: adding new entry for parent of "path\\to\\1.PNG", "//?/path/to/folder"
2022/08/04 20:13:32 DEBUG : Creating backend with remote "Cloudflare:bucket"
2022/08/04 20:13:32 DEBUG : 1.PNG: Need to transfer - File not found at Destination
2022/08/04 20:13:34 DEBUG : 1.PNG: md5 = *redacted* OK
2022/08/04 20:13:34 INFO  : 1.PNG: Copied (new)
2022/08/04 20:13:34 INFO  :
Transferred:      147.395 KiB / 147.395 KiB, 100%, 145.918 KiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         1.9s

2022/08/04 20:13:34 DEBUG : 5 go routines active
2

hi,

if you want the files to be crypted, need to copy to EncryptCloudflare:
rclone copy /path/to/local/folder/1.PNG EncryptCloudflare:

if you want the files to be compressed, need to copy to CompressCloudflare:
rclone copy /path/to/local/folder/1.PNG CompressCloudflare:

note: about remote = Cloudflare:bucket
you are using that in two remotes, not sure that is good idea.

3

Oh. My bad.

So it is not possible to compress and encrypt both? Could I point CompressCloudflare to EncryptCloudflare to Cloudflare:bucket?

4

yes, you can do that with rclone.

not sure which is better
CompressCloudflare -> EncryptCloudflare -> Cloudflare:bucket
or
EncryptCloudflare -> CompressCloudflare -> Cloudflare:bucket

5

Thanks.

I just tested it and Cloudflare:bucket -> EncryptCloudflare -> CompressCloudflare does not work. Cloudflare:bucket -> CompressCloudflare -> EncryptCloudflare works.

6

I take that back. Neither works. Cloudflare:bucket -> EncryptCloudflare -> CompressCloudflare throws an MD5 error and does not upload anything. Cloudflare:bucket -> CompressCloudflare -> EncryptCloudflare uploads the file just fine, but it looks like the file is encrypted but not compressed.

Oh well.

7

I would say this is the correct order.

What is the error?

8

can you run a command that copies a single file, that will generate the exact error?
add -vv to the command for debug output and post the full debug log

9

I will have to set it up again to get the exact error/log, but the error was basically saying that the MD5 hash of the upload did not match the source file. It tried the upload three times and then failed.

10

You'd want to share your rclone.conf that you adjusted and the full command with a debug log.

11

Here is the error log:

C:\Users\[name]>"rclone.exe" --config="rclone.conf" copy -vv "1.PNG" CompressCloudflare:
2022/08/05 23:21:39 DEBUG : rclone: Version "v1.59.0" starting with parameters ["rclone.exe" "--config=rclone.conf" "copy" "-vv" "1.PNG" "CompressCloudflare:"]
2022/08/05 23:21:39 DEBUG : Creating backend with remote "1.PNG"
2022/08/05 23:21:39 DEBUG : Using config file from "rclone.conf"
2022/08/05 23:21:39 DEBUG : fs cache: adding new entry for parent of "1.PNG", "//?/C:/Users/..."
2022/08/05 23:21:39 DEBUG : Creating backend with remote "CompressCloudflare:"
2022/08/05 23:21:39 DEBUG : Creating backend with remote "Cloudflare:bucket/string"
2022/08/05 23:21:39 DEBUG : Creating backend with remote "Cloudflare:bucket"
2022/08/05 23:21:40 DEBUG : 1.PNG: Need to transfer - File not found at Destination
2022/08/05 23:21:41 DEBUG : 1.PNG.lE0CAAAAAAA.gz: md5 = e5f97b00b2ec77694d2c1b3a55c58658 OK
2022/08/05 23:21:41 DEBUG : 1.PNG: Computing md5 hash of encrypted source
2022/08/05 23:21:42 ERROR : 1.PNG: Failed to copy: BadDigest: The Content-MD5 you specified did not match what we received.
        status code: 400, request id: , host id:
2022/08/05 23:21:42 ERROR : Attempt 1/3 failed with 1 errors and: BadDigest: The Content-MD5 you specified did not match what we received.
        status code: 400, request id: , host id:
2022/08/05 23:21:42 DEBUG : 1.PNG: Need to transfer - File not found at Destination
2022/08/05 23:21:42 DEBUG : 1.PNG.lE0CAAAAAAA.gz: md5 = 292ebc8c1e5ca00fd1e79e0812070ca3 OK
2022/08/05 23:21:42 DEBUG : 1.PNG: Computing md5 hash of encrypted source
2022/08/05 23:21:43 ERROR : 1.PNG: Failed to copy: BadDigest: The Content-MD5 you specified did not match what we received.
        status code: 400, request id: , host id:
2022/08/05 23:21:43 ERROR : Attempt 2/3 failed with 1 errors and: BadDigest: The Content-MD5 you specified did not match what we received.
        status code: 400, request id: , host id:
2022/08/05 23:21:43 DEBUG : 1.PNG: Need to transfer - File not found at Destination
2022/08/05 23:21:43 DEBUG : 1.PNG.lE0CAAAAAAA.gz: md5 = 0ef7e272fd0c429abffe28dad89244ea OK
2022/08/05 23:21:43 DEBUG : 1.PNG: Computing md5 hash of encrypted source
2022/08/05 23:21:44 ERROR : 1.PNG: Failed to copy: BadDigest: The Content-MD5 you specified did not match what we received.
        status code: 400, request id: , host id:
2022/08/05 23:21:44 ERROR : Attempt 3/3 failed with 1 errors and: BadDigest: The Content-MD5 you specified did not match what we received.
        status code: 400, request id: , host id:
2022/08/05 23:21:44 INFO  :
Transferred:      393.721 KiB / 393.721 KiB, 100%, 86.905 KiB/s, ETA 0s
Errors:                 1 (retrying may help)
Elapsed time:         4.9s

2022/08/05 23:21:44 DEBUG : 7 go routines active
2022/08/05 23:21:44 Failed to copy: BadDigest: The Content-MD5 you specified did not match what we received.
        status code: 400, request id: , host id:

Here is my rclone.conf

[Cloudflare]
type = s3
provider = Cloudflare
access_key_id = *redacted*
secret_access_key = *redacted*
region = auto
endpoint = *redacted*

[EncryptCloudflare]
type = crypt
remote = Cloudflare:bucket
password = *redacted*
password2 = *redacted*

[CompressCloudflare]
type = compress
remote = EncryptCloudflare:
13

That turned out to be a bug which I've had a go at fixing here if you want to give it a test.

v1.60.0-beta.6409.699b49456.fix-crypt-unwrap on branch fix-crypt-unwrap (uploaded in 15-30 mins)

14

Unfortunately I am not in a position to test a beta version. However, I will for sure give this a go when v1.60 is released! Thank you for tackling this so quickly. Amazing.

1 Like
15

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.