Rclone encrypted back cache issue

Suspected Bug
1

What is the problem you are having with rclone?

When accessing remote via encrypted back-end, it doesn't see actual files status on the server:

$ rclone lsd encrypted_backend_s3:
$ rclone lsd my_s3:my-files
           0 2021-09-21 13:42:26        -1 test

What I did:

  1. created my_s3 endpoint
  2. on top of it, created encrypted_backend endpoint
  3. tested sync command
  4. remove all of the files from the remote S3 manually
  5. tried to re-sync, but rclone consider that files hasn't changed
  6. I tried to lsd on the encryption endpoint, and it shows me nothing
  7. Same command on the same endpoint,but not encrypted - shows the test folder, that I created manually.

P.S. While collecting the information, I noticed a folder matching remote side name ~/.config/rclone - when I deleted it and re-run sync - it's told that it's uploading something this time, but nothing appeared on the S3 - so it's false upload message.
Feels like something is quite broken if using encrypt backend =(

What is your rclone version (output from rclone version)

Both 1.55.1 & 1.56.1 (I had 1.55 but updated on the latest one to check on things).

Successfully updated rclone from version v1.55.1 to version v1.56.1

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Linux 64bit

Which cloud storage system are you using? (eg Google Drive)

MinIO

The command you were trying to run (eg rclone copy /tmp remote:tmp)

I tried both --cache-db-purge --cache-mode off, but it didn't work out.

The rclone config contents with secrets removed.


[local_s3]
type = s3
provider = Minio
env_auth = false
endpoint = http://127.0.0.1:<...>
region = us-east-1


[my_s3]
type = s3
disable_http2 = true
provider = Minio
access_key_id = <...>
secret_access_key = <...>
endpoint = https://<...>

[encrypted_backend_s3]
type = crypt
remote = my_s3:my-files
filename_encryption = obfuscate
directory_name_encryption = true
password = <...>
password2 = <...>

A log from the command with the -vv flag

$ rclone sync local_s3:my-uploads encrypted_backend_s3 -vv
2021/09/21 13:51:13 DEBUG : rclone: Version "v1.56.1" starting with parameters ["rclone" "sync" "local_s3:my-uploads" "encrypted_backend_s3" "-vv"]
2021/09/21 13:51:13 DEBUG : Creating backend with remote "local_s3:my-uploads"
2021/09/21 13:51:13 DEBUG : Using config file from "/home/user/.config/rclone/rclone.conf"
2021/09/21 13:51:13 DEBUG : Creating backend with remote "encrypted_backend_s3"
2021/09/21 13:51:13 DEBUG : fs cache: renaming cache item "encrypted_backend_s3" to be canonical "/home/user/.config/rclone/encrypted_backend_s3"
2021/09/21 13:51:13 DEBUG : Screenshot 2021-05-07 at 17.10.49.png: Size and modification time the same (differ by 0s, within tolerance 1ns)
2021/09/21 13:51:13 DEBUG : Screenshot 2021-05-07 at 17.10.49.png: Unchanged skipping
2021/09/21 13:51:13 DEBUG : Local file system at /home/user/.config/rclone/encrypted_backend_s3: Waiting for checks to finish
2021/09/21 13:51:13 DEBUG : uploaded_files/10/CS-Fi_Agreement_signed1.pdf: Size and modification time the same (differ by 0s, within tolerance 1ns)
2021/09/21 13:51:13 DEBUG : uploaded_files/10/CS-Fi_Agreement_signed1.pdf: Unchanged skipping
<...>
Many other entries like that
<...>
2021/09/21 13:51:13 DEBUG : Local file system at /home/user/.config/rclone/encrypted_backend_s3: Waiting for transfers to finish
2021/09/21 13:51:13 DEBUG : Waiting for deletions to finish
2021/09/21 13:51:13 INFO  : There was nothing to transfer
2021/09/21 13:51:13 INFO  : 
Transferred:              0 / 0 Byte, -, 0 Byte/s, ETA -
Checks:                16 / 16, 100%
Elapsed time:         0.0s
2

I can't figure out what you are asking as it might be too early for me :slight_smile:

That means the file is already on the remote and the source and destination match so it does nothing.

If you rclone ls that file on the remote, are you not seeing it? Is that what you mean?

That's for the cache remote which you are not using so it wouldn't do anything.

I don't see where you are using any cache mode to turn on / off as that's for a mount and you are using a sync command.

3

Yes. The problem is, that the remote is absolutely empty, like zero files, zero folders. So I'm getting false uploads.

Please, let me know if anything else is required... except for the coffee probably, I would guess it would arrive cold, if I try to deliver it to you, anyway =)

4

Run:

rclone ls encrypted_backend_s3:uploaded_files/10/CS-Fi_Agreement_signed1.pdf -vv

And share the output.

5

You are missing a : after encrypted_backend_s3 and so it's treating it like a local directory.

2 Likes
6

And that's coffee time!

1 Like
7

Thanks, that was just a comma indeed!

8

Have a good one! :slight_smile:

9

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.