Some crypt remotes have leading ./, sync causes re-uploads

What is the problem you are having with rclone?

I have been backing up files with rclone for a couple of years. In the spring 2022 I switched to using Amazon AWS S3 Deep Glacier to save costs. This worked until now when I noticed that rclone tried to re-upload files I knew for a fact were already backed up. I don't know what caused this change or when.

I am using crypt to encode the contents and filenames. When I run either rclone ls or rclone sync I can see that for the files rclone is trying to re-upload the remote file has a leading ./ and local file does not. An example from rclone sync --dry-run output is below. So it seems that rclone treats these as different files due to difference in path and tries to upload the file without ./ and tries to delete the file with ./.

...
2022-08-13 10:47:10 NOTICE: Some Directory/Example File.flac: Skipped copy as --dry-run is set (size 36.620Mi)
2022-08-13 10:47:16 NOTICE: ./Some Directory/Example File.flac: Skipped delete as --dry-run is set (size 36.620Mi)
...

I have multiple remotes configured. All of them use S3 and crypt but different buckets. For simplicity's sake I only included two example remotes: s3_crypt_a which has the re-upload problem and s3_crypt_b which works fine. I noticed that the affected remote (s3_crypt_a) has configuration as shown below whereas the working remote (s3_crypt_b) does not have these explicitly stated.

filename_encryption = standard
directory_name_encryption = true

However according to documentation of Crypt (rclone.org) these should be the default values so I don't know if that actually matters.

I can delete the previous files and re-upload them without ./ but as Deep Glacier has early deletion fee and re-uploading also costs money so I'd really like to understand what happened and how I could avoid it in the future.

Run the command 'rclone version' and share the full output of the command.

rclone v1.58.0
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.10.102.1-microsoft-standard-WSL2 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.17.8
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Amazon AWS S3 Deep Glacier

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Sync commands:

rclone sync --immutable --size-only --progress --transfers 32 "/mnt/c/A_Source/" s3_crypt_a:.
rclone sync --immutable --size-only --progress --transfers 32 "/mnt/c/B_Source/" s3_crypt_b:.

ls commands:

rclone ls --fast-list s3_crypt_a:
rclone ls --fast-list s3_crypt_b:

The rclone config contents with secrets removed.

[s3]
type = s3
provider = AWS
env_auth = false
access_key_id = !!OMITTED!!
secret_access_key = !!OMITTED!!
region = eu-north-1
location_constraint = eu-north-1
acl = private
storage_class = DEEP_ARCHIVE
bucket_acl = private
upload_cutoff = 5G
chunk_size = 2G
upload_concurrency = 8

[s3_crypt_a]
type = crypt
remote = s3:a
filename_encryption = standard
directory_name_encryption = true
password = !!OMITTED!!
password2 = !!OMITTED!!

[s3_crypt_b]
type = crypt
remote = s3:b
password = !!OMITTED!!
password2 = !!OMITTED!!

A log from the command with the -vv flag

...
2022-08-13 10:47:10 NOTICE: Some Directory/Example File.flac: Skipped copy as --dry-run is set (size 36.620Mi)
2022-08-13 10:47:16 NOTICE: ./Some Directory/Example File.flac: Skipped delete as --dry-run is set (size 36.620Mi)
...

I think getting rid of the . at the end of s3_crypt_a:. will fix the problem.

Pathname normalisation is tricky across different types of backend.

Unfortunately that doesn't seem to fix the problem for me. rclone sync --dry-run --immutable --size-only --progress --transfers 32 "/mnt/c/A_Source/" s3_crypt_a: still indicates that everything would be re-uploaded without ./ and the existing files with leading ./ would be deleted.

Hmm...

You could do a sync without --immutable and use --track-renames to sort the mess out. That should rename the files without their initial ./ - something like this

rclone sync --dry-run --track-renames --checksum --progress --transfers 32 "/mnt/c/A_Source/" s3_crypt_a:

I think rclone uploading them with an initial ./ in the past was a bug that got fixed at some point.

That could work for some cases but --track-renames does not work with crypt remote and renaming target files with AWS Glacier Deep Archive is not possible without copying and deleting the files.

As this data set is only less than 2 terabytes I think I will save the trouble for everyone and just delete the existing files and re-upload them. Your theory about this being caused by a bug which has since been fixed sounds plausible so I think I won't be running into this problem again.

Thank you for the help and especially for the amazing tool! Cheers!

1 Like

Sorry for your trouble here. Doing a re-upload sound like the most sensible solution since --track-renames won't work.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.