Advice: Using different paths with an encrypted remote for compare-dest or backup-dir

What is the problem you are having with rclone?

I need to make an rclone sync to a remote, and use the --compare-dest or --backup-dir flags. I want the sync to be encrypted with file name encryption, so I need to use a crypt backend. However, the path used by the crypt backend needs to be different for --compare-dest (or --backup-dir) to that the sync needs to go to. In effect, the prefix of each path is unencrypted.

For example:-


My backend remote might be a sftp one. My encrypted remotes need to NOT encrypt backups/full and backups/differential.

I've tried defining different remotes, either crypt or alias, and this isn't allowed (Failed to sync: parameter to --copy-dest has to be on the same remote as destination) - the check is overly broad (func SameConfigArr() just checks names match).

I'm out of ideas.

What is your rclone version (output from rclone version)

rclone v1.55.1
- os/type: darwin
- os/arch: amd64
- go/version: go1.16.3
- go/linking: dynamic
- go/tags: cmount

Which OS you are using and how many bits (eg Windows 7, 64 bit)

macOS Mojave 10.4.4 on a Mac Pro trashcan.

Which cloud storage system are you using? (eg Google Drive)

  • local

Can you paste an example command line that you tried? Try --backup-dir first as that is a bit simpler (conceptually) to use.

That would be challenging - I've substantially changed my script since I posted. My specific failure occurred when defining two remotes, say, --compare-dest differential: /some/source/path full: where remotes were generated on the fly and appended (by shell script) to a local conf file*, something like

Attempt 1

; The data below is appended at runtime by my script
type = alias
remote = encrypted:backups/differential

type = alias
remote = encrypted:backups/full

Attempt 2

type = local; or anything else

; The data below is appended at runtime by my script
type = crypt
remote = unecrypted:backups/differential

type = crypt
remote = unecrypted:backups/full

The fundamental limit is the check in func SameConfigArr(), which wants names to match, rather than doing a deeper structural comparison.

Aside: I have now integrated my crypt backends using Keychain Access on OS X, but it's painful. If my script ever works, I'll share it.

*Aside: multiple conf files would make life so much easier, as would using a line based format friendly to POSIX rather than INI and being able to define multiple remotes on the command line. rclone's a superb tool, with an outstanding reputation, which is why I wanted to try it, but it's not a script friendly as rsync. I'm on the fence with it; not supporting hard links and symbolic links is a major need that's missing for proper file system backups and replacing rsync for local copies. It's winning feature is encryption (although it doesn't seem to have any way to ease the rotation of encryption keys).

Yes. The deeper structural comparison is hard though. It would have to be implemented by each backend probably as they are the only ones that know enough about their config file.

Maybe on the rclone wiki?

Did you check out the connection string way of defining a remote? That ticks all the boxes above I think.

Rclone does support symlinks. It does also support hardlinks as it will copy the files just fine, it just duplicates the storage! I don't know of many cloud storage backends which have the concepts of hardlinks. Google drive used to, but they took that away recently.

You have to re-encrypt the data if you want to change the key. Rclone doesn't have the idea of user keys which unlock a master key so that you can rotate the user keys.

It maybe should... However if a user key has been compromised then the master key may have been compromised too.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.