--crypt-server-side-across configs is reencrypting. Plus additional question

Suspected Bug
1

What is the problem you are having with rclone?

**What I am really trying to do **: I am backing up a crypt remote to another with identical passwords but (a) I need to use filters and (b) I don't want it to rencrypt (breaks some later deduplication). Trying to see if --crypt-server-side-across-configs would work for me but in testing, I hit what I think is a bug (or user error)

What I am talking about here: Simple test to play and understand. I created a two crypt remotes with the same password and tried to copy with --crypt-server-side-across-configs. It does not seem to work.

Run the command 'rclone version' and share the full output of the command.

rclone v1.62.2
- os/version: darwin 13.4 (64 bit)
- os/kernel: 22.5.0 (arm64)
- os/type: darwin
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.20.2
- go/linking: dynamic
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

crypt with local and webdav to test

The command you were trying to run (eg rclone copy /tmp remote:tmp)

I've tried a few different permutations but

rclone copy cA: cB: --crypt-server-side-across-configs -vv

was my main test but then I thought that maybe it didn't like that I didn't change name encryption type so I did it again.

rclone copy cA: cB: --crypt-server-side-across-configs -vv

I tested both with local and then again webDAV. I thought that maybe an issue with local and server-side copy. (the WebDAV server is also rclone but that is not important)

Finally, I had the thought that maybe it was that I was doing a copy and not a move. The documentation doesn't say anything about that but it talks about name changes.

rclone move cA: cB: --crypt-server-side-across-configs -vv
rclone move cA: cC: --crypt-server-side-across-configs -vv

The rclone config contents with secrets removed.

[base]
type = webdav
url = http://127.0.0.1:8080
vendor = other

[base1]
type = alias
remote = /Users/***/rclone_play/

[cA]
type = crypt
remote = base:A
password = o-z8EnU6EoXfxtPtQGJbReVYnQ
password2 = mEkC41K1u3NjXGyHOuXVLd96LA

[cB]
type = crypt
remote = base:B
password = o-z8EnU6EoXfxtPtQGJbReVYnQ
password2 = mEkC41K1u3NjXGyHOuXVLd96LA

[cC]
type = crypt
remote = base:C
filename_encryption = off
directory_name_encryption = false
password = o-z8EnU6EoXfxtPtQGJbReVYnQ
password2 = mEkC41K1u3NjXGyHOuXVLd96LA

A log from the command with the -vv flag

2023/06/10 11:05:28 DEBUG : Setting --config "/Users/***/rclone_play/config.cfg" from environment variable RCLONE_CONFIG="/Users/***/rclone_play/config.cfg"
2023/06/10 11:05:28 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "copy" "cA:" "cC:" "--crypt-server-side-across-configs" "-vv"]
2023/06/10 11:05:28 DEBUG : Creating backend with remote "cA:"
2023/06/10 11:05:28 DEBUG : Using config file from "/Users/***/rclone_play/config.cfg"
2023/06/10 11:05:28 DEBUG : cA: detected overridden config - adding "{Db_Y9}" suffix to name
2023/06/10 11:05:28 DEBUG : Creating backend with remote "base:A"
2023/06/10 11:05:28 DEBUG : found headers:
2023/06/10 11:05:28 DEBUG : fs cache: renaming cache item "cA:" to be canonical "cA{Db_Y9}:"
2023/06/10 11:05:28 DEBUG : Creating backend with remote "cC:"
2023/06/10 11:05:28 DEBUG : cC: detected overridden config - adding "{Db_Y9}" suffix to name
2023/06/10 11:05:28 DEBUG : Creating backend with remote "base:C"
2023/06/10 11:05:28 DEBUG : found headers:
2023/06/10 11:05:28 DEBUG : fs cache: renaming cache item "cC:" to be canonical "cC{Db_Y9}:"
2023/06/10 11:05:28 DEBUG : Encrypted drive 'cC{Db_Y9}:': Waiting for checks to finish
2023/06/10 11:05:28 DEBUG : testfile.txt: Sizes identical
2023/06/10 11:05:28 DEBUG : testfile.txt: Unchanged skipping
2023/06/10 11:05:28 DEBUG : Encrypted drive 'cC{Db_Y9}:': Waiting for transfers to finish
2023/06/10 11:05:28 INFO  : There was nothing to transfer
2023/06/10 11:05:28 INFO  :
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         0.1s

2023/06/10 11:05:28 DEBUG : 7 go routines active

Am I missing something? I think I am using this properly but I could be wrong

Additional Question (or Feature Request)

It would be great, once this is figured out, if --crypt-server-side-across-configs worked across any two identical-passwords crypt remotes. So you could do a backup with filtering and not re-encrypt.

2

I have never investigated it in details but for exactly similar purpose I use --server-side-across-configs and it works. So I never even looked at --crypt-server-side-across-configs. You can quickly try.

I have a feeling that something is not entirely clear with documentation around these flags.

3

Good catch!

--server-side-across-configs whether or not also paired with --crypt-server-side-across-configs seems to work but not for local-to-local (counter to expectations) and not webdav to local or local to webdav (I didn't think this would have worked but really wish it would).

Summary:

  • Can someone clarify --crypt-server-side-across-configs vs --server-side-across-configs
  • Is it a bug that --crypt-server-side-across-configs and/or --server-side-across-configs does not work local-to-local?
  • Any thoughts on feature request of identically encrypted crypt across remotes not re-encrypting

Side note since I didn't include it: when you reencrypt, the nonce changes which means everything changes. This breaks any kind of deduplication of the crypted file. I suspect this is an edge case concern however. There is the performance hit of reencrypting but that is secondary.

edits to clarify parentheticals

4

Agree with your questions - just never bothered to dig deeper - I still suspect some docs legacy issues. For this every time I catch some problem I post PR. But this one requires more understanding of internals for me to try to correct.

Sometimes I just take practical approach. A does not work but B does - forget about A.

5

I agree with counter to expectations but this at least is documented. Local does not support copy which two paragraphs later is made clear that it means server side feature

6

And I think cancel my post marking as solution - or this thread will close in 3 days. I think that we should get to the bottom of this issue and finally make it clear.

Maybe it is real bug, maybe docs - for sure something does not adds up now.

7

That makes sense. Iā€™m no longer at the computer but I should test with move.

I canceled the solution though it did solve my problem

1 Like
8

It does with with move.

For my uses, local-to-local copy would have been fine but since I can't do it, I may just have to trick rclone into doing my bidding by having it serve via WebDAV and doing it. (I do lose mtime however. But that is related to a different feature request!)

1 Like
9

yeah - but lets try to clarify/fix --crypt-server-side-across-configs vs --server-side-across-configs. It will be one less issue when done

10

--crypt-server-side-across-configs just works for the crypt backend.

--server-side-across-configs works for all backends. I'd use this one and forget the crypt one. We invented the backend specific flags first but it became clear we could do this generically for all backends and made this flag.

Local can't do server side copies.

$ rclone backend features / | grep Copy
		"Copy": false,

Copy the underlying remotes is recommended here, however this won't work with filters.

11

Would it be ok to remove it from docs (I am happy to do PR) and just leave --server-side-across-configs? What it is now is a wee confusing.

12

It is possible to remove commands from the docs and/or the help strings using

Hide:     fs.OptionHideCommandLine, // stops the option appearing in the flags (it still works though)
Hide:     fs.OptionHideConfigurator, // stops the option appearing in rclone config
Hide:    fs.OptionHideBoth, // does both

In the option definition in the relevant backend file.

However I'm not sure any of those remove the actual docs from the doc file.

The best approach might be to note in the help for --crypt-server-side-across-configs and --drive-server-side-across-configs that these are deprecated and the user should be using --server-side-across-configs instead. (Note that the source of this help is in the backends/crypt/crypt.go and backends/drive/drive.go files - everything else is auto generated).

1 Like