Rclone sync behaviour with encryption

Help and Support
1

What is the problem you are having with rclone?

I am trying to get rclone sync working with encryption for the 'Mega S4' s3 service.

With no encryption & no special flags files are re-uploaded regardless of whether they have been changed. This appears to be an issue with 'Modification times differ' with the Mega service.

However, without encryption, by using --s3-use-multipart-uploads false I was able to prevent unchanged files being re-uploaded.

I am now trying to 'fix' the behaviour for encrypted files and am seeing a couple of things I don't understand:

  1. the log says that MD5 sums are being calculated but file uploads start immediately, apparently not waiting to see if the locally calculated MD5 sums match those reported by the destination.

  2. Each time I run the rclone sync command the log file reports different md5 sums.

These are just questions that have arisen during my troubleshooting, but if any one knows how to achieve my goal that would of course be appreciated!

Run the command 'rclone version'

rclone v1.69.1

  • os/version: ubuntu 24.10 (64 bit)
  • os/kernel: 6.11.0-14-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.24.0
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using?

Mega S4

The command you were trying to run

rclone --config ./config_file -vv --metadata --s3-use-multipart-uploads false --transfers 4 --bwlimit 00:00,2621440b sync /home/felix/rclone_test/testfiles encrypted:/

Please run 'rclone config redacted' and share the full output.

[encrypted]
type = crypt
remote = remote:rclone-testing/
filename_encryption = standard
password = XXX
password2 = XXX

[remote]
access_key_id = XXX
secret_access_key = XXX
endpoint = s3.eu-central-1.s4.mega.io
region = 
max_upload_parts = 10000
type = s3
provider = Other
folder = /
bucket = rclone-testing
fast_list = false
server_side_encryption =

One log from the command that you were trying to run

felix@um690x:~/rclone_test$ ./runit 
2025/02/16 16:58:25 INFO  : Starting bandwidth limiter at 2.500Mi Byte/s
2025/02/16 16:58:25 DEBUG : rclone: Version "v1.69.1" starting with parameters ["rclone" "--config" "./config_file" "-vv" "--metadata" "--s3-use-multipart-uploads" "false" "--transfers" "4" "--bwlimit" "00:00,2621440b" "sync" "/home/felix/rclone_test/testfiles" "encrypted:/"]
2025/02/16 16:58:25 DEBUG : Creating backend with remote "/home/felix/rclone_test/testfiles"
2025/02/16 16:58:25 DEBUG : Using config file from "/home/felix/rclone_test/config_file"
2025/02/16 16:58:25 DEBUG : Creating backend with remote "encrypted:/"
2025/02/16 16:58:26 DEBUG : Creating backend with remote "remote:rclone-testing"
2025/02/16 16:58:26 DEBUG : remote: detected overridden config - adding "{mehnf}" suffix to name
2025/02/16 16:58:26 DEBUG : S3 bucket rclone-testing: Disabling multipart uploads
2025/02/16 16:58:26 DEBUG : fs cache: renaming cache item "remote:rclone-testing" to be canonical "remote{mehnf}:rclone-testing"
2025/02/16 16:58:26 DEBUG : fs cache: switching user supplied name "remote:rclone-testing" for canonical name "remote{mehnf}:rclone-testing"
2025/02/16 16:58:26 DEBUG : Encrypted drive 'encrypted:/': Waiting for checks to finish
2025/02/16 16:58:26 DEBUG : file4: Modification times differ by 1h57m41.350474287s: 2025-02-16 14:40:46.649525713 +0000 GMT, 2025-02-16 16:38:28 +0000 UTC
2025/02/16 16:58:26 DEBUG : file4: Computing md5 hash of encrypted source
2025/02/16 16:58:26 DEBUG : file7: Modification times differ by 1h57m41.938452664s: 2025-02-16 14:40:47.061547336 +0000 GMT, 2025-02-16 16:38:29 +0000 UTC
2025/02/16 16:58:26 DEBUG : Encrypted drive 'encrypted:/': Waiting for transfers to finish
2025/02/16 16:58:26 DEBUG : file7: Computing md5 hash of encrypted source
2025/02/16 16:59:26 INFO  : 
Transferred:      116.278 MiB / 356.000 MiB, 33%, 1.925 MiB/s, ETA 2m4s
Checks:                 2 / 2, 100%
Transferred:            0 / 2, 0%
Elapsed time:       1m0.0s
Transferring:
 *                                         file4: 62% /96.000Mi, 890.856Ki/s, 41s
 *                                         file7: 21% /260Mi, 1.055Mi/s, 3m13s

2025/02/16 17:00:10 DEBUG : file4: md5 = 86b1eb9825df856b3c96eab352c0af5a OK
2025/02/16 17:00:10 INFO  : file4: Copied (replaced existing)
2025/02/16 17:00:26 INFO  : 
Transferred:      225.774 MiB / 356.023 MiB, 63%, 1.753 MiB/s, ETA 1m14s
Checks:                 2 / 2, 100%
Transferred:            1 / 2, 50%
Elapsed time:       2m0.0s
Transferring:
 *                                         file7: 49% /260Mi, 1.488Mi/s, 1m27s

2025/02/16 17:01:26 INFO  : 
Transferred:      338.770 MiB / 356.023 MiB, 95%, 1.855 MiB/s, ETA 9s
Checks:                 2 / 2, 100%
Transferred:            1 / 2, 50%
Elapsed time:       3m0.0s
Transferring:
 *                                         file7: 93% /260Mi, 1.849Mi/s, 9s

2025/02/16 17:01:36 DEBUG : file7: md5 = 44be8841d433efe5f093bb5255e170a8 OK
2025/02/16 17:01:36 INFO  : file7: Copied (replaced existing)
2025/02/16 17:01:36 DEBUG : Waiting for deletions to finish
2025/02/16 17:01:36 INFO  : 
Transferred:      356.087 MiB / 356.087 MiB, 100%, 1.738 MiB/s, ETA 0s
Checks:                 2 / 2, 100%
Transferred:            2 / 2, 100%
Elapsed time:      3m10.6s

2025/02/16 17:01:36 DEBUG : 7 go routines active

Another log - to show that calculated MD5 hashes are different

felix@um690x:~/rclone_test$ ./runit 
2025/02/16 17:04:34 INFO  : Starting bandwidth limiter at 2.500Mi Byte/s
2025/02/16 17:04:34 DEBUG : rclone: Version "v1.69.1" starting with parameters ["rclone" "--config" "./config_file" "-vv" "--metadata" "--s3-use-multipart-uploads" "false" "--transfers" "4" "--bwlimit" "00:00,2621440b" "sync" "/home/felix/rclone_test/testfiles" "encrypted:/"]
2025/02/16 17:04:34 DEBUG : Creating backend with remote "/home/felix/rclone_test/testfiles"
2025/02/16 17:04:34 DEBUG : Using config file from "/home/felix/rclone_test/config_file"
2025/02/16 17:04:34 DEBUG : Creating backend with remote "encrypted:/"
2025/02/16 17:04:34 DEBUG : Creating backend with remote "remote:rclone-testing"
2025/02/16 17:04:34 DEBUG : remote: detected overridden config - adding "{mehnf}" suffix to name
2025/02/16 17:04:34 DEBUG : S3 bucket rclone-testing: Disabling multipart uploads
2025/02/16 17:04:34 DEBUG : fs cache: renaming cache item "remote:rclone-testing" to be canonical "remote{mehnf}:rclone-testing"
2025/02/16 17:04:34 DEBUG : fs cache: switching user supplied name "remote:rclone-testing" for canonical name "remote{mehnf}:rclone-testing"
2025/02/16 17:04:34 DEBUG : Encrypted drive 'encrypted:/': Waiting for checks to finish
2025/02/16 17:04:34 DEBUG : file7: Modification times differ by 2h17m39.938452664s: 2025-02-16 14:40:47.061547336 +0000 GMT, 2025-02-16 16:58:27 +0000 UTC
2025/02/16 17:04:34 DEBUG : file7: Computing md5 hash of encrypted source
2025/02/16 17:04:34 DEBUG : file4: Modification times differ by 2h17m39.350474287s: 2025-02-16 14:40:46.649525713 +0000 GMT, 2025-02-16 16:58:26 +0000 UTC
2025/02/16 17:04:34 DEBUG : Encrypted drive 'encrypted:/': Waiting for transfers to finish
2025/02/16 17:04:34 DEBUG : file4: Computing md5 hash of encrypted source
2025/02/16 17:05:34 INFO  : 
Transferred:      116.810 MiB / 356.000 MiB, 33%, 1.963 MiB/s, ETA 2m1s
Checks:                 2 / 2, 100%
Transferred:            0 / 2, 0%
Elapsed time:       1m0.0s
Transferring:
 *                                         file7: 17% /260Mi, 884.199Ki/s, 4m7s
 *                                         file4: 73% /96.000Mi, 1.099Mi/s, 22s

2025/02/16 17:06:03 DEBUG : file4: md5 = 0ff1f27f8720a6a6c90a4401345dbdee OK
2025/02/16 17:06:03 INFO  : file4: Copied (replaced existing)
2025/02/16 17:06:34 INFO  : 
Transferred:      226.930 MiB / 356.023 MiB, 64%, 1.878 MiB/s, ETA 1m8s
Checks:                 2 / 2, 100%
Transferred:            1 / 2, 50%
Elapsed time:       2m0.0s
Transferring:
 *                                         file7: 50% /260Mi, 1.767Mi/s, 1m13s

2025/02/16 17:07:34 INFO  : 
Transferred:      340.458 MiB / 356.023 MiB, 96%, 1.794 MiB/s, ETA 8s
Checks:                 2 / 2, 100%
Transferred:            1 / 2, 50%
Elapsed time:       3m0.0s
Transferring:
 *                                         file7: 94% /260Mi, 1.792Mi/s, 8s

2025/02/16 17:07:44 DEBUG : file7: md5 = 1e9ffc976f1f633b2a6dc272bd2e97de OK
2025/02/16 17:07:44 INFO  : file7: Copied (replaced existing)
2025/02/16 17:07:44 DEBUG : Waiting for deletions to finish
2025/02/16 17:07:44 INFO  : 
Transferred:      356.087 MiB / 356.087 MiB, 100%, 1.731 MiB/s, ETA 0s
Checks:                 2 / 2, 100%
Transferred:            2 / 2, 100%
Elapsed time:      3m10.6s

2025/02/16 17:07:44 DEBUG : 7 go routines active
felix@um690x:~/rclone_test$

A log file for an unencrypted rclone sync to show that it works

felix@um690x:~/rclone_unencrypted$ ./runit
2025/02/16 14:28:47 INFO  : Starting bandwidth limiter at 2.500Mi Byte/s
2025/02/16 14:28:47 DEBUG : rclone: Version "v1.67.0" starting with parameters ["rclone" "--config" "./config_file" "-vvvvv" "--metadata" "--stats" "1s" "--s3-use-multipart-uploads" "false" "--transfers" "4" "--bwlimit" "00:00,2621440b" "--filter-from" "./filter_list" "sync" "/home/felix/rclone_unencrypted/testfiles" "remote:rclone-unencrypted" "--dest-after" "destafter" "--format" "M"]
2025/02/16 14:28:47 DEBUG : Creating backend with remote "/home/felix/rclone_unencrypted/testfiles"
2025/02/16 14:28:47 DEBUG : Using config file from "/home/felix/rclone_unencrypted/config_file"
2025/02/16 14:28:47 DEBUG : Creating backend with remote "remote:rclone-unencrypted"
2025/02/16 14:28:47 DEBUG : remote: detected overridden config - adding "{mehnf}" suffix to name
2025/02/16 14:28:47 DEBUG : Resolving service "s3" region "us-east-1"
2025/02/16 14:28:47 DEBUG : S3 bucket rclone-unencrypted: Disabling multipart uploads
2025/02/16 14:28:47 DEBUG : fs cache: renaming cache item "remote:rclone-unencrypted" to be canonical "remote{mehnf}:rclone-unencrypted"
2025/02/16 14:28:47 DEBUG : S3 bucket rclone-unencrypted: Waiting for checks to finish
2025/02/16 14:28:47 DEBUG : file4: Modification times differ by 48m39.158838764s: 2025-02-16 13:08:17.841161236 +0000 GMT, 2025-02-16 13:56:57 +0000 UTC
2025/02/16 14:28:47 DEBUG : file7: Modification times differ by 42m0.359466736s: 2025-02-16 13:34:56.640533264 +0000 GMT, 2025-02-16 14:16:57 +0000 UTC
2025/02/16 14:28:47 DEBUG : file4: md5 = b3425e8c2d9f87af23982d308d6884aa OK
2025/02/16 14:28:47 INFO  : file4: Updated modification time in destination
2025/02/16 14:28:47 DEBUG : file4: Unchanged skipping
2025/02/16 14:28:48 DEBUG : file7: md5 = 4e7824036d8967e01267416cbde2b861 OK
2025/02/16 14:28:48 INFO  : file7: Updated modification time in destination
2025/02/16 14:28:48 DEBUG : file7: Unchanged skipping
2025/02/16 14:28:48 DEBUG : S3 bucket rclone-unencrypted: Waiting for transfers to finish
2025/02/16 14:28:48 DEBUG : Waiting for deletions to finish
2025/02/16 14:28:48 INFO  : There was nothing to transfer
2025/02/16 14:28:48 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Checks:                 2 / 2, 100%
Elapsed time:         0.7s

2025/02/16 14:28:48 DEBUG : 9 go routines active
2

No point to investigate old rclone version. Sorry.

Please install the latest one and try again.

3

Version upgraded and log files above updated :slight_smile:

1 Like
4

As this is for encrypted remote - rclone encryption permute the encryption key so the same content results with different encrypted content every time you upload it. So nothing weird here. You should never see the same hash.

The rest definitely shows that there is some incompatibility with M4 Mega S3 implementation and rclone.

5

As modtime does not work what you could do is to use --checksum flag with your sync. Instead of default mtime&size used to detect changes it will use checksum&size.

6

Thank you, I did not know that.

My thinking is that the destination will be able to store and return the MD5 hash of the files in order to allow hash comparison (and avoid upload) but that this update of the encryption key would prevent that from working?

Can this update of the encryption key be disabled?

7

Crypt does not support hashes. In theory it could be done - but was never implemented - crypt: adding metadata (including hashes) to crypt files · Issue #3667 · rclone/rclone · GitHub

No.

I think in this case what has to be done is to fix mtime problem with this S3 provider (if possible). Otherwise it is close to useless really

1 Like
8

I have added checksum to my command. This now seems to rely only on the size of the file :frowning:

rclone --config ./config_file -vv --checksum --metadata --s3-use-multipart-uploads false --transfers 4 --bwlimit 00:00,2621440b sync /home/felix/rclone_test/testfiles encrypted:/

RESULT:

2025/02/16 19:28:32 INFO  : Starting bandwidth limiter at 2.500Mi Byte/s
2025/02/16 19:28:32 DEBUG : rclone: Version "v1.69.1" starting with parameters ["rclone" "--config" "./config_file" "-vv" "--checksum" "--metadata" "--s3-use-multipart-uploads" "false" "--transfers" "4" "--bwlimit" "00:00,2621440b" "sync" "/home/felix/rclone_test/testfiles" "encrypted:/"]
2025/02/16 19:28:32 DEBUG : Creating backend with remote "/home/felix/rclone_test/testfiles"
2025/02/16 19:28:32 DEBUG : Using config file from "/home/felix/rclone_test/config_file"
2025/02/16 19:28:32 DEBUG : Creating backend with remote "encrypted:/"
2025/02/16 19:28:32 DEBUG : Creating backend with remote "remote:rclone-testing"
2025/02/16 19:28:32 DEBUG : remote: detected overridden config - adding "{mehnf}" suffix to name
2025/02/16 19:28:32 DEBUG : S3 bucket rclone-testing: Disabling multipart uploads
2025/02/16 19:28:32 DEBUG : fs cache: renaming cache item "remote:rclone-testing" to be canonical "remote{mehnf}:rclone-testing"
2025/02/16 19:28:32 DEBUG : fs cache: switching user supplied name "remote:rclone-testing" for canonical name "remote{mehnf}:rclone-testing"
2025/02/16 19:28:32 NOTICE: Encrypted drive 'encrypted:/': --checksum is in use but the source and destination have no hashes in common; falling back to --size-only
2025/02/16 19:28:32 DEBUG : file4: Size of src and dst objects identical
2025/02/16 19:28:32 DEBUG : file4: Unchanged skipping
2025/02/16 19:28:32 DEBUG : file7: Size of src and dst objects identical
2025/02/16 19:28:32 DEBUG : file7: Unchanged skipping
2025/02/16 19:28:32 DEBUG : Encrypted drive 'encrypted:/': Waiting for checks to finish
2025/02/16 19:28:32 DEBUG : Encrypted drive 'encrypted:/': Waiting for transfers to finish
2025/02/16 19:28:32 DEBUG : Waiting for deletions to finish
2025/02/16 19:28:32 INFO  : There was nothing to transfer
2025/02/16 19:28:32 INFO  : 
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Checks:                 2 / 2, 100%
Elapsed time:         0.1s

2025/02/16 19:28:32 DEBUG : 6 go routines active
9

Yes. Sorry I was not clear - it only makes sense for syncing to unencrypted.

For crypt you really need mtime working.

10

Okay. Many thanks for all your help!

  1. Would this work if I stopped using file name encryption?

  2. Are you able to help me understand why the unencrypted transfer (logs in original post) works? Is that because without crypt md5 hashes for comparison are available from the destination?

11

It is irrelevant for issue you are experiencing. With or without names encryptions it will be the same.

Yes.

12

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.