Target 1 in sync with 2 and 3, but target 2 not in sync with 3

Help and Support
1

What is the problem you are having with rclone?

I store a bunch of files at two different cloud providers. I'm migrating away from provider 1 to provider 3, but keeping provider 2. I synced from provider 1 to provider 3. Since I'm cancelling my subscription with provider 1, I wanted to sanity check that a sync of provider 2 to provider 3 changes nothing.

The sanity check did not hold.

❯ rclone sync -P --bwlimit 2M --max-delete 0 --transfers 1 crypt-target-1: crypt-target-2:
Enter configuration password:
password:
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:               859 / 859, 100%
Elapsed time:        11.0s
❯ rclone sync -P --bwlimit 2M --max-delete 0 --transfers 1 crypt-target-1: crypt-target-3:
Enter configuration password:
password:
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:               859 / 859, 100%
Elapsed time:        11.9s
❯ rclone sync -P --bwlimit 2M --max-delete 0 --transfers 1 crypt-target-2: crypt-target-3:
Enter configuration password:
password:
Transferred:      288.094k / 3.292 GBytes, 0%, 331.445 kBytes/s, ETA 2h53m34s
Checks:                44 / 87, 51%
Transferred:            0 / 3, 0%
Elapsed time:         5.5s
Checking:
 *                              [some file name]: checking
 *                              [some file name]: checking
 *                              [some file name]: checking
 *                              [some file name]: checking
 *                              [some file name]: checking
 *                              [some file name]: checking
 *                              [some file name]: checking
 *                              [some file name]: checking
Transferring:
 *                              [some file name]:  1% /23.718M, 0/s, -^C
❯ rclone sync -P --bwlimit 2M --max-delete 0 --transfers 1 crypt-target-1: crypt-target-3:
Enter configuration password:
password:
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:               859 / 859, 100%
Elapsed time:        11.7s
❯ rclone sync -P --bwlimit 2M --max-delete 0 --transfers 1 crypt-target-1: crypt-target-2:
Enter configuration password:
password:
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:               859 / 859, 100%
Elapsed time:        11.2s

If provider 1 is in sync with 2 and 3, then how come provider 2 is not in sync with 3?

Run the command 'rclone version' and share the full output of the command.

rclone v1.53.3-DEV
- os/arch: linux/amd64
- go version: go1.18

This was the latest one that's available on Ubuntu 22.04 LTS.

Which cloud storage system are you using? (eg Google Drive)

  • crypt-target-1 is webdav wrapped in crypt
  • crypt-target-2 is s3 wrapped in crypt
  • crypt-target-3 is drive wrapped in crypt

These are not the actual names of the targets. I've renamed them to something universal so it's easier to understand the situation and to keep personal info off the public internet.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

As above:

rclone sync -P --bwlimit 2M --max-delete 0 --transfers 1 crypt-target-2: crypt-target-3:

The rclone config contents with secrets removed.

Each crypt target references a webdav/s3/drive target. This is my config with my personal info redacted, the targets renamed, and unrelated targets removed:

[webdav-target-1]
type = webdav
url = https://[redacted]
vendor = other
user = [redacted]
pass = [redacted]

[crypt-target-1]
type = crypt
remote = webdav-target-1:[redacted]
filename_encryption = standard
directory_name_encryption = true
password = [redacted]
password2 = [redacted]

[s3-target-2]
type = s3
provider = Other
env_auth = false
access_key_id = [redacted]
secret_access_key = [redacted]
endpoint = [redacted]
location_constraint = de-fra-1
acl = private

[crypt-target-2]
type = crypt
remote = s3-target-2:[redacted]
filename_encryption = standard
directory_name_encryption = true
password = [redacted]
password2 = [redacted]

[drive-target-3]
type = drive
client_id = [redacted]
client_secret = [redacted]
scope = drive.file
token = [redacted]

[crypt-target-3]
type = crypt
remote = drive-target-3:[redacted]
directory_name_encryption = true
password = [redacted]
password2 = [redacted]

A log from the command with the -vv flag

❯ rclone sync -vv -P --bwlimit 2M --max-delete 0 --transfers 1 crypt-target-2: crypt-target-3:
2023/01/02 22:50:06 DEBUG : rclone: Version "v1.53.3-DEV" starting with parameters ["rclone" "sync" "-vv" "-P" "--bwlimit" "2M" "--max-delete" "0" "--transfers" "1" "crypt-target-2:" "crypt-target-3:"]
Enter configuration password:
password:
2023/01/02 22:50:09 DEBUG : Using config file from "/home/myusername/.config/rclone/rclone.conf"
2023/01/02 22:50:09 INFO  : Starting bandwidth limiter at 2MBytes/s
2023/01/02 22:50:09 DEBUG : Creating backend with remote "crypt-target-2:"
2023/01/02 22:50:09 DEBUG : Creating backend with remote "s3-target-2:[redacted]"
2023/01/02 22:50:09 DEBUG : Creating backend with remote "crypt-target-3:"
2023/01/02 22:50:09 DEBUG : Creating backend with remote "drive-target-3:[redacted]"
2023/01/02 22:50:09 DEBUG : Google drive root 'Vault': root_folder_id = "root" - save this in the config to speed up startup
2023-01-02 22:50:10 DEBUG : [redacted]: Size and modification time the same (differ by 0s, within tolerance 1ms)
2023-01-02 22:50:10 DEBUG : [redacted]: Unchanged skipping
2023-01-02 22:50:10 DEBUG : [redacted]: Size and modification time the same (differ by 0s, within tolerance 1ms)
2023-01-02 22:50:10 DEBUG : [redacted]: Unchanged skipping
2023-01-02 22:50:10 DEBUG : [redacted]: Size and modification time the same (differ by 0s, within tolerance 1ms)
2023-01-02 22:50:10 DEBUG : [redacted]: Unchanged skipping
2023-01-02 22:50:10 DEBUG : [redacted]: Size and modification time the same (differ by 0s, within tolerance 1ms)
2023-01-02 22:50:10 DEBUG : [redacted]: Unchanged skipping
2023-01-02 22:50:10 DEBUG : [redacted]: Size and modification time the same (differ by 0s, within tolerance 1ms)
2023-01-02 22:50:10 DEBUG : [redacted]: Unchanged skipping
2023-01-02 22:50:10 DEBUG : [redacted]: Modification times differ by 4m51.2824767s: 2022-06-06 13:57:03.7175233 +0200 CEST, 2022-06-06 12:01:55 +0000 UTC
2023-01-02 22:50:10 DEBUG : [redacted]: Size and modification time the same (differ by 0s, within tolerance 1ms)
2023-01-02 22:50:10 DEBUG : [redacted]: Unchanged skipping
2023-01-02 22:50:10 DEBUG : [redacted]: Size and modification time the same (differ by 0s, within tolerance 1ms)
2023-01-02 22:50:10 DEBUG : [redacted]: Unchanged skipping
2023-01-02 22:50:10 DEBUG : [redacted]: Size and modification time the same (differ by 0s, within tolerance 1ms)
2023-01-02 22:50:10 DEBUG : [redacted]: Unchanged skipping
2023-01-02 22:50:10 DEBUG : [redacted]: Size and modification time the same (differ by 0s, within tolerance 1ms)
2023-01-02 22:50:10 DEBUG : [redacted]: Unchanged skipping
2023-01-02 22:50:11 DEBUG : [redacted]: Size and modification time the same (differ by 0s, within tolerance 1ms)
2023-01-02 22:50:11 DEBUG : [redacted]: Modification times differ by 5m43.7618903s: 2022-10-20 20:38:43.2381097 +0200 CEST, 2022-10-20 18:44:27 +0000 UTC
2023-01-02 22:50:11 DEBUG : [redacted]: Size and modification time the same (differ by 0s, within tolerance 1ms)
2023-01-02 22:50:11 DEBUG : [redacted]: Unchanged skipping
2023-01-02 22:50:11 DEBUG : [redacted]: Modification times differ by 39m9.0036905s: 2022-10-20 18:17:11.9963095 +0200 CEST, 2022-10-20 16:56:21 +0000 UTC
2023-01-02 22:50:11 DEBUG : [redacted]: Size and modification time the same (differ by 0s, within tolerance 1ms)
2023-01-02 22:50:11 DEBUG : [redacted]: Unchanged skipping
2023-01-02 22:50:11 DEBUG : [redacted]: Unchanged skipping
2023-01-02 22:50:11 DEBUG : [redacted]: Size and modification time the same (differ by 0s, within tolerance 1ms)
2023-01-02 22:50:11 DEBUG : [redacted]: Unchanged skipping
2023-01-02 22:50:11 DEBUG : [redacted]: Size and modification time the same (differ by 0s, within tolerance 1ms)
2023-01-02 22:50:11 DEBUG : [redacted]: Unchanged skipping
2023-01-02 22:50:11 DEBUG : [redacted]: Size and modification time the same (differ by 0s, within tolerance 1ms)
2023-01-02 22:50:11 DEBUG : [redacted]: Unchanged skipping
2023-01-02 22:50:11 DEBUG : [redacted]: Size and modification time the same (differ by 0s, within tolerance 1ms)
2023-01-02 22:50:11 DEBUG : [redacted]: Unchanged skipping
2023-01-02 22:50:11 DEBUG : [redacted]: Size and modification time the same (differ by 0s, within tolerance 1ms)
2023-01-02 22:50:11 DEBUG : [redacted]: Unchanged skipping
2023-01-02 22:50:11 DEBUG : [redacted]: Size and modification time the same (differ by 0s, within tolerance 1ms)
2023-01-02 22:50:11 DEBUG : [redacted]: Unchanged skipping
2023-01-02 22:50:11 DEBUG : [redacted]: Size and modification time the same (differ by 0s, within tolerance 1ms)
2023-01-02 22:50:11 DEBUG : [redacted]: Unchanged skipping
2023-01-02 22:50:11 DEBUG : [redacted]: Size and modification time the same (differ by 0s, within tolerance 1ms)
2023-01-02 22:50:11 DEBUG : [redacted]: Unchanged skipping
2023-01-02 22:50:11 DEBUG : [redacted]: Sending chunk 0 length 8388608
Transferred:             0 / 3.292 GBytes, 0%, 0 Bytes/s, ETA -
Checks:                23 / 87, 26%
Transferred:            0 / 3, 0%
Elapsed time:         5.9s
Checking:
 *                              [some file name]: checking
 *                              [some file name]: checking
 *                              [some file name]: checking
 *                              [some file name]: checking
 *                              [some file name]: checking
 *                              [some file name]: checking
 *                              [some file name]: checking
 *                              [some file name]: checking
Transferring:
 *                              [some file name]:  0% /623.693M, 0/s, -^C

It looks like the the modification times of some files differ by at least a few minutes. This feels more significant and random than a bit of clock drift or loss of timestamp precision. What could be the actual reason?

I really assumed that if the files on provider 1 and 2 were equal, and the files on provider 1 and 3 were equal, then the files on provider 2 and 3 would also be equal (aka the transitive property).

2

That is a really, really old version so you'd want to update that.

There's not really a lot of log to analyze.

You'd want share the final ouputs of the sync from 1->2 and then 2->3 and see the counts.

If they don't match up, you'd want to share the whole log as with missing file names/directories, it's really a guessing game to look at that log.

3

This may well be due to modification times.

Note that the source webdav doesn't support modification times so rclone won't be using them for syncing.

Syncing webdav -> s3 and webdav -> drive will preserve the apparent modification time which will be whatever the webdav server returned.

However syncing from s3 -> drive will use modification times, so if the webdav server was inconsistent with the times then you'd see changes in the modification times.

The way to check that the data has really arrived is to do rclone check s3: drive: which will use md5 checksums to check your data is intact.

4

I did, the output is near the top of the original post. I did re-run in case you meant with -vv included. When I sync target 1 (WebDAV + crypt) to target 2 (S3-compatible storage + crypt) or target 3 (Google Drive + crypt), all output seems to be along these lines:

2023-01-03 19:18:47 DEBUG : [some file name]: Sizes identical
2023-01-03 19:18:47 DEBUG : [some file name]: Unchanged skipping

But based on that log, I have the impression that it's only comparing file sizes between the two targets.

Interesting. When comparing between s3 and drive rclone indeed seems to use a combination of size and modification date. Quoth the log:

2023-01-03 19:30:35 DEBUG : [some file name]: Size and modification time the same (differ by 0s, within tolerance 1ms)
2023-01-03 19:30:35 DEBUG : [some file name]: Unchanged skipping

The docs said hashes are not stored for crypt, so I ran cryptcheck instead. Output appears to be along the lines of:

2023/01/03 19:32:54 DEBUG : [some file name]: OK

Is it safe to conclude that when doing a threeway sync between s3, webdav and drive targets, it's better not to use webdav as a source because it doesn't support modification times and may therefore trigger unnecessary file syncing when switching sources?

5

Not really as you haven't shared an output of a file being recopied over because of an issue.

That's not being recopied.

That's not being recopied either.

If you can share the debug log line of a file being recopied, that will show the reason why and if it just the mod times, depending on how much 'skew' you are comfortable with, you can change that window.

If not, not using WebDav as the source is probably best and I actually probably wouldn't anyway as I want mod times.

6

I pulled this one from my original post:

2023-01-02 22:50:11 DEBUG : [redacted]: Modification times differ by 39m9.0036905s: 2022-10-20 18:17:11.9963095 +0200 CEST, 2022-10-20 16:56:21 +0000 UTC

I'm hoping that if I repeat the sync process using the S3 remote as a source, it will straighten out the modification times once and for all as long as I never use a WebDAV remote as a source again. I'd prefer doing that over introducing a permanent modification time skew tolerance to my sync script.

Would that be a viable strategy?

7

That does make sense to me and sounds viable.

WebDav is good for a very small subset of things imo and I wouldn't use it.

8

Alright, thank you both for your help!

1 Like
9

The cause of this is either the webdav server gave a different time the second time the file was read, or that the file changed but it size didn't.

If the latter, then it is a genuine change and needs to be synced.

Note that since webdav doesn't support modtimes or checksums you are doing a --size-only sync when syncing from webdav.

You might consider using the --update flag when syncing from webdav - this will then use the time provided by the server and sync changes if the time increases.

(Note to self: rclone could really do with notion of read only modification time precision as the times when synced from webdav should always be consistent.)

10

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.