What is the problem you are having with rclone?
Subsequent rclone sync keep syncing the same files due to skewed modtime.
Run the command 'rclone version' and share the full output of the command.
$ rclone version
rclone v1.67.0
- os/version: darwin 14.6.1 (64 bit)
- os/kernel: 23.6.0 (x86_64)
- os/type: darwin
- os/arch: amd64
- go/version: go1.22.4
- go/linking: dynamic
- go/tags: cmount
Which cloud storage system are you using? (eg Google Drive)
Microsoft OneDrive Personal
The command you were trying to run (eg `rclone copy /tmp
rclone sync
The rclone config contents with secrets removed.
Encrypted union of multiple OneDrive accounts. Nothing fancy, mostly defaults.
onedrive x 10 -> union -> crypt
# 10x Onedrive Personal accounts
[.od01_a]
type = onedrive
client_id = XXX
client_secret = XXX
token = XXX
drive_id = XXX
drive_type = personal
...
[.od02_l]
type = onedrive
client_id = XXX
client_secret = XXX
token = XXX
drive_id = XXX
drive_type = personal
[.OD01-UNION]
type = union
upstreams = .od01_a:a .od01_b:b .od01_c:c .od01_d:d .od02_g:g .od02_h:h .od02_i:i .od02_j:j .od02_k:k .od02_l:l
action_policy = epall
create_policy = mfs
search_policy = ff
cache_time = 60
[OD01-ENC]
type = crypt
remote = .OD01-UNION:
password = XXX
password2 = XXX
filename_encoding = base32768
A log from the command with the -vv flag
initially I simply sync local directory containing 10 files. Nothing unusual here. It works perfectly.
$ rclone sync /Volumes/DBSamsung-SSD-500-Temp/restore/files1 OD01-ENC:test/files1 -vv
2024/08/18 12:58:41 DEBUG : rclone: Version "v1.67.0" starting with parameters ["rclone" "sync" "/Volumes/DBSamsung-SSD-500-Temp/restore/files1" "OD01-ENC:test/files1" "-vv"]
2024/08/18 12:58:41 DEBUG : Creating backend with remote "/Volumes/DBSamsung-SSD-500-Temp/restore/files1"
2024/08/18 12:58:41 DEBUG : Using config file from "/Users/kptsky/.config/rclone/rclone.conf"
...
Transferred: 1000.244 MiB / 1000.244 MiB, 100%, 4.856 MiB/s, ETA 0s
Transferred: 10 / 10, 100%
Elapsed time: 3m21.6s
but when I try to run it again it is when it becomes interesting. Sync thinks that all files (or most in other tests) changed and have to be transferred again.
$ rclone sync /Volumes/DBSamsung-SSD-500-Temp/restore/files1 OD01-ENC:test/files1 -vv --dry-run
2024/08/18 13:02:30 DEBUG : rclone: Version "v1.67.0" starting with parameters ["rclone" "sync" "/Volumes/DBSamsung-SSD-500-Temp/restore/files1" "OD01-ENC:test/files1" "-vv" "--dry-run"]
2024/08/18 13:02:30 DEBUG : Creating backend with remote "/Volumes/DBSamsung-SSD-500-Temp/restore/files1"
2024/08/18 13:02:30 DEBUG : Using config file from "/Users/kptsky/.config/rclone/rclone.conf"
2024/08/18 13:02:30 DEBUG : Creating backend with remote "OD01-ENC:test/files1"
2024/08/18 13:02:30 DEBUG : Creating backend with remote ".OD01-UNION:终Ԋ䏮蠂漶骯屯ꉸ┟/䩣胬䀼䅁骉㐆嫶候䪿"
2024/08/18 13:02:30 DEBUG : Creating backend with remote ".od02_l:l"
2024/08/18 13:02:30 DEBUG : Creating backend with remote ".od02_g:g"
2024/08/18 13:02:30 DEBUG : Creating backend with remote ".od02_i:i"
2024/08/18 13:02:30 DEBUG : Creating backend with remote ".od01_b:b"
2024/08/18 13:02:30 DEBUG : Creating backend with remote ".od02_k:k"
2024/08/18 13:02:30 DEBUG : Creating backend with remote ".od02_j:j"
2024/08/18 13:02:30 DEBUG : Creating backend with remote ".od02_h:h"
2024/08/18 13:02:30 DEBUG : Creating backend with remote ".od01_d:d"
2024/08/18 13:02:30 DEBUG : Creating backend with remote ".od01_a:a"
2024/08/18 13:02:30 DEBUG : Creating backend with remote ".od01_c:c"
2024/08/18 13:02:30 DEBUG : Creating backend with remote ".od01_a:a/终Ԋ䏮蠂漶骯屯ꉸ┟/䩣胬䀼䅁骉㐆嫶候䪿"
2024/08/18 13:02:31 DEBUG : Creating backend with remote ".od02_g:g/终Ԋ䏮蠂漶骯屯ꉸ┟/䩣胬䀼䅁骉㐆嫶候䪿"
2024/08/18 13:02:31 DEBUG : Creating backend with remote ".od01_b:b/终Ԋ䏮蠂漶骯屯ꉸ┟/䩣胬䀼䅁骉㐆嫶候䪿"
2024/08/18 13:02:31 DEBUG : Creating backend with remote ".od01_c:c/终Ԋ䏮蠂漶骯屯ꉸ┟/䩣胬䀼䅁骉㐆嫶候䪿"
2024/08/18 13:02:31 DEBUG : Creating backend with remote ".od02_h:h/终Ԋ䏮蠂漶骯屯ꉸ┟/䩣胬䀼䅁骉㐆嫶候䪿"
2024/08/18 13:02:31 DEBUG : Creating backend with remote ".od02_k:k/终Ԋ䏮蠂漶骯屯ꉸ┟/䩣胬䀼䅁骉㐆嫶候䪿"
2024/08/18 13:02:31 DEBUG : Creating backend with remote ".od02_j:j/终Ԋ䏮蠂漶骯屯ꉸ┟/䩣胬䀼䅁骉㐆嫶候䪿"
2024/08/18 13:02:31 DEBUG : Creating backend with remote ".od02_i:i/终Ԋ䏮蠂漶骯屯ꉸ┟/䩣胬䀼䅁骉㐆嫶候䪿"
2024/08/18 13:02:31 DEBUG : Creating backend with remote ".od01_d:d/终Ԋ䏮蠂漶骯屯ꉸ┟/䩣胬䀼䅁骉㐆嫶候䪿"
2024/08/18 13:02:32 DEBUG : Creating backend with remote ".od02_l:l/终Ԋ䏮蠂漶骯屯ꉸ┟/䩣胬䀼䅁骉㐆嫶候䪿"
2024/08/18 13:02:33 DEBUG : union root '终Ԋ䏮蠂漶骯屯ꉸ┟/䩣胬䀼䅁骉㐆嫶候䪿': actionPolicy = *policy.EpAll, createPolicy = *policy.Mfs, searchPolicy = *policy.FF
2024/08/18 13:02:33 DEBUG : file003.bin: Modification times differ by -63.400271ms: 2024-08-16 10:42:43.063400271 +0200 CEST, 2024-08-16 08:42:43 +0000 UTC
2024/08/18 13:02:33 DEBUG : file001.bin: Modification times differ by -374.029783ms: 2024-08-16 10:42:41.374029783 +0200 CEST, 2024-08-16 08:42:41 +0000 UTC
2024/08/18 13:02:33 DEBUG : file008.bin: Modification times differ by -273.014046ms: 2024-08-16 10:42:47.273014046 +0200 CEST, 2024-08-16 08:42:47 +0000 UTC
2024/08/18 13:02:33 DEBUG : file010.bin: Modification times differ by -958.277552ms: 2024-08-16 10:42:48.958277552 +0200 CEST, 2024-08-16 08:42:48 +0000 UTC
2024/08/18 13:02:33 DEBUG : file009.bin: Modification times differ by -120.467686ms: 2024-08-16 10:42:48.120467686 +0200 CEST, 2024-08-16 08:42:48 +0000 UTC
2024/08/18 13:02:33 DEBUG : Encrypted drive 'OD01-ENC:test/files1': Waiting for checks to finish
2024/08/18 13:02:33 DEBUG : file002.bin: Modification times differ by -218.780719ms: 2024-08-16 10:42:42.218780719 +0200 CEST, 2024-08-16 08:42:42 +0000 UTC
2024/08/18 13:02:33 NOTICE: file003.bin: Skipped copy as --dry-run is set (size 100Mi)
2024/08/18 13:02:33 NOTICE: file010.bin: Skipped copy as --dry-run is set (size 100Mi)
2024/08/18 13:02:33 NOTICE: file008.bin: Skipped copy as --dry-run is set (size 100Mi)
2024/08/18 13:02:33 NOTICE: file009.bin: Skipped copy as --dry-run is set (size 100Mi)
2024/08/18 13:02:33 NOTICE: file001.bin: Skipped copy as --dry-run is set (size 100Mi)
2024/08/18 13:02:33 DEBUG : file007.bin: Modification times differ by -433.916816ms: 2024-08-16 10:42:46.433916816 +0200 CEST, 2024-08-16 08:42:46 +0000 UTC
2024/08/18 13:02:33 DEBUG : file005.bin: Modification times differ by -747.74874ms: 2024-08-16 10:42:44.74774874 +0200 CEST, 2024-08-16 08:42:44 +0000 UTC
2024/08/18 13:02:33 DEBUG : file006.bin: Modification times differ by -578.161567ms: 2024-08-16 10:42:45.578161567 +0200 CEST, 2024-08-16 08:42:45 +0000 UTC
2024/08/18 13:02:33 DEBUG : file004.bin: Modification times differ by -903.427916ms: 2024-08-16 10:42:43.903427916 +0200 CEST, 2024-08-16 08:42:43 +0000 UTC
2024/08/18 13:02:33 NOTICE: file005.bin: Skipped copy as --dry-run is set (size 100Mi)
2024/08/18 13:02:33 DEBUG : Encrypted drive 'OD01-ENC:test/files1': Waiting for transfers to finish
2024/08/18 13:02:33 NOTICE: file007.bin: Skipped copy as --dry-run is set (size 100Mi)
2024/08/18 13:02:33 NOTICE: file006.bin: Skipped copy as --dry-run is set (size 100Mi)
2024/08/18 13:02:33 NOTICE: file002.bin: Skipped copy as --dry-run is set (size 100Mi)
2024/08/18 13:02:33 NOTICE: file004.bin: Skipped copy as --dry-run is set (size 100Mi)
2024/08/18 13:02:33 DEBUG : Waiting for deletions to finish
2024/08/18 13:02:33 NOTICE:
Transferred: 1000 MiB / 1000 MiB, 100%, 0 B/s, ETA -
Checks: 10 / 10, 100%
Transferred: 10 / 10, 100%
Elapsed time: 3.4s
2024/08/18 13:02:33 DEBUG : 27 go routines active
The same happens when I try the latest rclone beta v1.68.0.
It works as expected when I sync directly to OneDrive. Modtime differences are order of magnitude smaller (micro vs milli seconds). It means that problem is not on OneDrive side nor local storage IMO.
It seems like Union + Crypt remotes disturb modtime...