Hummel
February 13, 2025, 9:30am
1
The problem is that the rclone sync between MS Onedrive and the host keeps updating the same files. Each sync the same 50+ changes are in the logs.
rclone v1.69.0
os/version: raspbian 11.11 (64 bit)
os/kernel: 6.1.21-v8+ (aarch64)
os/type: linux
os/arch: arm64 (ARMv8 compatible)
go/version: go1.23.4
go/linking: static
go/tags: none
> sudo rclone sync <source> <target> --modify-window 1s --config .config/rclone/rclone.conf -vv
[Onedrive-XXX]
type = onedrive
token = {"access_token":"..."}
drive_id = 1234567890
drive_type = personal
# first run
2025/02/13 10:14:59 DEBUG : Pictures/Camera Roll/2024/06/20240609_165332754_iOS.heic: quickxor = b5981c5f640c9498a56b33198379b496f13c47b5 OK
2025/02/13 10:14:59 DEBUG : Pictures/Camera Roll/2024/06/20240609_165332754_iOS.heic.68824d3e.partial: renamed to: Pictures/Camera Roll/2024/06/20240609_165332754_iOS.heic
2025/02/13 10:14:59 INFO : Pictures/Camera Roll/2024/06/20240609_165332754_iOS.heic: Copied (replaced existing)
---
2025/02/13 10:15:01 INFO : Pictures/Camera Roll/2021/12: Set directory modification time (using SetModTime)
# second run
2025/02/13 10:18:58 DEBUG : Pictures/Camera Roll/2024/06/20240609_165332754_iOS.heic: quickxor = b5981c5f640c9498a56b33198379b496f13c47b5 OK
2025/02/13 10:18:58 DEBUG : Pictures/Camera Roll/2024/06/20240609_165332754_iOS.heic.68824d3e.partial: renamed to: Pictures/Camera Roll/2024/06/20240609_165332754_iOS.heic
2025/02/13 10:18:58 INFO : Pictures/Camera Roll/2024/06/20240609_165332754_iOS.heic: Copied (replaced existing)
---
2025/02/13 10:18:59 INFO : Pictures/Camera Roll/2021/12: Set directory modification time (using SetModTime)
---
Transferred: 103.066 MiB / 103.066 MiB, 100%, 12.607 MiB/s, ETA 0s
Checks: 2577 / 2577, 100%
Transferred: 56 / 56, 100%
Elapsed time: 9.4s
Please run your sync with -vv flag and post it full here. Nobody can guess all details from your snippets.
You can limit sync only to some subdir to make is shorter, e.g.:
rclone sync onedrive:Pictures/Camera Roll/2024/06 . -vv
Also what is your target?
It is unlikely a bug as I can not reproduce it. Sync to/from OneDrive works perfectly fine. So most likely it is something specific to your configuration or OneDrive account.
Hummel
February 13, 2025, 10:51am
3
The target is an SSD drive on a Raspberry Pi 4 with Debian. Files permission 644.
sudo rclone sync OnedriveXYZ:"/Pictures/Camera Roll/2021/12/" "/home/XYZ/personal/Pictures/Camera Roll/2021/12/" --modify-window 1s --config /home/XYZ/.config/rclone/rclone.conf -vv
2025/02/13 11:49:56 DEBUG : rclone: Version "v1.69.0" starting with parameters ["rclone" "sync" "Onedrive-XYZ:/Pictures/Camera Roll/2021/12/" "/home/XYZ/personal/Pictures/Camera Roll/2021/12/" "--modify-window" "1s" "--config" "/home/XYZ/.config/rclone/rclone.conf" "-vv"]
2025/02/13 11:49:56 DEBUG : Creating backend with remote "Onedrive-XYZ:/Pictures/Camera Roll/2021/12/"
2025/02/13 11:49:56 DEBUG : Using config file from "/home/XYZ/.config/rclone/rclone.conf"
2025/02/13 11:49:57 DEBUG : fs cache: renaming cache item "Onedrive-XYZ:/Pictures/Camera Roll/2021/12/" to be canonical "Onedrive-XYZ:Pictures/Camera Roll/2021/12"
2025/02/13 11:49:57 DEBUG : Creating backend with remote "/home/XYZ/personal/Pictures/Camera Roll/2021/12/"
2025/02/13 11:49:57 DEBUG : fs cache: renaming cache item "/home/XYZ/personal/Pictures/Camera Roll/2021/12/" to be canonical "/home/XYZ/personal/Pictures/Camera Roll/2021/12"
2025/02/13 11:49:58 DEBUG : 20211201_153501000_iOS.mp4: Size and modification time the same (differ by 0s, within tolerance 1s)
2025/02/13 11:49:58 DEBUG : 20211201_153501000_iOS.mp4: Unchanged skipping
2025/02/13 11:49:58 DEBUG : 20211206_103044553_iOS.jpg: Size and modification time the same (differ by 0s, within tolerance 1s)
2025/02/13 11:49:58 DEBUG : 20211206_103044553_iOS.jpg: Unchanged skipping
2025/02/13 11:49:58 DEBUG : 20211209_210623000_iOS.jpg: Size and modification time the same (differ by 0s, within tolerance 1s)
2025/02/13 11:49:58 DEBUG : 20211209_210623000_iOS.jpg: Unchanged skipping
2025/02/13 11:49:58 DEBUG : 20211216_104056000_iOS.mov: Size and modification time the same (differ by 0s, within tolerance 1s)
2025/02/13 11:49:58 DEBUG : 20211216_104056000_iOS.mov: Unchanged skipping
2025/02/13 11:49:58 DEBUG : 20211223_090431574_iOS.heic: Sizes differ (src 3872482 vs dst 1275974)
2025/02/13 11:49:58 DEBUG : Local file system at /home/XYZ/personal/Pictures/Camera Roll/2021/12: Waiting for checks to finish
2025/02/13 11:49:58 DEBUG : Local file system at /home/XYZ/personal/Pictures/Camera Roll/2021/12: Waiting for transfers to finish
2025/02/13 11:49:58 DEBUG : 20211223_090431574_iOS.heic: quickxor = 156450c2ffe7d00902a0c5b4195b67bfa80aaa59 OK
2025/02/13 11:49:58 DEBUG : 20211223_090431574_iOS.heic.6cd22a32.partial: renamed to: 20211223_090431574_iOS.heic
2025/02/13 11:49:58 INFO : 20211223_090431574_iOS.heic: Copied (replaced existing)
2025/02/13 11:49:58 DEBUG : Waiting for deletions to finish
2025/02/13 11:49:58 INFO :
Transferred: 1.217 MiB / 1.217 MiB, 100%, 0 B/s, ETA -
Checks: 5 / 5, 100%
Transferred: 1 / 1, 100%
Elapsed time: 0.7s
2025/02/13 11:49:58 DEBUG : 8 go routines active```
It is well known and documented problem (or feature:)):
If you want to store any files using OneDrive and make sure that they do not interfere in any way with your content use rclone crypt.
Otherwise play by rules set by Microsoft. Of course you can complain to them but it is probably waste of time.
Some workaround would be to use --ignore-size flag (maybe only for heic files) but still what you download won't be what you uploaded.
If you use Apple devices then for trouble free experience the best option is to stay within their ecosystem. And eventually use OneDrive as a dump cloud storage only, with crypt on top to prevent MS from being creative about what to do with your files.
Hummel
February 13, 2025, 11:57am
5
Thanks for the explanation. If I am correct iPhone can be set to save photos in compatible format. Would that resolve the issue?
AFAIK there is no other format for live photos. It is Apple specific.
If you use OneDrive to store media from iPhone then live photos will be lost (or rather converted to stills). Up to you.
And still you have to deal with rclone sync issue.
BTW - iCloud support is coming to rclone. The last rclone version already supports iDrive. iPhotos should be at some point in the future.