Unchanged Photos to OneDrive are frequently Reuploaded

[nightflame (Steve)]

I use rClone to backup all my files from Linux laptop to OneDrive Personal. However I am finding that JPEG photos that I have not changed are being re-uploaded to OnDrive.

rclone v1.50.2

• os/arch: linux/amd64
• go version: go1.14.4

Linux 5.8.0-50-generic #56-Ubuntu SMP x86_64 x86_64 x86_64 GNU/Linux

I use the following command:

rclone sync --copy-links /DATA/LOCAL/ onedrive:LOCAL 

Simple config:

[onedrive]
type = onedrive
token = {"access_token":"stuff"}
drive_id = stuff
drive_type = personal

Here is the debug of when unchanged files are resynced:

2021/05/08 12:37:12 DEBUG : 20181225-060457-SW1-000.jpg: Sizes differ (src 8630795 vs dst 7400740)
2021/05/08 12:37:12 DEBUG : 20181225-060457-SW1-000.jpg: Starting multipart upload
2021/05/08 12:37:13 DEBUG : 20181225-060457-SW1-000.jpg: Uploading segment 0/8630795 size 8630795
2021/05/08 12:37:24 DEBUG : 20181225-060457-SW1-000.jpg: SHA-1 = 7221b4e84fad93c05df6fbb796dc6201cb23f536 OK
2021/05/08 12:37:24 INFO  : 20181225-060457-SW1-000.jpg: Copied (replaced existing)

To my surprise the file size is different from the original!

After some analysis it looks like OneDrive is removing the MP Image Type (which accounts for most of the filesize change) and changing the XMP metdata data on some of the images. Specifically introducing MicrosoftPhoto:LastKeywordXMP. It also seems to change the image container as well (to JFIF standard 1.01)

I have done a full analysis with exiftool, but cannot link to the web page with the full results from here.

Oddly this does not happen all the time, and not always on the same files. Sometimes after changing the file after 5 uploads OneDrive will accept the original and not change it.

I sync around 400GB to OneDrive and this only happens on JPG image files. So I do not believe this is an rClone issue, but would be interested if others experience a similar issue.

I have raised a support case with Microsoft but am not getting much progress.

[Animosity022]

Check out:

https://rclone.org/onedrive/#unexpected-file-size-hash-differences-on-sharepoint

[nightflame (Steve)]

Thanks, I had checked that out but it does not really help. My issue is with OneDrive Personal, and not only are the filesizes (therefore hash) are different, the date is also different.

If I look at the version history, one minute after the upload the file is updated.

For example:

    Sat at 12:38 Steve 7.06MB
    Sat at 12:37 Steve 8.23MB

The file in OneDrive is now 1MB less than the original!

If I download the OneDrive version and compare the Original file:

file 20181225-060457-SW1-000-orig.jpg

20181225-060457-SW1-000-orig.jpg: JPEG image data, Exif standard: [TIFF image data, little-endian, direntries=15, manufacturer=Panasonic, model=DMC-LX15, orientation=upper-left, xresolution=214, yresolution=222, resolutionunit=2, software=Ver.1.0 , datetime=2018:12:25 06:04:57], baseline, precision 8, 5472x3080, components 3

And

file 20181225-060457-SW1-000-one.jpg

20181225-060457-SW1-000-one.jpg: JPEG image data, JFIF standard 1.01, resolution (DPI), density 180x180, segment length 16, Exif Standard: [TIFF image data, big-endian, direntries=18, description=Ayutthaya, the Ancient Capital of Thailand, manufacturer=Panasonic, model=DMC-LX15, orientation=upper-left, xresolution=294, yresolution=302, resolutionunit=2, software=Ver.1.0 , datetime=2018:12:25 06:04:57], baseline, precision 8, 5472x3080, components 3

Also it would appear the 1.2MB HD Image Prview has been pruned from the OneDrive version:

exiftool -All 20181225-060457-SW1-000-orig.jpg | egrep -i "preview|thumb"

MP Image Type : Large Thumbnail (full HD equivalent)

Preview Image : (Binary data 1232353 bytes)`

No sign of the PreviewImage in the OneDrive version:

exiftool -a -b -W thumb-orig-%f_%t%-c.%s -preview:all 20181225-060457-SW1-000-one.jpg 0 output files created

JPEG EOI Warning = [minor] Error reading PreviewImage from file

If I look closer at the file the XMP MetaData has been altered. With some Microsoft specific attributes added to the OneDrive version:

[XMP-microsoft] LastKeywordXMP
[XMP-microsoft] LastKeywordIPTC

OneDrive is changing the file (metadata and thumbnail). Rclone will always resync these files. Which I guess is the correct behaviour cos they have changed! Kinda makes OneDrive useless for me, unless I "protect" image files by using rclone encryption.

[Animosity022]

On the page, it tells you if you want to use OneDrive, you'd have to use:

--ignore-checksum --ignore-size

There isn't much rclone can do about it as OneDrive is modifying the files after upload.

If the dates are close, you can also use:

      --modify-window duration               Max time diff to be considered the same (default 1ns)

To set a gap there. Personally, I wouldn't do either as I'd find a better solution as for me, that's also unacceptable and why I don't use OneDrive.