I am using Rclone to backup my Onedrive account to my Synology NAS.
The Sync command runs fine, I then run a Rclone check every week which which gives a "sizes differ" error.
This only seems to affect photos uploaded using the Onedrive iPhone client on my phone, files added using the OneDrive Windows client are unaffected.
Confusingly I have the same setup to backup my Wife's Onedrive account, she also uses the iOS Onedrive client to upload photos, however rclone check does not lead to any "sizes differ" errors, when run on her account.
I wondered if anyone had ideas on how I can debug this further or resolve it?
rclone v1.58.1
os/version: unknown
os/kernel: 3.10.108 (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.17.9
go/linking: static
go/tags: none
OneDrive (Personal)
rclone copy /tmp remote:tmp)
rclone check "OneDrive:" "/volume2/OneDrive/" --config="/volume2/Local/Rclone/Rclone.conf" --one-way --size-only --log-file=/volume2/Local/Rclone/Rclone_Debug.txt --log-level INFO
[OneDrive]
type = onedrive
token = REDACTED
drive_id = REDACTED
drive_type = personal
-vv flag
2022/06/13 13:56:32 INFO : Using sha1 for hash comparisons
2022/06/13 13:56:33 ERROR : Pictures/Camera Roll/20220515_152649180_iOS.heic: Sizes differ
2022/06/13 13:56:33 ERROR : Pictures/Camera Roll/20220601_133848446_iOS.heic: Sizes differ
2022/06/13 13:56:33 ERROR : Pictures/Camera Roll/20220531_152406613_iOS.heic: Sizes differ
2022/06/13 13:56:33 ERROR : Pictures/Camera Roll/20220601_135158268_iOS.heic: Sizes differ
2022/06/13 13:56:33 ERROR : Pictures/Camera Roll/20220601_080946389_iOS.heic: Sizes differ
2022/06/13 13:56:33 ERROR : Pictures/Camera Roll/20220601_135213249_iOS.heic: Sizes differ
2022/06/13 13:56:33 ERROR : Pictures/Camera Roll/20220601_164209544_iOS.heic: Sizes differ
2022/06/13 13:56:33 ERROR : Pictures/Camera Roll/20220601_175449428_iOS.heic: Sizes differ
2022/06/13 13:56:33 ERROR : Pictures/Camera Roll/20220603_190335288_iOS.heic: Sizes differ
2022/06/13 13:56:33 ERROR : Pictures/Camera Roll/20220608_083424283_iOS.heic: Sizes differ
2022/06/13 13:56:33 ERROR : Pictures/Camera Roll/20220609_085904107_iOS.heic: Sizes differ
2022/06/13 13:56:33 ERROR : Pictures/Camera Roll/20220609_085912947_iOS.heic: Sizes differ
2022/06/13 13:56:33 ERROR : Pictures/Camera Roll/20220609_091400282_iOS.heic: Sizes differ
2022/06/13 13:56:33 ERROR : Pictures/Camera Roll/20220609_104446830_iOS.heic: Sizes differ
2022/06/13 13:56:33 ERROR : Pictures/Camera Roll/20220601_081012361_iOS.heic: Sizes differ
2022/06/13 13:56:33 ERROR : Pictures/Camera Roll/20220501_105802401_iOS.heic: Sizes differ
2022/06/13 13:56:33 ERROR : Pictures/Camera Roll/20220601_135014199_iOS.heic: Sizes differ
2022/06/13 13:56:33 ERROR : Pictures/Camera Roll/20220609_110047170_iOS.heic: Sizes differ
2022/06/13 13:56:33 ERROR : Pictures/Camera Roll/20220521_114148775_iOS.heic: Sizes differ
2022/06/13 13:56:33 ERROR : Pictures/Camera Roll/20220601_135052330_iOS.heic: Sizes differ
2022/06/13 13:56:33 ERROR : Pictures/Camera Roll/20220601_135021329_iOS.heic: Sizes differ
2022/06/13 13:56:33 ERROR : Pictures/Camera Roll/20220601_135202574_iOS.heic: Sizes differ
2022/06/13 13:56:33 ERROR : Pictures/Camera Roll/20220610_191326754_iOS.heic: Sizes differ
2022/06/13 13:56:34 ERROR : Pictures/2020-2029/2022/20220306_141108685_iOS.jpg: Sizes differ
2022/06/13 13:56:34 ERROR : Pictures/2020-2029/2022/20220409_195800672_iOS.heic: Sizes differ
2022/06/13 13:56:34 ERROR : Pictures/2020-2029/2022/20220411_175504595_iOS.heic: Sizes differ
2022/06/13 13:56:34 ERROR : Pictures/2020-2029/2022/20220413_183140349_iOS.heic: Sizes differ
2022/06/13 13:56:34 ERROR : Pictures/2020-2029/2022/20220415_142244769_iOS.heic: Sizes differ
2022/06/13 13:56:34 ERROR : Pictures/2020-2029/2022/20220226_171524152_iOS.jpg: Sizes differ
2022/06/13 13:56:34 ERROR : Pictures/2020-2029/2022/20220226_171531236_iOS.jpg: Sizes differ
2022/06/13 13:56:34 ERROR : Pictures/2020-2029/2022/20220306_150324319_iOS.jpg: Sizes differ
2022/06/13 13:56:34 ERROR : Pictures/2020-2029/2022/20220408_225822587_iOS.heic: Sizes differ
2022/06/13 13:56:40 ERROR : Pictures/2020-2029/2022/FC/Flower_Bed/20220606_192524657_iOS.heic: Sizes differ
2022/06/13 13:56:40 ERROR : Pictures/2020-2029/2022/FC/Flower_Bed/20220606_192552205_iOS.heic: Sizes differ
2022/06/13 13:56:40 ERROR : Pictures/2020-2029/2022/Tess/20220414_182856098_iOS.heic: Sizes differ
2022/06/13 13:56:40 ERROR : Pictures/2020-2029/2022/Tess/20220321_145410074_iOS.heic: Sizes differ
2022/06/13 13:56:40 ERROR : Pictures/2020-2029/2022/Tess/20220415_131253862_iOS.heic: Sizes differ
2022/06/13 13:56:40 ERROR : Pictures/2020-2029/2022/Tess/20220321_205911478_iOS.heic: Sizes differ
2022/06/13 13:56:40 ERROR : Pictures/2020-2029/2022/Tess/20220317_191447931_iOS.heic: Sizes differ
2022/06/13 13:56:40 ERROR : Pictures/2020-2029/2022/Tess/20220322_163043765_iOS.heic: Sizes differ
2022/06/13 13:56:40 ERROR : Pictures/2020-2029/2022/Tess/20220506_165815310_iOS.heic: Sizes differ
2022/06/13 13:56:40 ERROR : Pictures/2020-2029/2022/Tess/20220410_171515953_iOS.heic: Sizes differ
2022/06/13 13:56:40 ERROR : Pictures/2020-2029/2022/Tess/20220417_151855796_iOS.heic: Sizes differ
2022/06/13 13:57:32 INFO :
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Errors: 43 (retrying may help)
Checks: 17082 / 17082, 100%
Elapsed time: 1m0.3s
2022/06/13 13:57:47 NOTICE: Local file system at /volume2/OneDrive/: 43 differences found
2022/06/13 13:57:47 NOTICE: Local file system at /volume2/OneDrive/: 43 errors while checking
2022/06/13 13:57:47 NOTICE: Local file system at /volume2/OneDrive/: 18475 matching files
2022/06/13 13:57:47 INFO :
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Errors: 43 (retrying may help)
Checks: 18518 / 18518, 100%
Elapsed time: 1m14.6s
2022/06/13 13:57:47 Failed to check with 43 errors: last error was: 43 differences found
ncw
June 13, 2022, 2:45pm
2
I'm pretty sure this is the same as this issue caused by a bug in onedrive:
opened 11:11AM - 16 Sep 21 UTC
closed 10:42PM - 15 Nov 21 UTC
Needs: Attention
#### Category
- [ ] Question
- [ ] Documentation issue
- [x] Bug
#### Expe… cted or Desired Behavior
The REST API shows the size of live photos of type HEIC uploaded with the iPhone to be X bytes.
#### Observed Behavior
However when the media for these are downloaded they show a different value Y bytes which is much shorter.
Here is the file when listing it via the API
```
2021/09/16 11:06:17 DEBUG : GET /v1.0/drives/7de8256bfa42446/items/7DE8256BFA42446!103:/big_swan.heic: HTTP/1.1
Host: graph.microsoft.com
User-Agent: rclone/v1.55.1-DEV
Authorization: XXXX
Accept-Encoding: gzip
```
And the response (JSON prettified and edited for privacy)
```
2021/09/16 11:06:17 DEBUG : HTTP RESPONSE (req 0xc000475000)
2021/09/16 11:06:17 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: no-store
Client-Request-Id: 4d33f2d3-99c2-4112-9622-a1f3d3779c15
Content-Type: application/json;odata.metadata=minimal;odata.streaming=true;IEEE754Compatible=false;charset=utf-8
Date: Wed, 15 Sep 2021 23:06:17 GMT
Odata-Version: 4.0
Request-Id: 4d33f2d3-99c2-4112-9622-a1f3d3779c15
Strict-Transport-Security: max-age=31536000
Vary: Accept-Encoding
X-Ms-Ags-Diagnostic: {"ServerInfo":{"DataCenter":"Australia Southeast","Slice":"E","Ring":"4","ScaleUnit":"001","RoleInstance":"ML1PEPF00004AFB"}}
7f7
{
"@odata.context": "https://graph.microsoft.com/v1.0/$metadata#drives('7de8256bfa42446')/items/$entity",
"@microsoft.graph.downloadUrl": "https://public.dm.files.1drv.com/y4mmCSrkuSjRthyO6ZFDft4_m_LxFe8J7BBWl2ylp6zXYSS-YZBeuQ_U6DF4nV4bv1FDl2DgP_6Fp4FO4OF6CWR6YiDiCnrqZrIzRK25BtSj8etPi5pALEqzzFH2LBsty-hXLK0plO2TohCr9FeRfvZGRJU9cC_j20yB6QXPR_wBxwMvpsKDBH382f10ibeeYt4DbBMNMfVCr2qrmyzkg8HtjRxRT_MG9oUhSws2NQ5POkg-smbi7X86P20FynIs0UA0ieIv3jPL-yt-CSNTVbBR66FwelRlUkPz25sC4QNb88",
"createdDateTime": "2021-09-13T22:18:06.877Z",
"cTag": "aYzo3REU4MjU2QkZBNDI0NDYhNDQ3NC4yNTg",
"eTag": "aN0RFODI1NkJGQTQyNDQ2ITQ0NzQuNg",
"id": "7DE8256BFA42446!4474",
"lastModifiedDateTime": "2021-09-13T22:27:05.36Z",
"name": "big_swan.heic",
"size": 6352331,
"webUrl": "XXX",
"reactions": {
"commentCount": 0
},
"createdBy": {
"application": {
"displayName": "OneDrive",
"id": "480f7c59"
},
"user": {
"displayName": "user",
"id": "7de8256bfa42446"
}
},
"lastModifiedBy": {
"user": {
"displayName": "user",
"id": "7de8256bfa42446"
}
},
"parentReference": {
"driveId": "7de8256bfa42446",
"driveType": "personal",
"id": "7DE8256BFA42446!103",
"name": "Pictures",
"path": "/drive/root:/Pictures"
},
"file": {
"mimeType": "image/heic",
"hashes": {
"quickXorHash": "zZ7mPwDhaBHFKsHrBYeywOD0sdM=",
"sha1Hash": "F7FD365360A267FE2249BCB3F290198F7851AF6D",
"sha256Hash": "6B876FF49433ACEA015F1C59DA851095192D2EE2F76CD36922EC184DC334D4FF"
}
},
"fileSystemInfo": {
"createdDateTime": "2021-09-13T22:18:06.877Z",
"lastModifiedDateTime": "2021-09-13T22:22:35.663Z"
},
"image": {
"height": 3024,
"width": 4032
},
"pendingOperations": {
"pendingContentUpdate": {
"queuedDateTime": "2021-09-13T22:22:14.92689Z"
}
},
"photo": {
"cameraMake": "Apple",
"cameraModel": "iPhone SE (2nd generation)",
"exposureDenominator": 60,
"exposureNumerator": 1,
"focalLength": 3.99,
"fNumber": 1.8,
"iso": 125,
"orientation": 1,
"takenDateTime": "2021-09-14T10:17:38Z"
},
"shared": {
"scope": "users",
"owner": {
"user": {
"displayName": "user",
"id": "7de8256bfa42446"
}
}
}
}
0
```
We can quite clearly see that the size was reported as 6352331
And here, when the file was downloaded
```
2021/09/16 11:06:57 DEBUG : HTTP REQUEST (req 0xc000473e00)
2021/09/16 11:06:57 DEBUG : GET /y4mMADT1W1ERPFupg839lgG66eqarRtxUxtzRQhKM05UgHQVC-k5-uMitBX0M_XJwk67x1YoUAgmJxb4QFucfZ6Vl1m5Kpeos0KT8h1tFiqGJGm0DKv5jJjd5IuiJ2BTtFr75bGHoeTZzr9RgegmTWC-okZw6907YRvXfP8AaizQlQJHfq4ubbUwjj8pWT5q9muM8TQYeEfVP23dOXBw1yN8J-4E_8JCx0PTiKTT7EgQDn6DMPy9Y04Zqi4VC8NNFNQ7Nvu7YtaT3J_Yzv2cFw9nUhhiY92-39wcsbqAufXkyo/big_swan.heic HTTP/1.1
Host: public.dm.files.1drv.com
User-Agent: rclone/v1.55.1-DEV
Authorization: XXXX
Referer: https://graph.microsoft.com/v1.0/drives/7de8256bfa42446/items/7DE8256BFA42446!4474/content
Accept-Encoding: gzip
```
The response is
```
2021/09/16 11:06:58 DEBUG : HTTP RESPONSE (req 0xc000473e00)
2021/09/16 11:06:58 DEBUG : HTTP/2.0 200 OK
Content-Length: 2247095
Accept-Ranges: bytes
Cache-Control: public
Content-Disposition: attachment; filename="big_swan.heic"
Content-Location: https://public.dm.files.1drv.com/y4mpvYnqMS8mVrBGQyut9YVg5U-AH9S-z_Unc5kYv198tTOhpt22gmzNnzK_ThPuz_oSf10M7ZwY3SGvYuYE159X1MRR5FRrVc8W93j2nnFcsD37ufi03POeDjN8y1ytw7Tpr9xi6QhRXNW5JRqO7PwnsGV17xXUzh52isCt8rqJ88NRrhofNm6VICh9jJw0XE3Y31IYzGwShWEPE08AtvTqfkZ6qwkEty9S9tvnJh9Mze38wJK9tCyAiF0XdE0G9Nmas1U24Q9EPw8rPdrGhAQlCpzZnDJ_NiWjcn6zjevypk
Content-Type: image/heic
Ctag: aYzo3REU4MjU2QkZBNDI0NDYhNDQ3NC4yNTg
Date: Wed, 15 Sep 2021 23:06:57 GMT
Etag: aN0RFODI1NkJGQTQyNDQ2ITQ0NzQuNg
Expires: Tue, 14 Dec 2021 23:06:58 GMT
Last-Modified: Mon, 13 Sep 2021 22:18:07 GMT
Ms-Cv: JxPlQ7cDXkKWfy4GiTfIRA.0
P3p: CP="BUS CUR CONo FIN IVDo ONL OUR PHY SAMo TELo"
Strict-Transport-Security: max-age=31536000; includeSubDomains
Www-Authenticate: Bearer realm="OneDriveAPI", error="invalid_token", error_description="Invalid auth token"
X-Asmversion: UNKNOWN; 19.758.906.2003
X-Cache: CONFIG_NOCACHE
X-Content-Type-Options: nosniff
X-Msedge-Ref: Ref A: 0AAA19AA453B46FCB7ACF9D4064CFDD9 Ref B: AKL30EDGE0215 Ref C: 2021-09-15T23:06:57Z
X-Msnserver: DM5SCH102211612
X-Preauthinfo: rv;poba;
X-Sqldataorigin: S
X-Streamorigin: X
```
Which quite clearly shows a `Content-Length` of 2247095 which is not equal to the 6352331 reported in the listing.
#### Steps to Reproduce
Uploading a live image from iPhone and examining it via the API.
#### References
This issue looks very similar to https://github.com/OneDrive/onedrive-api-docs/issues/123 - however that issue is marked as resolved, so I've opened a new issue.
This was originally reported in https://github.com/rclone/rclone/issues/5580 by @kelmes who has example images.
Ole
June 13, 2022, 3:50pm
3
Yes, it looks like a known OneDrive limitation with Live Photos from iOS. You can find some additional infomation here:
@asdffdsa Good question, I guess the answer is yes.
@Andrew_Xxx : It looks like the files are Live Photos uploaded from the iOS OneDrive App.
If so, then the issue is most likely due to OneDrive’s partial support of Live Photos. Upload and storage from the iOS app were introduced in December 2020 . The remaining implementation is still work-in-progress and this introduces some “surprising” situations when mounting and synchronizing - also in the native OneDrive client on Windows.
The issue(s) …
I work around the limitation by using --ignore-size when syncing and --download when checking. Please note that this workaround only syncs the still-picture not the movieclip - and it relies on modification dates being correctly updated.
Vanburen:
Confusingly I have the same setup to backup my Wife's Onedrive account, she also uses the iOS Onedrive client to upload photos, however rclone check does not lead to any "sizes differ" errors, when run on her account.
I guess your wife has disabled Live Photos - or an old iPhone.
Thank you @ncw and @Ole .
That was the issue, since I am not that bothered about having live photos I have disabled it on my phone and no longer have sizes differ issue on new uploads.
1 Like
system
June 17, 2022, 3:39pm
5
This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.