OneDrive lsjson reporting the wrong filesize

Help and Support
1

What is the problem you are having with rclone?

OneDrive (and/or rclone) is reporting the wrong size for a file causing it to sync when it shouldn't. (Both within rclone directly and my own wrapper)

Demonstrated by copying

What is your rclone version (output from rclone version)

rclone v1.55.1
- os/type: darwin
- os/arch: amd64
- go/version: go1.16.3
- go/linking: dynamic
- go/tags: cmount

Which OS you are using and how many bits (eg Windows 7, 64 bit)

macOS, 64bit

Which cloud storage system are you using? (eg Google Drive)

OneDrive and local

The command you were trying to run (eg rclone copy /tmp remote:tmp)

See narrative below. It is answered inline

The rclone config contents with secrets removed.

[onedrive]
type = onedrive
token = {"access_token":"REDACTED","expiry":"2021-05-29T12:07:40.786469-06:00"}
drive_id = REDACTED
drive_type = personal
client_id = REDACTED
client_secret = REDACTED

A log from the command with the -vv flag

Again, I do answer this. See narrative below

Question Narrative

Sorry these names are really long. They are the result of my tool constantly seeing a conflict from this behavior but everything I do below is not using that tool.

First, the file I am copying (pipped to jq to view it better)

$ rclone lsjson --hash "onedrive:Pictures/Camera Roll/2021/05/20210528_174653713_iOS.jpg.20210529T072120.B.20210529T094856.A.20210529T101900.B"|jq

[
  {
    "Path": "20210528_174653713_iOS.jpg.20210529T072120.B.20210529T094856.A.20210529T101900.B",
    "Name": "20210528_174653713_iOS.jpg.20210529T072120.B.20210529T094856.A.20210529T101900.B",
    "Size": 5737844,
    "MimeType": "application/octet-stream",
    "ModTime": "2021-05-28T20:19:10Z",
    "IsDir": false,
    "Hashes": {
      "SHA-1": "59b96292e740d09c078a8650cdf4270b01bfbf41"
    },
    "ID": "1af35bc3b364a58c#1AF35BC3B364A58C!295381"
  }
]

That is just for reference. Now, let's download it and call it testfile (where this is a "local" remote)

$ rclone -vv copyto "onedrive:Pictures/Camera Roll/2021/05/20210528_174653713_iOS.jpg.20210529T072120.B.20210529T094856.A.20210529T101900.B" testfile

2021/05/29 11:14:18 DEBUG : Using config file from "/Users/<USER>/.config/rclone/rclone.conf"
2021/05/29 11:14:18 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "-vv" "copyto" "onedrive:Pictures/Camera Roll/2021/05/20210528_174653713_iOS.jpg.20210529T072120.B.20210529T094856.A.20210529T101900.B" "testfile"]
2021/05/29 11:14:18 DEBUG : Creating backend with remote "onedrive:Pictures/Camera Roll/2021/05/20210528_174653713_iOS.jpg.20210529T072120.B.20210529T094856.A.20210529T101900.B"
2021/05/29 11:14:21 DEBUG : fs cache: adding new entry for parent of "onedrive:Pictures/Camera Roll/2021/05/20210528_174653713_iOS.jpg.20210529T072120.B.20210529T094856.A.20210529T101900.B", "onedrive:Pictures/Camera Roll/2021/05"
2021/05/29 11:14:21 DEBUG : Creating backend with remote "."
2021/05/29 11:14:21 DEBUG : fs cache: renaming cache item "." to be canonical "/Users/<USER>/OneDrive/Pictures/Camera Roll/2021/05"
2021/05/29 11:14:22 DEBUG : 20210528_174653713_iOS.jpg.20210529T072120.B.20210529T094856.A.20210529T101900.B: Need to transfer - File not found at Destination
2021/05/29 11:14:24 DEBUG : 20210528_174653713_iOS.jpg.20210529T072120.B.20210529T094856.A.20210529T101900.B: SHA-1 = 59b96292e740d09c078a8650cdf4270b01bfbf41 OK
2021/05/29 11:14:24 INFO  : 20210528_174653713_iOS.jpg.20210529T072120.B.20210529T094856.A.20210529T101900.B: Copied (new) to: testfile
2021/05/29 11:14:24 INFO  :
Transferred:   	    2.950M / 2.950 MBytes, 100%, 1.487 MBytes/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:        11.0s

2021/05/29 11:14:24 DEBUG : 9 go routines active

Now look at the file locally

$ rclone lsjson --hash testfile|jq

[
  {
    "Path": "testfile",
    "Name": "testfile",
    "Size": 3093417,
    "MimeType": "application/octet-stream",
    "ModTime": "2021-05-28T14:19:10-06:00",
    "IsDir": false,
    "Hashes": {
      "CRC-32": "49bed3c6",
      "DropboxHash": "5b54e1925097863431505c2f70c49d3be14603d175b26fda594296a92a421aaf",
      "MD5": "4d78b4d0f8eae50d23a5dfd055e58022",
      "MailruHash": "7f28b86f801c890e2dffe3bc465b06bbf8a2633d",
      "QuickXorHash": "04215aae46464286ef6e5b768e8cf1f7aa7f91f0",
      "SHA-1": "59b96292e740d09c078a8650cdf4270b01bfbf41",
      "Whirlpool": "43d928d800b1d71028324c5824bf26fd51efb8625b721db7667aaa0ce156c4d16732592513ce3a62e168dd6293b171cc7dbc615d7e56c0897ca0022d18765f41"
    }
  }
]

Just as a sanity check

$ sha1sum testfile
59b96292e740d09c078a8650cdf4270b01bfbf41  testfile
$ ls -l testfile
-rw-r--r--  1 <USER>  staff  3093417 May 28 14:19 testfile

The hashes agree but the file sizes most certainly do not. I suspect OneDrive is reporting the wrong file size. This is likely not an rclone issue and rather a OneDrive issue but I thought I'd ask here first. And of course, you can use --checksum for sync but that adds considerable effort for a local remote where hashes are not otherwise stored. (not to mention, you can't do it if you wrap OneDrive with crypt)

Any thoughts?

2

what does onedrive website report?

3

It's documented on the OneDrive rclone page:

https://rclone.org/onedrive/#unexpected-file-size-hash-differences-on-sharepoint

4

I saw that but it’s for sharepoint, and for office files. This is jpeg and OneDrive Personal

5

How did the files get to OneDrive in the first place? On my Android phone I'm using the OneDrive app to auto-upload, and the last picture I took doesn't show the problem...

% rclone lsjson --hash OneDriveSweharris:Pictures/Camera\ Roll/PXL_20210519_203447135.jpg | jq .
[
  {
    "Path": "PXL_20210519_203447135.jpg",
    "Name": "PXL_20210519_203447135.jpg",
    "Size": 4916396,
    "MimeType": "image/jpeg",
    "ModTime": "2021-05-19T20:34:47Z",
    "IsDir": false,
    "Hashes": {
      "SHA-1": "82ab47fa6251d5daf888a93f147b2353509c22da"
    },
    "ID": "19731fbe049a1fd2#19731FBE049A1FD2!2292"
  }
]
% rclone copy OneDriveSweharris:Pictures/Camera\ Roll/PXL_20210519_203447135.jpg .
% ls -l PXL_20210519_203447135.jpg 
-rw-r--r-- 1 sweh sweh  4916396 May 19 16:34 PXL_20210519_203447135.jpg

I believe OneDrive can cache data and have consistency issues for a small amount of time (it's why I think large uploads sometimes cause a failure and retry) but I've never seen that last for any major period of time.

6

Interesting. I am 99.9% sure those were uploaded with the iOS auto upload. And then moved using rclone move a few times from the false-conflict

Either way, your experience adds to my suspicions that this is not an rclone issue but a OneDrive one.

I wonder how much it happens. Maybe the warning on the docs need to be expanded from just sharepoint.

Thanks!

7

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.