OneDriver Personal filesizes are wrong?

Help and Support
1

Preface

I am less asking for help and more asking to know whether this has happened before. Trying to isolate and make a single usable example is doable but not worth the time if I don't need to.

What is the problem you are having with rclone?

This may or may not be rclone. It may be OneDrive's API but....

File sizes returned in lsjson do not agree with the file when I download the file. (hashes do agree!!!)

What is your rclone version (output from rclone version)

rclone v1.52.3
- os/arch: darwin/amd64
- go version: go1.14.7

Which OS you are using and how many bits (eg Windows 7, 64 bit)

macOS Catalina

Which cloud storage system are you using? (eg Google Drive)

OneDrive

The rest of the template doesn't really follow.

I am working on my rclone sync tool and trying to debug why some files are getting transfered.

I finally found one of the issues (maybe more but they are on me, not rclone). lsjson reports

{"Path":"01_david-pam_11-21-04.jpg","Name":"01_david-pam_11-21-04.jpg","Size":25731,"MimeType":"image/jpeg","ModTime":"2004-11-24T16:14:28Z","IsDir":false,"Hashes":{"SHA-1":"29513dd3c1e03d4ce5e48b8ddbfbbaa046b0521e"},"ID":"1af35bc3b364a58c#1AF35BC3B364A58C!15902"}

but when I download the file, or compare my local one, I get the following

{"Path":"01_david-pam_11-21-04.jpg","Name":"01_david-pam_11-21-04.jpg","Size":19238,"MimeType":"image/jpeg","ModTime":"2004-11-24T09:14:28.000000000-07:00","IsDir":false,"Hashes":{"CRC-32":"2e0a38b9","DropboxHash":"a2a87847aaebfca14ccb6fa59b0491f0d10b64862a87cd2020aeb31d9a643d7d","MD5":"0ded0f21ba426723356661b1b38c1720","MailruHash":"1bf40e03f79e05cc21649e1bc0bb75bd2d95fab0","QuickXorHash":"7ece4fb98be1343ab115b0f759138545a289c9f1","SHA-1":"29513dd3c1e03d4ce5e48b8ddbfbbaa046b0521e","Whirlpool":"8fe3bdddf974385559491ebc50c4821503965969e88aab1266b30b6818c041a60ec3c00a4420f4456cea78764894948529b5bba3c8f815ab4e89d6659a48f462"}},

The docs say that file sizes are a common issue with SharePoint but this is OneDrive personal.

Has anyone seen this before?

(again, not asking for help per se. I know what flags to use to get around it. Just wondering if this is an understood problem)

2

hi,
that is very strange as the sha-1 checksums match.

what does the local file manager state the file size is?

when you posted, you should have been asked for more information.

  • debug log documenting the problem. see the upload and download details including sizes and hashes
  • the config file, redacting passwords and ids
3

I answered that they weren't relevant as the call is just lsjson. I am using rclone for another purpose where, for now, I am just working on the lsdata. So a debug log doesn't make sense as it is literally just listing the files. There is no error per se.

but if you insist,

$ rclone -vvv lsjson --hash onedrive:Pictures/Photos/Unsorted/albums_from_winokur_us/david-pam_wedding
2020/08/25 16:56:26 DEBUG : rclone: Version "v1.52.3" starting with parameters ["rclone" "-vvv" "lsjson" "--hash" "onedrive:Pictures/Photos/Unsorted/albums_from_winokur_us/david-pam_wedding"]
2020/08/25 16:56:26 DEBUG : Using config file from "/Users/jgwinok/.config/rclone/rclone.conf"

...JSON output...

2020/08/25 16:56:29 DEBUG : 7 go routines active

and

[onedrive]
type = onedrive
token = {"access_token":"...REDACTED...","expiry":"2020-08-25T17:53:01.887186-06:00"}
drive_id = ...REDACTED...
drive_type = personal

To answer your other question, macOS finder say 19kb (note that macOS has decided that a byte is 1000 bits which pisses me off). ls -l says 19238

4

what i was suggesting that a debug log of the upload/download of the files, to see the file sizes during transfer.

5

what i was suggesting that a debug log of the upload/download of the files, to see the file sizes during transfer.

Ahh. Sorry. I didn't get that. (also sorry if my response sounded snarky. Having a shitty day otherwise and you replied before I could edit it to make me sound like less of an ass)

They were uploaded with the OneDrive macOS client, not rclone. I would be really surprised if this is an rclone bug and not a OneDrive bug. I guess the only thing from rclone's side would be to note that this can happen....

6

thanks, no worries.

i see that you are writing a sync tool in python.
when i was writing my sync tool in python, i would get snarky all the time.

maybe there is a bug with onedrive macos client.
but what is really strange is that the sha1 checksums match, but not file size

7

Yeah. It is very odd. I have something like 3000 files like that out of 85,000.

On the plus side, it exposed another edge case in my code I didn't account for (matching mtimes and mismatches sizes on a new, non-previously synced directory). It's not a major bug as I am super careful that every operation starts with a backup.

Part of why I tried this was because I was having issues with the macOS client but my other computer is so old and slow that it's hard to know what's what.

I wish I could reproduce it. When I upload the offending file with rclone it gets it right.

8

I'm pretty sure we've seen this before.

I wonder if that is the same bug...

9

Wow. Old find.

It seems plausible that it’s the same. Though it was on files uploaded via their first-party client. I guess that’s not immune if it uses the same or similar API/process.

Minor rant but, why the hell is OneDrive Support asking you to use user voice to get something fixed? I see the utility in prioritization of bugs but this feels backwards!

/rant

Anyway, not sure exactly why rclone copy didn’t like replacing them either. I suspect that OneDrive’s side didn’t update since rclone did try and then returned errors on mismatched sizes.

What I did was use lsjson on both sides then wrote a simple python script to tell me all files where they were on both but the sizes didn’t match. Then I dumped that to a file and used rclone delete with files-from to delete them on OneDrive. Then I used copy to push them back up.

This was all necessitated since the OneDrive client on my (admittedly old and underpowered) Mac has just shit the bed and can’t do anything (and OneDrive support has been pretty useless). Given that I don’t use office at home, I am considering dumping OneDrive for GDrive or Dropbox. Rclone makes it so easy to move everything over from a VPS to seed!

Thanks

10

My thoughts exactly!

Sometimes they respond to bugs reported on the GitHub repo for the API docs.

Looks like you fixed the problem though :crossed_fingers:

11

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.