Can’t update certain files with certain modtime on OneDrive

Suspected Bug
1

What is the problem you are having with rclone?

rclone can't update files with certain content with certain modification times on OneDrive.

Run the command 'rclone version' and share the full output of the command.

rclone v1.64.0
- os/version: darwin 14.0 (64 bit)
- os/kernel: 23.0.0 (arm64)
- os/type: darwin
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.21.1
- go/linking: dynamic
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

OneDrive Personal

The command you were trying to run (eg rclone copy /tmp remote:tmp)

touch -t 2309290000 file_v1.bin
touch -t 2309290001 file_v2.bin
rclone copyto file_v1.bin data:test/1 -vv --dump=responses --retries=1 &>log1.txt
rclone copyto file_v2.bin data:test/1 -vv --dump=responses --retries=1 &>log2.txt

file_v1.bin (5.5 KB)
file_v2.bin (5.5 KB)

(Added .txt extension to upload.)

The rclone config contents with secrets removed.

[data]
type = onedrive
client_id = <my client id>
client_secret = <my client secret>
drive_id = <my drive id>
drive_type = personal
token = <my token>

A log from the command with the -vv flag

2023/09/29 15:21:09 ERROR : file_v2.bin: Failed to copy: invalidRequest: Bad Argument
2023/09/29 15:21:09 ERROR : Attempt 1/1 failed with 1 errors and: invalidRequest: Bad Argument
2023/09/29 15:21:09 Failed to copyto: invalidRequest: Bad Argument

log1.txt (18.0 KB)
log2.txt (13.1 KB)

Additional info

Issue doesn't reproduce if modification time of those files is the same or if I upload files with simple text content.

The only similar issue I found is:

--backup-dir workaround also helped in my case.

2

The error comes on a totally innocent looking transaction.

My money is on a OneDrive bug. Rclone users have found several of those over the years.

Did you check the OneDrive GitHub?

I haven't tried to reproduce this yet though.

3

There are 15 issues for the Issues · OneDrive/onedrive-api-docs · GitHub query but all of them are closed...
I quickly skimmed over them, this one seems to be the most similar: Intermittent 400 Bad Request response for createUploadSession · Issue #1064 · OneDrive/onedrive-api-docs · GitHub
It's closed too, but there's comment at the end:

I have similar issue at this moment.
I found reason of this.
You (and me) set only lastModifiedDateTime.
Just add CreatedDateTime.

Though I checked and rclone does set CreatedDateTime in onedrive backend.

Can anybody else reproduce it with those files on their OneDrive?

4

I did manage to replicate this!

I experimented further, and as far as I can see the modtimes of the files don't matter its the contents that matter, and in particular we must be overwriting file_v1.bin with file_v2.bin. You can upload file_v1.bin and file_v2.bin on their own just fine, and also overwrite file_v2.bin with file_v1.bin just not the other way round.

Eg

$ rclone delete onedrive:test/1
$ rclone copyto file_v1.bin onedrive:test/1 --retries 1
$ rclone copyto file_v2.bin onedrive:test/1 --retries 1
2023/10/01 12:10:38 ERROR : file_v2.bin: Failed to copy: invalidRequest: Bad Argument
2023/10/01 12:10:38 ERROR : Attempt 1/1 failed with 1 errors and: invalidRequest: Bad Argument
2023/10/01 12:10:38 Failed to copyto: invalidRequest: Bad Argument

$ rclone delete onedrive:test/1
$ rclone copyto file_v2.bin onedrive:test/1 --retries 1
$ rclone copyto file_v1.bin onedrive:test/1 --retries 1
$ rclone copyto file_v2.bin onedrive:test/1 --retries 1
2023/10/01 12:10:59 ERROR : file_v2.bin: Failed to copy: invalidRequest: Bad Argument
2023/10/01 12:10:59 ERROR : Attempt 1/1 failed with 1 errors and: invalidRequest: Bad Argument
2023/10/01 12:10:59 Failed to copyto: invalidRequest: Bad Argument

$ rclone delete onedrive:test/1
$ touch file_v1.bin 
$ touch file_v2.bin 
$ rclone copyto file_v2.bin onedrive:test/1 --retries 1
$ rclone copyto file_v1.bin onedrive:test/1 --retries 1
$ rclone copyto file_v2.bin onedrive:test/1 --retries 1
2023/10/01 12:12:44 ERROR : file_v2.bin: Failed to copy: invalidRequest: Bad Argument
2023/10/01 12:12:44 ERROR : Attempt 1/1 failed with 1 errors and: invalidRequest: Bad Argument
2023/10/01 12:12:44 Failed to copyto: invalidRequest: Bad Argument

I thought maybe the files might have the same quickxor hash but they don't. They only differ by one bit, so I guess it makes sense that the quickxor hash only differs by one bit.

$ rclone hashsum quickxor . --include "*.bin"
000000000000000000000020e015000000000000  file_v1a.bin
000000000000000000000000e015000000000000  file_v2a.bin

I tried to replicate this with onedrive business and it did not replicate. It worked just fine there!

So I can't think this is an rclone bug.

I couldn't find a relevant issue in the onedrive github. I've had good success reporting issues there in the past and getting them fixed. Can we narrow down the exact kind of file that causes this problem?

5

Nice!

I couldn't reproduce it when modtime was the same. Caused me a lot of trouble to reproduce on another machine initially because I just cp'ed files and that changed modtime to be about the same. Haha.

Interesting.

Do you mean to try with smaller files with similar structure? I think reporting those example files as-is is fine too.
It's from Time Machine sparsebundle backup image btw. There're a lot of files like that there which are mostly empty (0xFF bytes), then change by few bytes and that cause problem with updating them on OneDrive.

Do you want me to report that to OneDrive repo?

6

If you are happy to do that, that would be great.

I'd do logs with --dump bodies and edit out the binary data by hand - the logs you posted are missing the parameters to the start upload call.

Note that it works on OneDrive business too!

If you cc me @ncw I'll follow along too!

7

Created Can't update file with certain content on OneDrive Personal · Issue #1734 · OneDrive/onedrive-api-docs · GitHub

1 Like
8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.