Errors uploading to onedrive

Suspected Bug
1

What is the problem you are having with rclone?

invalid request uploading to OneDrive with obfuscation

Run the command 'rclone version' and share the full output of the command.

rclone v1.68.2
- os/version: rocky 8.10 (64 bit)
- os/kernel: 4.18.0-553.22.1.el8_10.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.23.3
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

OneDrive personal

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone --log-level DEBUG --dump-bodies sync . OneDriveMusic:mp3/SONGS/ALBUM/Pretenders/The_Singles --modify-window 1s

The rclone config contents with secrets removed.

[OneDriveSweharris3]
type = onedrive
token = {"access_token":"blahb..."}
drive_id = XXX
drive_type = personal

[OneDriveMusic]
type = crypt   
remote = OneDriveSweharris3:Music
filename_encryption = obfuscate
password = XXX
password2 =

A log from the command with the -vv flag

This is the relevant section of the debug log

2024/12/05 11:40:49 DEBUG : 13.Dont_Get_Me_Wrong.mp3: Need to transfer - File not found at Destination
2024/12/05 11:40:48 DEBUG : 97.35.SDCI_VtI_bt_lGDCv.BE5: Starting multipart upload
2024/12/05 11:40:48 DEBUG : Encrypted drive 'OneDriveMusic:mp3/SONGS/ALBUM/Pretenders/The_Singles': Waiting for checks to finish
2024/12/05 11:40:48 DEBUG : Encrypted drive 'OneDriveMusic:mp3/SONGS/ALBUM/Pretenders/The_Singles': Waiting for transfers to finish
2024/12/05 11:40:48 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/12/05 11:40:48 DEBUG : HTTP REQUEST (req 0xc00029f7c0)
2024/12/05 11:40:48 DEBUG : POST /v1.0/drives/B3C8B207F224B0AF/items/B3C8B207F224B0AF!21651:/97.35.SDCI_VtI_bt_lGDCv.BE5:/createUploadSession HTTP/1.1^M
Host: graph.microsoft.com^M
User-Agent: rclone/v1.68.2^M
Content-Length: 124^M
Authorization: XXXX
Content-Type: application/json^M
Accept-Encoding: gzip^M
^M
{"item":{"fileSystemInfo":{"createdDateTime":"2022-10-04T21:24:26.485Z","lastModifiedDateTime":"2022-10-04T21:24:26.485Z"}}}
2024/12/05 11:40:48 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/12/05 11:40:48 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/12/05 11:40:48 DEBUG : HTTP RESPONSE (req 0xc00029f7c0)
2024/12/05 11:40:48 DEBUG : HTTP/2.0 400 Bad Request^M
Cache-Control: no-store, no-cache^M
Client-Request-Id: cd1f8b6e-02b2-442e-b803-e39127f59561^M
Content-Type: application/json^M
Date: Thu, 05 Dec 2024 16:40:47 GMT^M
Request-Id: cd1f8b6e-02b2-442e-b803-e39127f59561^M
Strict-Transport-Security: max-age=31536000^M
Vary: Accept-Encoding^M
X-Ms-Ags-Diagnostic: {"ServerInfo":{"DataCenter":"East US","Slice":"E","Ring":"5","ScaleUnit":"001","RoleInstance":"MN1PEPF00010855"}}^M
^M
{"error":{"code":"invalidRequest","message":"Invalid request","innerError":{"date":"2024-12-05T16:40:48","request-id":"cd1f8b6e-02b2-442e-b803-e39127f59561","client-request-id":"cd1f8b6e-02b2-442e-b803-e39127f59561"}}}
2024/12/05 11:40:48 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/12/05 11:40:48 DEBUG : info from Update error: 
null
2024/12/05 11:40:48 ERROR : 13.Dont_Get_Me_Wrong.mp3: Failed to copy: invalidRequest: Invalid request

Now what was interesting is last night's backup showed an error where the file in question appears to have been corrupted somehow

2024/12/05 09:33:06 INFO  : SONGS/ALBUM/Pretenders/The_Singles/13.Dont9Me_Wrong.mp3: Deleted
2024/12/05 09:40:40 ERROR : SONGS/ALBUM/Pretenders/The_Singles/13.Dont_Get_Me_Wrong.mp3: Failed to copy: invalidRequest: Invalid request

Notice the filename is wrong "Dont9Me" ?! And the file now won't upload.

So I worked out the minimal command (listed above) to replicate this so it doesn't go through the 8000+ files.

The obfuscated filename looks sane, so I'm not sure what's going on, here!

Now I have another sync which also threw up similar errors and I notice something odd about them all; they have _Get_ in the name!

eg

2024/12/05 10:05:49 INFO  : abeegees_thevery/06._I've_Gotta9A_Message_To_You.flac: Deleted
2024/12/05 10:12:04 INFO  : cvar_topofthepops2~70s.2/01._Won't3Fooled_Again.flac: Deleted
2024/12/05 10:12:06 INFO  : cvar_wowthatwasthe~70s7/03._Couldn't6It_Right.flac: Deleted
2024/12/05 10:12:27 ERROR : abeegees_thevery/06._I've_Gotta_Get_A_Message_To_You.flac: Failed to copy: invalidRequest: Invalid request
2024/12/05 10:12:55 ERROR : cvar_topofthepops2~70s.2/01._Won't_Get_Fooled_Again.flac: Failed to copy: invalidRequest: Invalid request
2024/12/05 10:12:56 ERROR : cvar_wowthatwasthe~70s7/03._Couldn't_Get_It_Right.flac: Failed to copy: invalidRequest: Invalid request

This also showed more oddities, but it fixed itself

2024/12/05 10:05:55 INFO  : ahookedon_hookedon/07._Hooked_on_Rodgers_ _Hammerstein.flac: Deleted
2024/12/05 10:05:56 INFO  : ahookedon_hookedon/03._Hooked_on_Classics_Part_1 2.flac: Deleted
2024/12/05 10:12:40 INFO  : ahookedon_hookedon/07._Hooked_on_Rodgers_+_Hammerstein.flac: Copied (new)
2024/12/05 10:12:41 INFO  : ahookedon_hookedon/03._Hooked_on_Classics_Part_1+2.flac: Copied (new)

So the "+" character needed help.

And, yes, it seems this is on the "new" Onedrive backend. sigh.

2

Oh, ho.... That is sounding very much like a OneDrive API bug.

Looks like the + escaping is broken.

All this has worked in rclone for ages so I suspect breakage at Microsoft. I just hope they mend it again soon!

3

I initially thought that, but...

The bit that confuses me is that OneDrive doesn't get to see the filename 'cos it's obfuscated. So the string _Get_ is only seen by rclone. Very odd.

Hmm, looking at those three files, even though the obfuscation distance is different, that string becomes _VtI_ each time..

13.Dont_Get_Me_Wrong.mp3         97.35.SDCI_VtI_bt_lGDCv.BE5
01._Won't_Get_Fooled_Again.flac  247.89._lDC'I_VtI_UDDAts_PvpxC.uApr
03._Couldn't_Get_It_Right.flac   172.58._RDJAsC'I_VtI_XI_gxvwI.uApr

And that explains it!

WHAT THE HELL.

More Microsoft breakage caused by changing backends!

4

I don't use rclone with OneDrive, but that one has been around forever as that's always been a Sharepoint thing. Praise be MS.

The obfuscation probably has to take into account those restrictions? :slight_smile:

5

The problem is that Microsoft never had one implementation of Onedrive; personal Onedrive had a different backend and different restrictions and this was never a problem.

But now it looks like they're trying to rationalise and normalise their technology stacks; personal onedrive is being migrated to a new backend that has different performance characteristics (we hit this recently with timestamps; it used to record milliseconds, now it's just seconds).

So stuff that has worked for years may now break.

Not the obfuscation layer because it shouldn't care what the underlying system is (onedrive, google, localfiles, sftp... it shouldn't matter). The onedrive backend is the place where "impossible" files should be "kludged".

6

Oh right, that would be the spot. I just recall battling with vti for years in SP...

7

My thoughts precisely!

I Didn't realise microsoft actually documented these limitations - I'm sure they didn't as the forms one came up on the forum a while back and also in issue #7910 and I'm pretty sure it wasn't documented then.

Do you think we should put this in the onedrive docs @sweh ? Probably in this section - fancy sending a PR?

Intersting insight from @Animosity022 about the changes moving us towards sharepoint / onedrive for business (which I think are already the same thing).

8

Hmm. I'm more wondering if we can work around this. It's not going to be predictable for the obfuscated use case, so documenting it won't necessarily help.

Ideally we should have the onedrive backend be able to detect if a filename would break things and rewrite it to avoid the issue, and on reading back it would undo the rewrite. So it wouldn't quite have the right name in onedrive, but it would look clean to rclone users. (This would also then allow for odd characters like a : to be used).

(Done correctly this could be a routine used by any backend, but I dunno if any other storage is as stupid as Microsoft!)