Purge command fails on OneDrive

BeansIsFat · September 11, 2020, 8:40pm

What is the problem you are having with rclone?

I am unable to use the purge command. The error indicates that my token contains an invalid signature. I edited the config to refresh the token and got the same error.

What is your rclone version (output from `rclone version`)

rclone v1.53.0
also tried on rclone v1.54.0-beta.4768.6a56ac103 with the same error

Which OS you are using and how many bits (eg Windows 7, 64 bit)

MacOS 10.14.6

Which cloud storage system are you using? (eg Google Drive)

OneDrive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone purge onedrive:temp

The rclone config contents with secrets removed.

[onedrive]
type = onedrive
token = {"access_token":"redacted","expiry":"2020-09-11T17:18:16.634655-04:00"}
drive_id = redacted
drive_type = business

A log from the command with the `-vv` flag

2020/09/11 16:20:57 DEBUG : rclone: Version "v1.53.0" starting with parameters ["rclone" "purge" "-vv" "onedrive:temp"]
2020/09/11 16:20:57 DEBUG : Creating backend with remote "onedrive:temp"
2020/09/11 16:20:57 DEBUG : Using config file from "/Users/BeansIsFat/.config/rclone/rclone.conf"
2020/09/11 16:21:00 ERROR : Attempt 1/3 failed with 1 errors and: unauthenticated: invalidSignature: Token contains invalid signature.
2020/09/11 16:21:00 ERROR : Attempt 2/3 failed with 1 errors and: unauthenticated: invalidSignature: Token contains invalid signature.
2020/09/11 16:21:00 ERROR : Attempt 3/3 failed with 1 errors and: unauthenticated: invalidSignature: Token contains invalid signature.
2020/09/11 16:21:00 DEBUG : 8 go routines active
2020/09/11 16:21:00 Failed to purge: unauthenticated: invalidSignature: Token contains invalid signature.

ncw · September 11, 2020, 9:47pm

Try

rclone config reconnect onedrive:

To refresh the token

BeansIsFat · September 11, 2020, 10:04pm

I tried that and got the same error after successfully refreshing the token

ncw · September 11, 2020, 10:35pm

Is it just purge that fails? What about rclone lsf onedrive: ?

BeansIsFat · September 11, 2020, 11:12pm

Yes, it's just the purge that fails. rclone lsf onedrive: works fine.

I do have another problem with copy or sync and changed files on this remote but I figured I'd solve this problem first as they don't seem related.

gulp79 · September 12, 2020, 2:48pm

same for me...

C:\Users\me>"D:\Portable\PortableApps\rclone\rclone.exe" -vv lsf onecrypt:Backup/Documents
2020/09/12 16:45:23 DEBUG : rclone: Version "v1.54.0-beta.4766.75de30cfa" starting with parameters ["D:\\Portable\\PortableApps\\rclone\\rclone.exe" "-vv" "lsf" "onecrypt:Backup/Documents"]
2020/09/12 16:45:23 DEBUG : Using config file from "D:\\Portable\\PortableApps\\rclone\\rclone.conf"
2020/09/12 16:45:23 DEBUG : Creating backend with remote "onecrypt:Backup/Documents"
2020/09/12 16:45:23 DEBUG : Creating backend with remote "onedrive:/Crypt/86.MlnvFA/178.Grfxphqwv"
2020/09/12 16:45:24 DEBUG : fs cache: renaming cache item "onedrive:/Crypt/86.MlnvFA/178.Grfxphqwv" to be canonical "onedrive:Crypt/86.MlnvFA/178.Grfxphqwv"
2020/09/12 16:45:24 DEBUG : fs cache: switching user supplied name "onedrive:/Crypt/86.MlnvFA/178.Grfxphqwv" for canonical name "onedrive:Crypt/86.MlnvFA/178.Grfxphqwv"
M&G/
Programmi&Guide&Utility/
2020/09/12 16:45:25 DEBUG : 5 go routines active

C:\Users\me>"D:\Portable\PortableApps\rclone\rclone.exe" -vv purge onecrypt:Backup/Documents
2020/09/12 16:45:35 DEBUG : rclone: Version "v1.54.0-beta.4766.75de30cfa" starting with parameters ["D:\\Portable\\PortableApps\\rclone\\rclone.exe" "-vv" "purge" "onecrypt:Backup/Documents"]
2020/09/12 16:45:35 DEBUG : Creating backend with remote "onecrypt:Backup/Documents"
2020/09/12 16:45:35 DEBUG : Using config file from "D:\\Portable\\PortableApps\\rclone\\rclone.conf"
2020/09/12 16:45:35 DEBUG : Creating backend with remote "onedrive:/Crypt/86.MlnvFA/178.Grfxphqwv"
2020/09/12 16:45:36 DEBUG : fs cache: renaming cache item "onedrive:/Crypt/86.MlnvFA/178.Grfxphqwv" to be canonical "onedrive:Crypt/86.MlnvFA/178.Grfxphqwv"
2020/09/12 16:45:36 DEBUG : fs cache: switching user supplied name "onedrive:/Crypt/86.MlnvFA/178.Grfxphqwv" for canonical name "onedrive:Crypt/86.MlnvFA/178.Grfxphqwv"
2020/09/12 16:45:37 ERROR : Attempt 1/3 failed with 1 errors and: unauthenticated: invalidSignature: Token contains invalid signature.
2020/09/12 16:45:37 ERROR : Attempt 2/3 failed with 1 errors and: unauthenticated: invalidSignature: Token contains invalid signature.
2020/09/12 16:45:38 ERROR : Attempt 3/3 failed with 1 errors and: unauthenticated: invalidSignature: Token contains invalid signature.
2020/09/12 16:45:38 DEBUG : 6 go routines active
2020/09/12 16:45:38 Failed to purge: unauthenticated: invalidSignature: Token contains invalid signature.

C:\Users\me>"D:\Portable\PortableApps\rclone\rclone.exe" -vv purge onedrive:/Crypt/86.MlnvFA/178.Grfxphqwv
2020/09/12 16:47:40 DEBUG : rclone: Version "v1.54.0-beta.4766.75de30cfa" starting with parameters ["D:\\Portable\\PortableApps\\rclone\\rclone.exe" "-vv" "purge" "onedrive:/Crypt/86.MlnvFA/178.Grfxphqwv"]
2020/09/12 16:47:40 DEBUG : Creating backend with remote "onedrive:/Crypt/86.MlnvFA/178.Grfxphqwv"
2020/09/12 16:47:40 DEBUG : Using config file from "D:\\Portable\\PortableApps\\rclone\\rclone.conf"
2020/09/12 16:47:41 DEBUG : fs cache: renaming cache item "onedrive:/Crypt/86.MlnvFA/178.Grfxphqwv" to be canonical "onedrive:Crypt/86.MlnvFA/178.Grfxphqwv"
2020/09/12 16:47:42 ERROR : Attempt 1/3 failed with 1 errors and: unauthenticated: invalidSignature: Token contains invalid signature.
2020/09/12 16:47:42 ERROR : Attempt 2/3 failed with 1 errors and: unauthenticated: invalidSignature: Token contains invalid signature.
2020/09/12 16:47:42 ERROR : Attempt 3/3 failed with 1 errors and: unauthenticated: invalidSignature: Token contains invalid signature.
2020/09/12 16:47:42 DEBUG : 6 go routines active
2020/09/12 16:47:42 Failed to purge: unauthenticated: invalidSignature: Token contains invalid signature.

ncw · September 12, 2020, 5:16pm

Strange!

Can you do a purge with

-vv --dump bodies --retries 1

So we can see the http transactions and post the result

BeansIsFat · September 13, 2020, 2:57am

Here's a link to the output of purge using

rclone purge --vv --dump bodies --retries 1 onedrive:temp

gist.github.com

https://gist.github.com/BeansIsFat/e2a4f5407ac09ce3db5aabb947aca465

gistfile1.txt

2020/09/12 22:44:35 DEBUG : rclone: Version "v1.53.0" starting with parameters ["rclone" "purge" "-vv" "--dump" "bodies" "--retries" "1" "onedrive:temp"]
2020/09/12 22:44:35 DEBUG : Creating backend with remote "onedrive:temp"
2020/09/12 22:44:35 DEBUG : Using config file from "/Users/redacted/.config/rclone/rclone.conf"
2020/09/12 22:44:35 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2020/09/12 22:44:35 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/09/12 22:44:35 DEBUG : HTTP REQUEST (req 0xc000133200)
2020/09/12 22:44:35 DEBUG : GET /v1.0/drives/b!-Jr3ZPe__kCZ7xLdl2FMoffWFSmIUF1AoWygTwxjtywWlD7heAFZQpK_8QbDA7ff/root HTTP/1.1
Host: graph.microsoft.com
User-Agent: rclone/v1.53.0
Authorization: XXXX

This file has been truncated. show original

gulp79 · September 13, 2020, 8:51pm

i've been able to remove the directory by mounting onedrive on linux (not tested on windows) and doing an "rm -rf" of target direcory. In this way, i've been able to get rid of it.
(i've also tested with onedrive web frontend, but is impossible to delete non empty dir on web, and i've a lot of files in subdir, so was not an option to do it manually on web )

ncw · September 14, 2020, 3:06pm

I can see what is happening there...

Rclone issues the DELETE on the directory

2020/09/12 22:44:36 DEBUG : HTTP REQUEST (req 0xc00066b300)
2020/09/12 22:44:36 DEBUG : DELETE /v1.0/drives/b!-Jr3ZPe__kCZ7xLdl2FMoffWFSmIUF1AoWygTwxjtywWlD7heAFZQpK_8QbDA7ff/items/01HSWNV2LEGZAH5A2BLJE3R3I4ACF3QDRK HTTP/1.1
Host: graph.microsoft.com
User-Agent: rclone/v1.53.0
Authorization: XXXX
Accept-Encoding: gzip

The server responds with a redirect

2020/09/12 22:44:36 DEBUG : HTTP RESPONSE (req 0xc00066b300)
2020/09/12 22:44:36 DEBUG : HTTP/1.1 302 Found
Cache-Control: private
Client-Request-Id: 9f245298-ad4b-4c78-b6c3-478208784d04
Content-Type: text/plain
Date: Sun, 13 Sep 2020 02:44:35 GMT
Location: https://redacted-my.sharepoint.com/_vti_bin/client.svc/v2.0/drives('b!-Jr3ZPe__kCZ7xLdl2FMoffWFSmIUF1AoWygTwxjtywWlD7heAFZQpK_8QbDA7ff')/items/01HSWNV2LEGZAH5A2BLJE3R3I4ACF3QDRK
Request-Id: 9f245298-ad4b-4c78-b6c3-478208784d04
Strict-Transport-Security: max-age=31536000
X-Ms-Ags-Diagnostic: {"ServerInfo":{"DataCenter":"North Central US","Slice":"SliceC","Ring":"3","ScaleUnit":"003","RoleInstance":"AGSFE_IN_30"}}
Content-Length: 0

So rclone does the request at that URL

2020/09/12 22:44:36 DEBUG : HTTP REQUEST (req 0xc0004be400)
2020/09/12 22:44:36 DEBUG : GET /_vti_bin/client.svc/v2.0/drives('b!-Jr3ZPe__kCZ7xLdl2FMoffWFSmIUF1AoWygTwxjtywWlD7heAFZQpK_8QbDA7ff')/items/01HSWNV2LEGZAH5A2BLJE3R3I4ACF3QDRK HTTP/1.1
Host: redacted-my.sharepoint.com
User-Agent: rclone/v1.53.0
Authorization: XXXX
Referer: https://graph.microsoft.com/v1.0/drives/b!-Jr3ZPe__kCZ7xLdl2FMoffWFSmIUF1AoWygTwxjtywWlD7heAFZQpK_8QbDA7ff/items/01HSWNV2LEGZAH5A2BLJE3R3I4ACF3QDRK
Accept-Encoding: gzip

However it has turned the DELETE into a GET request...

After a lot of thinking, I think this is actually a bug in the go runtime and I've reported that here.

Ideally Microsoft should be issuing a 307 redirect here which means don't change the HTTP verb. I think Go's behaviour in changing the DELETE is wrong though.

You can do this with rclone by using the --disable Purge flag which stops the special purge method for onedrive running and rclone will revert back to iterating the directory and deleting things individually.

It is possible to make a workaround for this if you are desperate - let me know!

BeansIsFat · September 14, 2020, 3:29pm

Thank you. The --disable purge flag is sufficient for now. I'll keep an eye on the go runtime bug in the meantime.

system · September 17, 2020, 3:29pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.