Purge command fails on OneDrive

BeansIsFat · September 11, 2020, 8:40pm

What is the problem you are having with rclone?

I am unable to use the purge command. The error indicates that my token contains an invalid signature. I edited the config to refresh the token and got the same error.

What is your rclone version (output from `rclone version`)

rclone v1.53.0
also tried on rclone v1.54.0-beta.4768.6a56ac103 with the same error

Which OS you are using and how many bits (eg Windows 7, 64 bit)

MacOS 10.14.6

Which cloud storage system are you using? (eg Google Drive)

OneDrive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone purge onedrive:temp

The rclone config contents with secrets removed.

[onedrive]
type = onedrive
token = {"access_token":"redacted","expiry":"2020-09-11T17:18:16.634655-04:00"}
drive_id = redacted
drive_type = business

A log from the command with the `-vv` flag

2020/09/11 16:20:57 DEBUG : rclone: Version "v1.53.0" starting with parameters ["rclone" "purge" "-vv" "onedrive:temp"]
2020/09/11 16:20:57 DEBUG : Creating backend with remote "onedrive:temp"
2020/09/11 16:20:57 DEBUG : Using config file from "/Users/BeansIsFat/.config/rclone/rclone.conf"
2020/09/11 16:21:00 ERROR : Attempt 1/3 failed with 1 errors and: unauthenticated: invalidSignature: Token contains invalid signature.
2020/09/11 16:21:00 ERROR : Attempt 2/3 failed with 1 errors and: unauthenticated: invalidSignature: Token contains invalid signature.
2020/09/11 16:21:00 ERROR : Attempt 3/3 failed with 1 errors and: unauthenticated: invalidSignature: Token contains invalid signature.
2020/09/11 16:21:00 DEBUG : 8 go routines active
2020/09/11 16:21:00 Failed to purge: unauthenticated: invalidSignature: Token contains invalid signature.

ncw · September 11, 2020, 9:47pm

Try

rclone config reconnect onedrive:

To refresh the token

BeansIsFat · September 11, 2020, 10:04pm

I tried that and got the same error after successfully refreshing the token

ncw · September 11, 2020, 10:35pm

Is it just purge that fails? What about rclone lsf onedrive: ?

BeansIsFat · September 11, 2020, 11:12pm

Yes, it's just the purge that fails. rclone lsf onedrive: works fine.

I do have another problem with copy or sync and changed files on this remote but I figured I'd solve this problem first as they don't seem related.

gulp79 · September 12, 2020, 2:48pm

same for me...

C:\Users\me>"D:\Portable\PortableApps\rclone\rclone.exe" -vv lsf onecrypt:Backup/Documents
2020/09/12 16:45:23 DEBUG : rclone: Version "v1.54.0-beta.4766.75de30cfa" starting with parameters ["D:\\Portable\\PortableApps\\rclone\\rclone.exe" "-vv" "lsf" "onecrypt:Backup/Documents"]
2020/09/12 16:45:23 DEBUG : Using config file from "D:\\Portable\\PortableApps\\rclone\\rclone.conf"
2020/09/12 16:45:23 DEBUG : Creating backend with remote "onecrypt:Backup/Documents"
2020/09/12 16:45:23 DEBUG : Creating backend with remote "onedrive:/Crypt/86.MlnvFA/178.Grfxphqwv"
2020/09/12 16:45:24 DEBUG : fs cache: renaming cache item "onedrive:/Crypt/86.MlnvFA/178.Grfxphqwv" to be canonical "onedrive:Crypt/86.MlnvFA/178.Grfxphqwv"
2020/09/12 16:45:24 DEBUG : fs cache: switching user supplied name "onedrive:/Crypt/86.MlnvFA/178.Grfxphqwv" for canonical name "onedrive:Crypt/86.MlnvFA/178.Grfxphqwv"
M&G/
Programmi&Guide&Utility/
2020/09/12 16:45:25 DEBUG : 5 go routines active

C:\Users\me>"D:\Portable\PortableApps\rclone\rclone.exe" -vv purge onecrypt:Backup/Documents
2020/09/12 16:45:35 DEBUG : rclone: Version "v1.54.0-beta.4766.75de30cfa" starting with parameters ["D:\\Portable\\PortableApps\\rclone\\rclone.exe" "-vv" "purge" "onecrypt:Backup/Documents"]
2020/09/12 16:45:35 DEBUG : Creating backend with remote "onecrypt:Backup/Documents"
2020/09/12 16:45:35 DEBUG : Using config file from "D:\\Portable\\PortableApps\\rclone\\rclone.conf"
2020/09/12 16:45:35 DEBUG : Creating backend with remote "onedrive:/Crypt/86.MlnvFA/178.Grfxphqwv"
2020/09/12 16:45:36 DEBUG : fs cache: renaming cache item "onedrive:/Crypt/86.MlnvFA/178.Grfxphqwv" to be canonical "onedrive:Crypt/86.MlnvFA/178.Grfxphqwv"
2020/09/12 16:45:36 DEBUG : fs cache: switching user supplied name "onedrive:/Crypt/86.MlnvFA/178.Grfxphqwv" for canonical name "onedrive:Crypt/86.MlnvFA/178.Grfxphqwv"
2020/09/12 16:45:37 ERROR : Attempt 1/3 failed with 1 errors and: unauthenticated: invalidSignature: Token contains invalid signature.
2020/09/12 16:45:37 ERROR : Attempt 2/3 failed with 1 errors and: unauthenticated: invalidSignature: Token contains invalid signature.
2020/09/12 16:45:38 ERROR : Attempt 3/3 failed with 1 errors and: unauthenticated: invalidSignature: Token contains invalid signature.
2020/09/12 16:45:38 DEBUG : 6 go routines active
2020/09/12 16:45:38 Failed to purge: unauthenticated: invalidSignature: Token contains invalid signature.

C:\Users\me>"D:\Portable\PortableApps\rclone\rclone.exe" -vv purge onedrive:/Crypt/86.MlnvFA/178.Grfxphqwv
2020/09/12 16:47:40 DEBUG : rclone: Version "v1.54.0-beta.4766.75de30cfa" starting with parameters ["D:\\Portable\\PortableApps\\rclone\\rclone.exe" "-vv" "purge" "onedrive:/Crypt/86.MlnvFA/178.Grfxphqwv"]
2020/09/12 16:47:40 DEBUG : Creating backend with remote "onedrive:/Crypt/86.MlnvFA/178.Grfxphqwv"
2020/09/12 16:47:40 DEBUG : Using config file from "D:\\Portable\\PortableApps\\rclone\\rclone.conf"
2020/09/12 16:47:41 DEBUG : fs cache: renaming cache item "onedrive:/Crypt/86.MlnvFA/178.Grfxphqwv" to be canonical "onedrive:Crypt/86.MlnvFA/178.Grfxphqwv"
2020/09/12 16:47:42 ERROR : Attempt 1/3 failed with 1 errors and: unauthenticated: invalidSignature: Token contains invalid signature.
2020/09/12 16:47:42 ERROR : Attempt 2/3 failed with 1 errors and: unauthenticated: invalidSignature: Token contains invalid signature.
2020/09/12 16:47:42 ERROR : Attempt 3/3 failed with 1 errors and: unauthenticated: invalidSignature: Token contains invalid signature.
2020/09/12 16:47:42 DEBUG : 6 go routines active
2020/09/12 16:47:42 Failed to purge: unauthenticated: invalidSignature: Token contains invalid signature.

ncw · September 12, 2020, 5:16pm

Strange!

Can you do a purge with

-vv --dump bodies --retries 1

So we can see the http transactions and post the result

BeansIsFat · September 13, 2020, 2:57am

Here's a link to the output of purge using

rclone purge --vv --dump bodies --retries 1 onedrive:temp

gist.github.com

https://gist.github.com/BeansIsFat/e2a4f5407ac09ce3db5aabb947aca465

gistfile1.txt

2020/09/12 22:44:35 DEBUG : rclone: Version "v1.53.0" starting with parameters ["rclone" "purge" "-vv" "--dump" "bodies" "--retries" "1" "onedrive:temp"]
2020/09/12 22:44:35 DEBUG : Creating backend with remote "onedrive:temp"
2020/09/12 22:44:35 DEBUG : Using config file from "/Users/redacted/.config/rclone/rclone.conf"
2020/09/12 22:44:35 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2020/09/12 22:44:35 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/09/12 22:44:35 DEBUG : HTTP REQUEST (req 0xc000133200)
2020/09/12 22:44:35 DEBUG : GET /v1.0/drives/b!-Jr3ZPe__kCZ7xLdl2FMoffWFSmIUF1AoWygTwxjtywWlD7heAFZQpK_8QbDA7ff/root HTTP/1.1
Host: graph.microsoft.com
User-Agent: rclone/v1.53.0
Authorization: XXXX

This file has been truncated. show original

gulp79 · September 13, 2020, 8:51pm

i've been able to remove the directory by mounting onedrive on linux (not tested on windows) and doing an "rm -rf" of target direcory. In this way, i've been able to get rid of it.
(i've also tested with onedrive web frontend, but is impossible to delete non empty dir on web, and i've a lot of files in subdir, so was not an option to do it manually on web )

ncw · September 14, 2020, 3:06pm

I can see what is happening there...

Rclone issues the DELETE on the directory

2020/09/12 22:44:36 DEBUG : HTTP REQUEST (req 0xc00066b300)
2020/09/12 22:44:36 DEBUG : DELETE /v1.0/drives/b!-Jr3ZPe__kCZ7xLdl2FMoffWFSmIUF1AoWygTwxjtywWlD7heAFZQpK_8QbDA7ff/items/01HSWNV2LEGZAH5A2BLJE3R3I4ACF3QDRK HTTP/1.1
Host: graph.microsoft.com
User-Agent: rclone/v1.53.0
Authorization: XXXX
Accept-Encoding: gzip

The server responds with a redirect

2020/09/12 22:44:36 DEBUG : HTTP RESPONSE (req 0xc00066b300)
2020/09/12 22:44:36 DEBUG : HTTP/1.1 302 Found
Cache-Control: private
Client-Request-Id: 9f245298-ad4b-4c78-b6c3-478208784d04
Content-Type: text/plain
Date: Sun, 13 Sep 2020 02:44:35 GMT
Location: https://redacted-my.sharepoint.com/_vti_bin/client.svc/v2.0/drives('b!-Jr3ZPe__kCZ7xLdl2FMoffWFSmIUF1AoWygTwxjtywWlD7heAFZQpK_8QbDA7ff')/items/01HSWNV2LEGZAH5A2BLJE3R3I4ACF3QDRK
Request-Id: 9f245298-ad4b-4c78-b6c3-478208784d04
Strict-Transport-Security: max-age=31536000
X-Ms-Ags-Diagnostic: {"ServerInfo":{"DataCenter":"North Central US","Slice":"SliceC","Ring":"3","ScaleUnit":"003","RoleInstance":"AGSFE_IN_30"}}
Content-Length: 0

So rclone does the request at that URL

2020/09/12 22:44:36 DEBUG : HTTP REQUEST (req 0xc0004be400)
2020/09/12 22:44:36 DEBUG : GET /_vti_bin/client.svc/v2.0/drives('b!-Jr3ZPe__kCZ7xLdl2FMoffWFSmIUF1AoWygTwxjtywWlD7heAFZQpK_8QbDA7ff')/items/01HSWNV2LEGZAH5A2BLJE3R3I4ACF3QDRK HTTP/1.1
Host: redacted-my.sharepoint.com
User-Agent: rclone/v1.53.0
Authorization: XXXX
Referer: https://graph.microsoft.com/v1.0/drives/b!-Jr3ZPe__kCZ7xLdl2FMoffWFSmIUF1AoWygTwxjtywWlD7heAFZQpK_8QbDA7ff/items/01HSWNV2LEGZAH5A2BLJE3R3I4ACF3QDRK
Accept-Encoding: gzip

However it has turned the DELETE into a GET request...

After a lot of thinking, I think this is actually a bug in the go runtime and I've reported that here.

github.com/golang/go

net/http: 302 redirect of DELETE to GET isn't RFC compliant

opened 03:02PM - 14 Sep 20 UTC

closed 11:36PM - 03 Dec 24 UTC

ncw

NeedsDecision

### What version of Go are you using (`go version`)? <pre> $ go version go …version go1.15 linux/amd64 </pre> ### Does this issue reproduce with the latest release? Yes ### What operating system and processor architecture are you using (`go env`)? <details><summary><code>go env</code> Output</summary><br><pre> $ go env GO111MODULE="" GOARCH="amd64" GOBIN="" GOCACHE="/home/ncw/.cache/go-build" GOENV="/home/ncw/.config/go/env" GOEXE="" GOFLAGS="" GOHOSTARCH="amd64" GOHOSTOS="linux" GOINSECURE="" GOMODCACHE="/home/ncw/go/pkg/mod" GONOPROXY="" GONOSUMDB="" GOOS="linux" GOPATH="/home/ncw/go" GOPRIVATE="" GOPROXY="https://proxy.golang.org,direct" GOROOT="/opt/go/go1.15" GOSUMDB="sum.golang.org" GOTMPDIR="" GOTOOLDIR="/opt/go/go1.15/pkg/tool/linux_amd64" GCCGO="/usr/bin/gccgo" AR="ar" CC="gcc" CXX="g++" CGO_ENABLED="1" GOMOD="" CGO_CFLAGS="-g -O2" CGO_CPPFLAGS="" CGO_CXXFLAGS="-g -O2" CGO_FFLAGS="-g -O2" CGO_LDFLAGS="-g -O2" PKG_CONFIG="pkg-config" GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build061961387=/tmp/go-build -gno-record-gcc-switches" </pre></details> ### What did you do? We used http.Client against a Microsoft Ondrive service. When we issued a DELETE response the server responded with a 302 redirect and http.Client retried with a GET request. We issue a DELETE request ``` 2020/09/12 22:44:36 DEBUG : HTTP REQUEST (req 0xc00066b300) 2020/09/12 22:44:36 DEBUG : DELETE /v1.0/drives/b!-Jr3ZPe__kCZ7xLdl2FMoffWFSmIUF1AoWygTwxjtywWlD7heAFZQpK_8QbDA7ff/items/01HSWNV2LEGZAH5A2BLJE3R3I4ACF3QDRK HTTP/1.1 Host: graph.microsoft.com User-Agent: rclone/v1.53.0 Authorization: XXXX Accept-Encoding: gzip ``` The server responds with a 302 redirect ``` 2020/09/12 22:44:36 DEBUG : HTTP RESPONSE (req 0xc00066b300) 2020/09/12 22:44:36 DEBUG : HTTP/1.1 302 Found Cache-Control: private Client-Request-Id: 9f245298-ad4b-4c78-b6c3-478208784d04 Content-Type: text/plain Date: Sun, 13 Sep 2020 02:44:35 GMT Location: https://redacted-my.sharepoint.com/_vti_bin/client.svc/v2.0/drives('b!-Jr3ZPe__kCZ7xLdl2FMoffWFSmIUF1AoWygTwxjtywWlD7heAFZQpK_8QbDA7ff')/items/01HSWNV2LEGZAH5A2BLJE3R3I4ACF3QDRK Request-Id: 9f245298-ad4b-4c78-b6c3-478208784d04 Strict-Transport-Security: max-age=31536000 X-Ms-Ags-Diagnostic: {"ServerInfo":{"DataCenter":"North Central US","Slice":"SliceC","Ring":"3","ScaleUnit":"003","RoleInstance":"AGSFE_IN_30"}} Content-Length: 0 ``` We then do a GET request on the redirected URL which doesn't work. ``` 2020/09/12 22:44:36 DEBUG : HTTP REQUEST (req 0xc0004be400) 2020/09/12 22:44:36 DEBUG : GET /_vti_bin/client.svc/v2.0/drives('b!-Jr3ZPe__kCZ7xLdl2FMoffWFSmIUF1AoWygTwxjtywWlD7heAFZQpK_8QbDA7ff')/items/01HSWNV2LEGZAH5A2BLJE3R3I4ACF3QDRK HTTP/1.1 Host: redacted-my.sharepoint.com User-Agent: rclone/v1.53.0 Authorization: XXXX Referer: https://graph.microsoft.com/v1.0/drives/b!-Jr3ZPe__kCZ7xLdl2FMoffWFSmIUF1AoWygTwxjtywWlD7heAFZQpK_8QbDA7ff/items/01HSWNV2LEGZAH5A2BLJE3R3I4ACF3QDRK Accept-Encoding: gzip ``` ### What did you expect to see? I expected http.Client to redirect with a DELETE request. According to [RFC7231](https://tools.ietf.org/html/rfc7231#section-6.4.3) > 6.4.3. 302 Found > > The 302 (Found) status code indicates that the target resource > resides temporarily under a different URI. Since the redirection > might be altered on occasion, the client ought to continue to use the > effective request URI for future requests. > > The server SHOULD generate a Location header field in the response > containing a URI reference for the different URI. The user agent MAY > use the Location field value for automatic redirection. The server's > response payload usually contains a short hypertext note with a > hyperlink to the different URI(s). > > Note: For historical reasons, a user agent MAY change the request > method from POST to GET for the subsequent request. If this > behavior is undesired, the 307 (Temporary Redirect) status code > can be used instead. I note that this only talks about POST to GET and not other methods in the historical reasons section. So I'm not sure Go's implementation is RFC compliant here - the RFC doesn't say that user agents may change any other sort of method from POST to GET. The code is here: https://github.com/golang/go/blob/66e66e71132034aa620ffbae9008f951da0f9f27/src/net/http/client.go#L496-L512 Which says for all methods other than GET or HEAD, change them to GET, whereas the RFC says you can change POST to GET only. So I think backwards compatibility might demand that POST becomes GET so something like this would be more RFC compliant. ```go if reqMethod == "POST" { redirectMethod = "GET" } ``` The current code seems to think the major distinction between 301/302 and 307/308 is whether the body is resent. I'm not sure the RFC supports that, but using that distinction I think the code code could look like ```go if ireq.outgoingLength() != 0 { redirectMethod = "GET" } ``` Reading the RFC it says nothing about request bodies vs headers. So I think to be most useful and most RFC compliant the 301,302 redirects should work just like the 307,308 redirects now we have the capability to resend the body if needed. See: https://github.com/golang/go/issues/18570 See: https://forum.rclone.org/t/purge-command-fails-on-onedrive/19048

Ideally Microsoft should be issuing a 307 redirect here which means don't change the HTTP verb. I think Go's behaviour in changing the DELETE is wrong though.

You can do this with rclone by using the --disable Purge flag which stops the special purge method for onedrive running and rclone will revert back to iterating the directory and deleting things individually.

It is possible to make a workaround for this if you are desperate - let me know!

BeansIsFat · September 14, 2020, 3:29pm

Thank you. The --disable purge flag is sufficient for now. I'll keep an eye on the go runtime bug in the meantime.

system · September 17, 2020, 3:29pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.