Onedrive for Business errors: "Failed to mkdir: 429" and "failed to get root: 429"


#1

I used rcone config to setup a onedrive for business config:

$ rclone config -v -v
2018/02/12 11:59:10 DEBUG : Using config file from "/home/user/.config/rclone/rclone.conf"
2018/02/12 11:59:10 DEBUG : rclone: Version "v1.39" starting with parameters ["rclone" "config" "-v" "-v"]
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> onedrive
Type of storage to configure.
Choose a number from below, or type in your own value
 1 / Amazon Drive
   \ "amazon cloud drive"
 2 / Amazon S3 (also Dreamhost, Ceph, Minio)
   \ "s3"
 3 / Backblaze B2
   \ "b2"
 4 / Box
   \ "box"
 5 / Cache a remote
   \ "cache"
 6 / Dropbox
   \ "dropbox"
 7 / Encrypt/Decrypt a remote
   \ "crypt"
 8 / FTP Connection
   \ "ftp"
 9 / Google Cloud Storage (this is not Google Drive)
   \ "google cloud storage"
10 / Google Drive
   \ "drive"
11 / Hubic
   \ "hubic"
12 / Local Disk
   \ "local"
13 / Microsoft Azure Blob Storage
   \ "azureblob"
14 / Microsoft OneDrive
   \ "onedrive"
15 / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
   \ "swift"
16 / Pcloud
   \ "pcloud"
17 / QingCloud Object Storage
   \ "qingstor"
18 / SSH/SFTP Connection
   \ "sftp"
19 / Webdav
   \ "webdav"
20 / Yandex Disk
   \ "yandex"
21 / http Connection
   \ "http"
Storage> 14
Microsoft App Client Id - leave blank normally.
client_id> 
Microsoft App Client Secret - leave blank normally.
client_secret> 
Remote config
Choose OneDrive account type?
 * Say b for a OneDrive business account
 * Say p for a personal OneDrive account
b) Business
p) Personal
b/p> b
Use auto config?
 * Say Y if not sure
 * Say N if you are working on a remote or headless machine
y) Yes
n) No
y/n> y
2018/02/12 11:59:21 DEBUG : Starting auth server on 127.0.0.1:53682
If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
Log in and authorize rclone for access
Waiting for code...
2018/02/12 11:59:21 DEBUG : Received request on auth server
2018/02/12 11:59:21 DEBUG : Successfully got code
Got code
2018/02/12 11:59:22 DEBUG : onedrive: Added new token to config, still needs to be saved
2018/02/12 11:59:22 DEBUG : Closing auth server
2018/02/12 11:59:22 DEBUG : Closed auth server with error: http: Server closed
2018/02/12 11:59:25 INFO  : Skipping API v1.0 endpoint https://company-my.sharepoint.com/_api/v1.0/me
2018/02/12 11:59:25 INFO  : Skipping API v2.0 endpoint https://company-my.sharepoint.com/_api/v2.0/me
2018/02/12 11:59:26 DEBUG : onedrive: Added new token to config, still needs to be saved
--------------------
[onedrive]
client_id = 
client_secret = 
token = {"access_token":"token"}
resource_url = https://company-my.sharepoint.com/

And when trying a few things out, i have some errors:

$ rclone mkdir onedrive:new
2018/02/12 12:00:03 ERROR : Attempt 1/3 failed with 1 errors and: 429 : 
2018/02/12 12:00:27 ERROR : Attempt 2/3 failed with 1 errors and: 429 : 
2018/02/12 12:00:47 ERROR : Attempt 3/3 failed with 1 errors and: 429 : 
2018/02/12 12:00:47 Failed to mkdir: 429 : 
$ rclone ls onedrive:
2018/02/12 12:02:08 Failed to create file system for "onedrive:": failed to get root: 429 :

Rclone config:

$ cat .config/rclone/rclone.conf 
[onedrive]
type = onedrive
client_id = 
client_secret = 
token = {"access_token":"token"}
resource_url = https://company-my.sharepoint.com/

And running the latest version too:

rclone --version
rclone v1.39
- os/arch: linux/amd64
- go version: go1.9.2

Rclone seems to be broken, unless i am doing something wrong?


#2

That isn’t a very helpful error is it!

429 is an HTTP error code which means “Too Many Requests” - basically try again later.

However rclone should do that trying again later for you.

What does this print?

rclone lsd --retries 1 -vv --dump bodies onedrive:

#3

Thanks for the response!

$ rclone lsd --retries 1 -vv --dump bodies onedrive:
2018/02/13 10:49:10 DEBUG : Using config file from "/home/user/.config/rclone/rclone.conf"
2018/02/13 10:49:10 DEBUG : rclone: Version "v1.39" starting with parameters ["rclone" "lsd" "--retries" "1" "-vv" "--dump" "bodies" "onedrive:"]
2018/02/13 10:49:10 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/02/13 10:49:10 DEBUG : HTTP REQUEST (req 0xc4201c6d00)
2018/02/13 10:49:10 DEBUG : GET /_api/v2.0/drives/me/root:/ HTTP/1.1
Host: company-my.sharepoint.com
User-Agent: rclone/v1.39
Authorization: XXXX
Accept-Encoding: gzip

2018/02/13 10:49:10 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/02/13 10:49:11 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/02/13 10:49:11 DEBUG : HTTP RESPONSE (req 0xc4201c6d00)
2018/02/13 10:49:11 DEBUG : HTTP/1.1 429 
Content-Length: 21
Content-Type: text/plain; charset=utf-8
Date: Tue, 13 Feb 2018 09:49:10 GMT
Microsoftsharepointteamservices: 16.0.0.7324
Ms-Cv: nkoL5FlgAFAZqamA/nA+kQ.0
P3p: CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI"
Request-Id: e40b4a9e-6059-5000-19a9-a980fe703e91
Retry-After: 120
Spiislatency: 5
Sprequestduration: 294
Sprequestguid: e40b4a9e-6059-5000-19a9-a980fe703e91
Strict-Transport-Security: max-age=31536000
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Ms-Invokeapp: 1; RequireReadOnly
X-Msedge-Ref: Ref A: 603A274CC6694443ACCF60876081AD19 Ref B: AMSEDGE0619 Ref C: 2018-02-13T09:49:11Z
X-Powered-By: ASP.NET

429 TOO MANY REQUESTS
2018/02/13 10:49:11 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/02/13 10:49:11 DEBUG : Couldn't decode error response: json: cannot unmarshal number into Go value of type api.Error
2018/02/13 10:49:11 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2018/02/13 10:49:11 DEBUG : pacer: low level retry 1/10 (error 429 : )
2018/02/13 10:49:11 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/02/13 10:49:11 DEBUG : HTTP REQUEST (req 0xc4201c6700)
2018/02/13 10:49:11 DEBUG : GET /_api/v2.0/drives/me/root:/ HTTP/1.1
Host: company-my.sharepoint.com
User-Agent: rclone/v1.39
Authorization: XXXX
Accept-Encoding: gzip

2018/02/13 10:49:11 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/02/13 10:49:11 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/02/13 10:49:11 DEBUG : HTTP RESPONSE (req 0xc4201c6700)
2018/02/13 10:49:11 DEBUG : HTTP/1.1 429 
Content-Length: 21
Content-Type: text/plain; charset=utf-8
Date: Tue, 13 Feb 2018 09:49:10 GMT
Microsoftsharepointteamservices: 16.0.0.7324
Ms-Cv: nkoL5HVQAFAZqax6kkuoFg.0
P3p: CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI"
Request-Id: e40b4a9e-5075-5000-19a9-ac7a924ba816
Retry-After: 120
Spiislatency: 1
Sprequestduration: 238
Sprequestguid: e40b4a9e-5075-5000-19a9-ac7a924ba816
Strict-Transport-Security: max-age=31536000
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Ms-Invokeapp: 1; RequireReadOnly
X-Msedge-Ref: Ref A: B99A38B41CBB4BC3A24AF8644BA00FAA Ref B: AMSEDGE0619 Ref C: 2018-02-13T09:49:11Z
X-Powered-By: ASP.NET

429 TOO MANY REQUESTS
2018/02/13 10:49:11 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/02/13 10:49:11 DEBUG : Couldn't decode error response: json: cannot unmarshal number into Go value of type api.Error
2018/02/13 10:49:11 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2018/02/13 10:49:11 DEBUG : pacer: low level retry 2/10 (error 429 : )
2018/02/13 10:49:11 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/02/13 10:49:11 DEBUG : HTTP REQUEST (req 0xc4201c6a00)
2018/02/13 10:49:11 DEBUG : GET /_api/v2.0/drives/me/root:/ HTTP/1.1
Host: company-my.sharepoint.com
User-Agent: rclone/v1.39
Authorization: XXXX
Accept-Encoding: gzip

2018/02/13 10:49:11 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/02/13 10:49:12 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/02/13 10:49:12 DEBUG : HTTP RESPONSE (req 0xc4201c6a00)
2018/02/13 10:49:12 DEBUG : HTTP/1.1 429 
Content-Length: 21
Content-Type: text/plain; charset=utf-8
Date: Tue, 13 Feb 2018 09:49:11 GMT
Microsoftsharepointteamservices: 16.0.0.7324
Ms-Cv: nkoL5IRQAFAZqaoQ5oGdeg.0
P3p: CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI"
Request-Id: e40b4a9e-5084-5000-19a9-aa10e6819d7a
Retry-After: 120
Spiislatency: 5
Sprequestduration: 393
Sprequestguid: e40b4a9e-5084-5000-19a9-aa10e6819d7a
Strict-Transport-Security: max-age=31536000
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Ms-Invokeapp: 1; RequireReadOnly
X-Msedge-Ref: Ref A: A8FB47DE73204EED8B933C5A7B21A479 Ref B: AMSEDGE0619 Ref C: 2018-02-13T09:49:12Z
X-Powered-By: ASP.NET

429 TOO MANY REQUESTS
2018/02/13 10:49:12 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/02/13 10:49:12 DEBUG : Couldn't decode error response: json: cannot unmarshal number into Go value of type api.Error
2018/02/13 10:49:12 DEBUG : pacer: Rate limited, increasing sleep to 80ms
2018/02/13 10:49:12 DEBUG : pacer: low level retry 3/10 (error 429 : )
2018/02/13 10:49:12 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/02/13 10:49:12 DEBUG : HTTP REQUEST (req 0xc4200fe300)
2018/02/13 10:49:12 DEBUG : GET /_api/v2.0/drives/me/root:/ HTTP/1.1
Host: company-my.sharepoint.com
User-Agent: rclone/v1.39
Authorization: XXXX
Accept-Encoding: gzip

2018/02/13 10:49:12 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/02/13 10:49:12 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/02/13 10:49:12 DEBUG : HTTP RESPONSE (req 0xc4200fe300)
2018/02/13 10:49:12 DEBUG : HTTP/1.1 429 
Content-Length: 21
Content-Type: text/plain; charset=utf-8
Date: Tue, 13 Feb 2018 09:49:11 GMT
Microsoftsharepointteamservices: 16.0.0.7324
Ms-Cv: nkoL5KUwAFAZqaB/PAyfsg.0
P3p: CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI"
Request-Id: e40b4a9e-30a5-5000-19a9-a07f3c0c9fb2
Retry-After: 120
Spiislatency: 1
Sprequestduration: 185
Sprequestguid: e40b4a9e-30a5-5000-19a9-a07f3c0c9fb2
Strict-Transport-Security: max-age=31536000
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Ms-Invokeapp: 1; RequireReadOnly
X-Msedge-Ref: Ref A: E30578A65AF2477FAE0A4417F5A2C841 Ref B: AMSEDGE0619 Ref C: 2018-02-13T09:49:12Z
X-Powered-By: ASP.NET

429 TOO MANY REQUESTS
2018/02/13 10:49:12 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/02/13 10:49:12 DEBUG : Couldn't decode error response: json: cannot unmarshal number into Go value of type api.Error
2018/02/13 10:49:12 DEBUG : pacer: Rate limited, increasing sleep to 160ms
2018/02/13 10:49:12 DEBUG : pacer: low level retry 4/10 (error 429 : )
2018/02/13 10:49:12 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/02/13 10:49:12 DEBUG : HTTP REQUEST (req 0xc4200fea00)
2018/02/13 10:49:12 DEBUG : GET /_api/v2.0/drives/me/root:/ HTTP/1.1
Host: company-my.sharepoint.com
User-Agent: rclone/v1.39
Authorization: XXXX
Accept-Encoding: gzip

2018/02/13 10:49:12 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/02/13 10:49:12 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/02/13 10:49:12 DEBUG : HTTP RESPONSE (req 0xc4200fea00)
2018/02/13 10:49:12 DEBUG : HTTP/1.1 429 
Content-Length: 21
Content-Type: text/plain; charset=utf-8
Date: Tue, 13 Feb 2018 09:49:11 GMT
Microsoftsharepointteamservices: 16.0.0.7324
Ms-Cv: nkoL5K9gAFAZqaEZQ3y82w.0
P3p: CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI"
Request-Id: e40b4a9e-60af-5000-19a9-a119437cbcdb
Retry-After: 120
Spiislatency: 1
Sprequestduration: 206
Sprequestguid: e40b4a9e-60af-5000-19a9-a119437cbcdb
Strict-Transport-Security: max-age=31536000
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Ms-Invokeapp: 1; RequireReadOnly
X-Msedge-Ref: Ref A: 0852E30CF4894B8EB23A29A04BA2E366 Ref B: AMSEDGE0619 Ref C: 2018-02-13T09:49:12Z
X-Powered-By: ASP.NET

429 TOO MANY REQUESTS
2018/02/13 10:49:12 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/02/13 10:49:12 DEBUG : Couldn't decode error response: json: cannot unmarshal number into Go value of type api.Error
2018/02/13 10:49:12 DEBUG : pacer: Rate limited, increasing sleep to 320ms
2018/02/13 10:49:12 DEBUG : pacer: low level retry 5/10 (error 429 : )
2018/02/13 10:49:12 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/02/13 10:49:12 DEBUG : HTTP REQUEST (req 0xc42034e300)
2018/02/13 10:49:12 DEBUG : GET /_api/v2.0/drives/me/root:/ HTTP/1.1
Host: company-my.sharepoint.com
User-Agent: rclone/v1.39
Authorization: XXXX
Accept-Encoding: gzip

2018/02/13 10:49:12 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/02/13 10:49:12 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/02/13 10:49:12 DEBUG : HTTP RESPONSE (req 0xc42034e300)
2018/02/13 10:49:12 DEBUG : HTTP/1.1 429 
Content-Length: 21
Content-Type: text/plain; charset=utf-8
Date: Tue, 13 Feb 2018 09:49:12 GMT
Microsoftsharepointteamservices: 16.0.0.7324
Ms-Cv: nkoL5MZgAFAZqagrCTALag.0
P3p: CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI"
Request-Id: e40b4a9e-60c6-5000-19a9-a82b09300b6a
Retry-After: 120
Spiislatency: 1
Sprequestduration: 275
Sprequestguid: e40b4a9e-60c6-5000-19a9-a82b09300b6a
Strict-Transport-Security: max-age=31536000
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Ms-Invokeapp: 1; RequireReadOnly
X-Msedge-Ref: Ref A: C71F2F7FF747491FBC187DF996AE89AD Ref B: AMSEDGE0619 Ref C: 2018-02-13T09:49:12Z
X-Powered-By: ASP.NET

429 TOO MANY REQUESTS
2018/02/13 10:49:12 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/02/13 10:49:12 DEBUG : Couldn't decode error response: json: cannot unmarshal number into Go value of type api.Error
2018/02/13 10:49:12 DEBUG : pacer: Rate limited, increasing sleep to 640ms
2018/02/13 10:49:12 DEBUG : pacer: low level retry 6/10 (error 429 : )
2018/02/13 10:49:12 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/02/13 10:49:12 DEBUG : HTTP REQUEST (req 0xc42034e600)
2018/02/13 10:49:12 DEBUG : GET /_api/v2.0/drives/me/root:/ HTTP/1.1
Host: company-my.sharepoint.com
User-Agent: rclone/v1.39
Authorization: XXXX
Accept-Encoding: gzip

2018/02/13 10:49:12 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/02/13 10:49:13 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/02/13 10:49:13 DEBUG : HTTP RESPONSE (req 0xc42034e600)
2018/02/13 10:49:13 DEBUG : HTTP/1.1 429 
Content-Length: 21
Content-Type: text/plain; charset=utf-8
Date: Tue, 13 Feb 2018 09:49:12 GMT
Microsoftsharepointteamservices: 16.0.0.7324
Ms-Cv: nkoL5NhQAFAZqaw94IZYMA.0
P3p: CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI"
Request-Id: e40b4a9e-50d8-5000-19a9-ac3de0865830
Retry-After: 120
Spiislatency: 3
Sprequestduration: 217
Sprequestguid: e40b4a9e-50d8-5000-19a9-ac3de0865830
Strict-Transport-Security: max-age=31536000
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Ms-Invokeapp: 1; RequireReadOnly
X-Msedge-Ref: Ref A: C7C3C69525554368AA1EDE4F9C2CC40E Ref B: AMSEDGE0619 Ref C: 2018-02-13T09:49:13Z
X-Powered-By: ASP.NET

429 TOO MANY REQUESTS
2018/02/13 10:49:13 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/02/13 10:49:13 DEBUG : Couldn't decode error response: json: cannot unmarshal number into Go value of type api.Error
2018/02/13 10:49:13 DEBUG : pacer: Rate limited, increasing sleep to 1.28s
2018/02/13 10:49:13 DEBUG : pacer: low level retry 7/10 (error 429 : )
2018/02/13 10:49:13 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/02/13 10:49:13 DEBUG : HTTP REQUEST (req 0xc4200fed00)
2018/02/13 10:49:13 DEBUG : GET /_api/v2.0/drives/me/root:/ HTTP/1.1
Host: company-my.sharepoint.com
User-Agent: rclone/v1.39
Authorization: XXXX
Accept-Encoding: gzip

2018/02/13 10:49:13 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/02/13 10:49:13 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/02/13 10:49:13 DEBUG : HTTP RESPONSE (req 0xc4200fed00)
2018/02/13 10:49:13 DEBUG : HTTP/1.1 429 
Content-Length: 21
Content-Type: text/plain; charset=utf-8
Date: Tue, 13 Feb 2018 09:49:13 GMT
Microsoftsharepointteamservices: 16.0.0.7324
Ms-Cv: nkoL5QjAAFAZqa+7GxJnCA.0
P3p: CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI"
Request-Id: e50b4a9e-c008-5000-19a9-afbb1b126708
Retry-After: 120
Spiislatency: 59
Sprequestduration: 226
Sprequestguid: e50b4a9e-c008-5000-19a9-afbb1b126708
Strict-Transport-Security: max-age=31536000
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Ms-Invokeapp: 1; RequireReadOnly
X-Msedge-Ref: Ref A: 144D48FD4F0B4CAE95884A5E66275A49 Ref B: AMSEDGE0619 Ref C: 2018-02-13T09:49:13Z
X-Powered-By: ASP.NET

429 TOO MANY REQUESTS
2018/02/13 10:49:13 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/02/13 10:49:13 DEBUG : Couldn't decode error response: json: cannot unmarshal number into Go value of type api.Error
2018/02/13 10:49:13 DEBUG : pacer: Rate limited, increasing sleep to 2s
2018/02/13 10:49:13 DEBUG : pacer: low level retry 8/10 (error 429 : )
2018/02/13 10:49:14 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/02/13 10:49:14 DEBUG : HTTP REQUEST (req 0xc4200ff100)
2018/02/13 10:49:14 DEBUG : GET /_api/v2.0/drives/me/root:/ HTTP/1.1
Host: company-my.sharepoint.com
User-Agent: rclone/v1.39
Authorization: XXXX
Accept-Encoding: gzip

2018/02/13 10:49:14 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/02/13 10:49:15 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/02/13 10:49:15 DEBUG : HTTP RESPONSE (req 0xc4200ff100)
2018/02/13 10:49:15 DEBUG : HTTP/1.1 429 
Content-Length: 21
Content-Type: text/plain; charset=utf-8
Date: Tue, 13 Feb 2018 09:49:14 GMT
Microsoftsharepointteamservices: 16.0.0.7324
Ms-Cv: nkoL5U9wAFAZqav6qypGdg.0
P3p: CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI"
Request-Id: e50b4a9e-704f-5000-19a9-abfaab2a4676
Retry-After: 120
Spiislatency: 2
Sprequestduration: 213
Sprequestguid: e50b4a9e-704f-5000-19a9-abfaab2a4676
Strict-Transport-Security: max-age=31536000
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Ms-Invokeapp: 1; RequireReadOnly
X-Msedge-Ref: Ref A: 1CB5687033A443119D68B38C3E82932F Ref B: AMSEDGE0619 Ref C: 2018-02-13T09:49:15Z
X-Powered-By: ASP.NET

429 TOO MANY REQUESTS
2018/02/13 10:49:15 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/02/13 10:49:15 DEBUG : Couldn't decode error response: json: cannot unmarshal number into Go value of type api.Error
2018/02/13 10:49:15 DEBUG : pacer: low level retry 9/10 (error 429 : )
2018/02/13 10:49:16 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/02/13 10:49:16 DEBUG : HTTP REQUEST (req 0xc42034e900)
2018/02/13 10:49:16 DEBUG : GET /_api/v2.0/drives/me/root:/ HTTP/1.1
Host: company-my.sharepoint.com
User-Agent: rclone/v1.39
Authorization: XXXX
Accept-Encoding: gzip

2018/02/13 10:49:16 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/02/13 10:49:17 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/02/13 10:49:17 DEBUG : HTTP RESPONSE (req 0xc42034e900)
2018/02/13 10:49:17 DEBUG : HTTP/1.1 429 
Content-Length: 21
Content-Type: text/plain; charset=utf-8
Date: Tue, 13 Feb 2018 09:49:16 GMT
Microsoftsharepointteamservices: 16.0.0.7324
Ms-Cv: nkoL5dHwAFAZqaEMCQ5k6g.0
P3p: CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI"
Request-Id: e50b4a9e-f0d1-5000-19a9-a10c090e64ea
Retry-After: 120
Spiislatency: 3
Sprequestduration: 243
Sprequestguid: e50b4a9e-f0d1-5000-19a9-a10c090e64ea
Strict-Transport-Security: max-age=31536000
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Ms-Invokeapp: 1; RequireReadOnly
X-Msedge-Ref: Ref A: 403FDAD624594CCE941F13A9159FBAE7 Ref B: AMSEDGE0619 Ref C: 2018-02-13T09:49:17Z
X-Powered-By: ASP.NET

429 TOO MANY REQUESTS
2018/02/13 10:49:17 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/02/13 10:49:17 DEBUG : Couldn't decode error response: json: cannot unmarshal number into Go value of type api.Error
2018/02/13 10:49:17 DEBUG : pacer: low level retry 10/10 (error 429 : )
2018/02/13 10:49:17 Failed to create file system for "onedrive:": failed to get root: 429 : 

So still 429 TOO MANY REQUESTS.


#4

Thanks for posting that. It looks like rclone is doing the retries it is supposed to.

However you seem to get that 429 error. It looks slightly malformed too - there is supposed to be explanatory text after the 429 here which is why the error message looks so odd.

I’m really not sure what is going on!

@olihey - you know a lot more about this than I do - any ideas?


#5

I tried it with all my 3 accounts and it works. It’s not even getting the ID for the drives root. My gut feeling is, there is something configured for your company that might provoke this error. Knowing the whole Microsoft rules stuff it could be that somehow access for 3rd party is disabled or something else.

Can you please try the beta version I created? This one uses a different API to access OneDrive. You can find builds for LInux and Windows here: https://drive.google.com/drive/folders/1ojgVP0jliEsU7krsO8CKF2M5uUT69xez


#6

Ok, i tried the version from the google drive, the dev version

rclonebeta --version
rclone v1.39-DEV
- os/arch: linux/amd64
- go version: go1.9

And that works fine. The version from using the script at curl https://rclone.org/install.sh | sudo bash -s beta

rclone --version
rclone v1.39-148-g70b48428β
- os/arch: linux/amd64
- go version: go1.9.3

Also works fine.


#7

I’m glad it is working. I don’t see anything in the changelog from v1.39 to the current beta which could have fixed it though.


#8

Hi, I’m using v 1.41 on MacOSX and below this message you will find the output from running that debug command against OneDrive for Business.

I get 9 x 429s before the 10th attempt eventually comes-up with a 200 and we get a listing.

Repeated attempt vary in the number of 429s but I don’t seem to get a clean first time 200 ever.

I am assuming that the fix introduced in this thread at version v1.39 DEV for Windows and Linux will have found its way into the current v1.41 Mac version?

If not, is there a way to test this other OneDrive API that you talk about above with the Mac version?

Thanks

2018/05/14 16:44:13 DEBUG : rclone: Version “v1.41” starting with parameters [“rclone” “lsd” “–retries” “1” “-vv” “–dump” “bodies” “OneDriveforBusiness:”]
2018/05/14 16:44:13 DEBUG : Using config file from “/Users/admin/.config/rclone/rclone.conf”
2018/05/14 16:44:13 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/05/14 16:44:13 DEBUG : HTTP REQUEST (req 0x422868900)
2018/05/14 16:44:13 DEBUG : GET /_api/v2.0/drives/me/root:/ HTTP/1.1
Host: xxxx-my.sharepoint.com
User-Agent: rclone/v1.41
Authorization: XXXX
Accept-Encoding: gzip

2018/05/14 16:44:13 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/05/14 16:44:15 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/05/14 16:44:15 DEBUG : HTTP RESPONSE (req 0x422868900)
2018/05/14 16:44:15 DEBUG : HTTP/1.1 429
Content-Length: 125
Content-Type: application/json; charset=utf-8
Date: Mon, 14 May 2018 14:44:15 GMT
Microsoftsharepointteamservices: 16.0.0.7625
P3p: CP=“ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI”
Spiislatency: 3
Sprequestduration: 188
X-Content-Type-Options: nosniff
X-Ms-Invokeapp: 1; RequireReadOnly
X-Msedge-Ref: Ref A: B89AC6F03CE746F08B1AEFADD110132C Ref B: VIEEDGE0616 Ref C: 2018-05-14T14:44:15Z
X-Powered-By: ASP.NET

{“error”:{“code”:“activityLimitReached”,“message”:“The request has been throttled”,“innerError”:{“code”:“throttledRequest”}}}
2018/05/14 16:44:15 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/05/14 16:44:15 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2018/05/14 16:44:15 DEBUG : pacer: low level retry 1/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2018/05/14 16:44:15 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/05/14 16:44:15 DEBUG : HTTP REQUEST (req 0x422c2a100)
2018/05/14 16:44:15 DEBUG : GET /_api/v2.0/drives/me/root:/ HTTP/1.1
Host: xxxx-my.sharepoint.com
User-Agent: rclone/v1.41
Authorization: XXXX
Accept-Encoding: gzip

2018/05/14 16:44:15 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/05/14 16:44:15 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/05/14 16:44:15 DEBUG : HTTP RESPONSE (req 0x422c2a100)
2018/05/14 16:44:15 DEBUG : HTTP/1.1 429
Content-Length: 125
Content-Type: application/json; charset=utf-8
Date: Mon, 14 May 2018 14:44:15 GMT
Microsoftsharepointteamservices: 16.0.0.7625
P3p: CP=“ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI”
Spiislatency: 3
Sprequestduration: 377
X-Content-Type-Options: nosniff
X-Ms-Invokeapp: 1; RequireReadOnly
X-Msedge-Ref: Ref A: 2F08EE4B6CAB4E868767E0E947BF8FF6 Ref B: VIEEDGE0616 Ref C: 2018-05-14T14:44:15Z
X-Powered-By: ASP.NET

{“error”:{“code”:“activityLimitReached”,“message”:“The request has been throttled”,“innerError”:{“code”:“throttledRequest”}}}
2018/05/14 16:44:15 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/05/14 16:44:15 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2018/05/14 16:44:15 DEBUG : pacer: low level retry 2/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2018/05/14 16:44:15 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/05/14 16:44:15 DEBUG : HTTP REQUEST (req 0x422868a00)
2018/05/14 16:44:15 DEBUG : GET /_api/v2.0/drives/me/root:/ HTTP/1.1
Host: xxxx-my.sharepoint.com
User-Agent: rclone/v1.41
Authorization: XXXX
Accept-Encoding: gzip

2018/05/14 16:44:15 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/05/14 16:44:15 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/05/14 16:44:15 DEBUG : HTTP RESPONSE (req 0x422868a00)
2018/05/14 16:44:15 DEBUG : HTTP/1.1 429
Content-Length: 125
Content-Type: application/json; charset=utf-8
Date: Mon, 14 May 2018 14:44:15 GMT
Microsoftsharepointteamservices: 16.0.0.7625
P3p: CP=“ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI”
Spiislatency: 2
Sprequestduration: 184
X-Content-Type-Options: nosniff
X-Ms-Invokeapp: 1; RequireReadOnly
X-Msedge-Ref: Ref A: C009EDA7BAF2463BA1078F7371B2835D Ref B: VIEEDGE0616 Ref C: 2018-05-14T14:44:15Z
X-Powered-By: ASP.NET

{“error”:{“code”:“activityLimitReached”,“message”:“The request has been throttled”,“innerError”:{“code”:“throttledRequest”}}}
2018/05/14 16:44:15 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/05/14 16:44:15 DEBUG : pacer: Rate limited, increasing sleep to 80ms
2018/05/14 16:44:15 DEBUG : pacer: low level retry 3/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2018/05/14 16:44:15 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/05/14 16:44:15 DEBUG : HTTP REQUEST (req 0x422868e00)
2018/05/14 16:44:15 DEBUG : GET /_api/v2.0/drives/me/root:/ HTTP/1.1
Host: xxxx-my.sharepoint.com
User-Agent: rclone/v1.41
Authorization: XXXX
Accept-Encoding: gzip

2018/05/14 16:44:15 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/05/14 16:44:16 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/05/14 16:44:16 DEBUG : HTTP RESPONSE (req 0x422868e00)
2018/05/14 16:44:16 DEBUG : HTTP/1.1 429
Content-Length: 125
Content-Type: application/json; charset=utf-8
Date: Mon, 14 May 2018 14:44:15 GMT
Microsoftsharepointteamservices: 16.0.0.7625
P3p: CP=“ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI”
Spiislatency: 2
Sprequestduration: 139
X-Content-Type-Options: nosniff
X-Ms-Invokeapp: 1; RequireReadOnly
X-Msedge-Ref: Ref A: 2F315ECE5D8247BEB70077D913635301 Ref B: VIEEDGE0616 Ref C: 2018-05-14T14:44:16Z
X-Powered-By: ASP.NET

{“error”:{“code”:“activityLimitReached”,“message”:“The request has been throttled”,“innerError”:{“code”:“throttledRequest”}}}
2018/05/14 16:44:16 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/05/14 16:44:16 DEBUG : pacer: Rate limited, increasing sleep to 160ms
2018/05/14 16:44:16 DEBUG : pacer: low level retry 4/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2018/05/14 16:44:16 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/05/14 16:44:16 DEBUG : HTTP REQUEST (req 0x422869100)
2018/05/14 16:44:16 DEBUG : GET /_api/v2.0/drives/me/root:/ HTTP/1.1
Host: xxxx-my.sharepoint.com
User-Agent: rclone/v1.41
Authorization: XXXX
Accept-Encoding: gzip

2018/05/14 16:44:16 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/05/14 16:44:16 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/05/14 16:44:16 DEBUG : HTTP RESPONSE (req 0x422869100)
2018/05/14 16:44:16 DEBUG : HTTP/1.1 429
Content-Length: 125
Content-Type: application/json; charset=utf-8
Date: Mon, 14 May 2018 14:44:16 GMT
Microsoftsharepointteamservices: 16.0.0.7625
P3p: CP=“ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI”
Spiislatency: 2
Sprequestduration: 183
X-Content-Type-Options: nosniff
X-Ms-Invokeapp: 1; RequireReadOnly
X-Msedge-Ref: Ref A: 72A24E88E82C4D7B9C3EA914E59B8201 Ref B: VIEEDGE0616 Ref C: 2018-05-14T14:44:16Z
X-Powered-By: ASP.NET

{“error”:{“code”:“activityLimitReached”,“message”:“The request has been throttled”,“innerError”:{“code”:“throttledRequest”}}}
2018/05/14 16:44:16 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/05/14 16:44:16 DEBUG : pacer: Rate limited, increasing sleep to 320ms
2018/05/14 16:44:16 DEBUG : pacer: low level retry 5/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2018/05/14 16:44:16 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/05/14 16:44:16 DEBUG : HTTP REQUEST (req 0x422869400)
2018/05/14 16:44:16 DEBUG : GET /_api/v2.0/drives/me/root:/ HTTP/1.1
Host: xxxx-my.sharepoint.com
User-Agent: rclone/v1.41
Authorization: XXXX
Accept-Encoding: gzip

2018/05/14 16:44:16 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/05/14 16:44:16 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/05/14 16:44:16 DEBUG : HTTP RESPONSE (req 0x422869400)
2018/05/14 16:44:16 DEBUG : HTTP/1.1 429
Content-Length: 125
Content-Type: application/json; charset=utf-8
Date: Mon, 14 May 2018 14:44:16 GMT
Microsoftsharepointteamservices: 16.0.0.7625
P3p: CP=“ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI”
Spiislatency: 1
Sprequestduration: 151
X-Content-Type-Options: nosniff
X-Ms-Invokeapp: 1; RequireReadOnly
X-Msedge-Ref: Ref A: A4D7C5110A59417395CFACB228D3C599 Ref B: VIEEDGE0616 Ref C: 2018-05-14T14:44:16Z
X-Powered-By: ASP.NET

{“error”:{“code”:“activityLimitReached”,“message”:“The request has been throttled”,“innerError”:{“code”:“throttledRequest”}}}
2018/05/14 16:44:16 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/05/14 16:44:16 DEBUG : pacer: Rate limited, increasing sleep to 640ms
2018/05/14 16:44:16 DEBUG : pacer: low level retry 6/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2018/05/14 16:44:16 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/05/14 16:44:16 DEBUG : HTTP REQUEST (req 0x422869700)
2018/05/14 16:44:16 DEBUG : GET /_api/v2.0/drives/me/root:/ HTTP/1.1
Host: xxxx-my.sharepoint.com
User-Agent: rclone/v1.41
Authorization: XXXX
Accept-Encoding: gzip

2018/05/14 16:44:16 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/05/14 16:44:16 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/05/14 16:44:16 DEBUG : HTTP RESPONSE (req 0x422869700)
2018/05/14 16:44:16 DEBUG : HTTP/1.1 429
Content-Length: 125
Content-Type: application/json; charset=utf-8
Date: Mon, 14 May 2018 14:44:16 GMT
Microsoftsharepointteamservices: 16.0.0.7625
P3p: CP=“ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI”
Spiislatency: 2
Sprequestduration: 153
X-Content-Type-Options: nosniff
X-Ms-Invokeapp: 1; RequireReadOnly
X-Msedge-Ref: Ref A: 7C64DB326DE4490F94DE0060E0795405 Ref B: VIEEDGE0616 Ref C: 2018-05-14T14:44:16Z
X-Powered-By: ASP.NET

{“error”:{“code”:“activityLimitReached”,“message”:“The request has been throttled”,“innerError”:{“code”:“throttledRequest”}}}
2018/05/14 16:44:16 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/05/14 16:44:16 DEBUG : pacer: Rate limited, increasing sleep to 1.28s
2018/05/14 16:44:16 DEBUG : pacer: low level retry 7/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2018/05/14 16:44:17 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/05/14 16:44:17 DEBUG : HTTP REQUEST (req 0x42269ad00)
2018/05/14 16:44:17 DEBUG : GET /_api/v2.0/drives/me/root:/ HTTP/1.1
Host: xxxx-my.sharepoint.com
User-Agent: rclone/v1.41
Authorization: XXXX
Accept-Encoding: gzip

2018/05/14 16:44:17 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/05/14 16:44:17 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/05/14 16:44:17 DEBUG : HTTP RESPONSE (req 0x42269ad00)
2018/05/14 16:44:17 DEBUG : HTTP/1.1 429
Content-Length: 125
Content-Type: application/json; charset=utf-8
Date: Mon, 14 May 2018 14:44:17 GMT
Microsoftsharepointteamservices: 16.0.0.7625
P3p: CP=“ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI”
Spiislatency: 2
Sprequestduration: 202
X-Content-Type-Options: nosniff
X-Ms-Invokeapp: 1; RequireReadOnly
X-Msedge-Ref: Ref A: 1AACEB2A5709447D9DC0FFFDF4A185C8 Ref B: VIEEDGE0616 Ref C: 2018-05-14T14:44:17Z
X-Powered-By: ASP.NET

{“error”:{“code”:“activityLimitReached”,“message”:“The request has been throttled”,“innerError”:{“code”:“throttledRequest”}}}
2018/05/14 16:44:17 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/05/14 16:44:17 DEBUG : pacer: Rate limited, increasing sleep to 2s
2018/05/14 16:44:17 DEBUG : pacer: low level retry 8/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2018/05/14 16:44:18 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/05/14 16:44:18 DEBUG : HTTP REQUEST (req 0x422869b00)
2018/05/14 16:44:18 DEBUG : GET /_api/v2.0/drives/me/root:/ HTTP/1.1
Host: xxxx-my.sharepoint.com
User-Agent: rclone/v1.41
Authorization: XXXX
Accept-Encoding: gzip

2018/05/14 16:44:18 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/05/14 16:44:18 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/05/14 16:44:18 DEBUG : HTTP RESPONSE (req 0x422869b00)
2018/05/14 16:44:18 DEBUG : HTTP/1.1 429
Content-Length: 125
Content-Type: application/json; charset=utf-8
Date: Mon, 14 May 2018 14:44:18 GMT
Microsoftsharepointteamservices: 16.0.0.7625
P3p: CP=“ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI”
Spiislatency: 2
Sprequestduration: 177
X-Content-Type-Options: nosniff
X-Ms-Invokeapp: 1; RequireReadOnly
X-Msedge-Ref: Ref A: 17CDDD873DEB4771B7E30C22708BCAC4 Ref B: VIEEDGE0616 Ref C: 2018-05-14T14:44:18Z
X-Powered-By: ASP.NET

{“error”:{“code”:“activityLimitReached”,“message”:“The request has been throttled”,“innerError”:{“code”:“throttledRequest”}}}
2018/05/14 16:44:18 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/05/14 16:44:18 DEBUG : pacer: low level retry 9/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2018/05/14 16:44:20 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/05/14 16:44:20 DEBUG : HTTP REQUEST (req 0x42269b100)
2018/05/14 16:44:20 DEBUG : GET /_api/v2.0/drives/me/root:/ HTTP/1.1
Host: xxxx-my.sharepoint.com
User-Agent: rclone/v1.41
Authorization: XXXX
Accept-Encoding: gzip

2018/05/14 16:44:20 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/05/14 16:44:21 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/05/14 16:44:21 DEBUG : HTTP RESPONSE (req 0x42269b100)
2018/05/14 16:44:21 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: no-cache, no-store
Content-Type: application/json;odata.metadata=minimal;odata.streaming=true;IEEE754Compatible=false;charset=utf-8
Date: Mon, 14 May 2018 14:44:21 GMT
Expires: -1
Microsoftsharepointteamservices: 16.0.0.7625
Ms-Cv: nmcUjYywAFC4dbqPEm39oA.0
Odata-Version: 4.0
P3p: CP=“ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI”
Pragma: no-cache
Request-Id: 8d14679e-b08c-5000-b875-ba8f126dfda0
Spclientservicerequestduration: 516
Sprequestduration: 714
Sprequestguid: 8d14679e-b08c-5000-b875-ba8f126dfda0
Strict-Transport-Security: max-age=31536000
Vary: Accept-Encoding
X-Aspnet-Version: 4.0.30319
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Ms-Invokeapp: 1; RequireReadOnly
X-Msedge-Ref: Ref A: BB2F3F2C9A114DC08990AC593BBDF412 Ref B: VIEEDGE0616 Ref C: 2018-05-14T14:44:21Z
X-Powered-By: ASP.NET
X-Sharepointhealthscore: 4
X-Sp-Serverstate: ReadOnly=0

39b
{"@odata.context":"https://xxxx-my.sharepoint.com/_api/v2.0/$metadata#items/$entity","@odata.type":"#oneDrive.item","@odata.id":"https://XXXX-my.sharepoint.com/_api/v2.0/drives/b!Ja…“folder”:{“childCount”:3},“root”:{},“size”:858941}
0

2018/05/14 16:44:21 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/05/14 16:44:21 DEBUG : pacer: Reducing sleep to 1.5s
2018/05/14 16:44:21 INFO : One drive root ‘’: Modify window is 1s
2018/05/14 16:44:22 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/05/14 16:44:22 DEBUG : HTTP REQUEST (req 0x422c94200)
2018/05/14 16:44:22 DEBUG : GET /_api/v2.0/drives/me/items/01IEGVLQN6Y2GOVW7725BZO354PWSELRRZ/children?top=1000 HTTP/1.1
Host: xxxx-my.sharepoint.com
User-Agent: rclone/v1.41
Authorization: XXXX
Accept-Encoding: gzip

2018/05/14 16:44:22 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/05/14 16:44:23 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/05/14 16:44:23 DEBUG : HTTP RESPONSE (req 0x422c94200)
2018/05/14 16:44:23 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: no-cache, no-store
Content-Type: application/json;odata.metadata=minimal;odata.streaming=true;IEEE754Compatible=false;charset=utf-8
Date: Mon, 14 May 2018 14:44:23 GMT
Expires: -1
Microsoftsharepointteamservices: 16.0.0.7625
Ms-Cv: nmcUjgmwAFC4dbzw8j6Jpw.0
Odata-Version: 4.0
P3p: CP=“ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI”
Pragma: no-cache
Request-Id: 8e14679e-b009-5000-b875-bcf0f23e89a7
Spclientservicerequestduration: 536
Sprequestduration: 874
Sprequestguid: 8e14679e-b009-5000-b875-bcf0f23e89a7
Strict-Transport-Security: max-age=31536000
Vary: Accept-Encoding
X-Aspnet-Version: 4.0.30319
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Ms-Invokeapp: 1; RequireReadOnly
X-Msedge-Ref: Ref A: 2F01AA67E14A4697A60623D39CD5183E Ref B: VIEEDGE0616 Ref C: 2018-05-14T14:44:23Z
X-Powered-By: ASP.NET
X-Sharepointhealthscore: 4
X-Sp-Serverstate: ReadOnly=0

1090
{"@odata.context":"https://xxxx-my.sharepoint.com/_api/v2.0/$metadata#items",“value”:[{"@odata.type":"#oneDrive.item","@odata.id":"https://XXXX-my.sharepoint.com/_api/v2.0/drives/b!Ja…}
0

2018/05/14 16:44:23 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/05/14 16:44:23 DEBUG : pacer: Reducing sleep to 1.125s
-1 2017-03-17 04:50:23 1 Attachments
-1 2015-05-05 06:05:25 1 Shared with Everyone
-1 2017-02-21 15:30:20 5 Infor reports
2018/05/14 16:44:23 DEBUG : 6 go routines active
2018/05/14 16:44:23 DEBUG : rclone: Version “v1.41” finishing with parameters [“rclone” “lsd” “–retries” “1” “-vv” “–dump” “bodies” “OneDriveforBusiness:”]


Onedrive for Business - activity limits and throttling
#9

I wonder if rclone is being too optimistic about backoff and retry timers for onedrive.

	minSleep                            = 10 * time.Millisecond
	maxSleep                            = 2 * time.Second

So it starts at 10ms and doubles it up to a maximum of 2S.

If you can compile rclone you can easily change these values.

Are there any official recommendations for backoff and retry periods for onedrive? Some of the providers (eg Google and Amazon) have an official strategy which rclone implements, but I didn’t see one for onedrive.


#10

Microsoft provides some recommendations here:

https://docs.microsoft.com/en-us/sharepoint/dev/general-development/how-to-avoid-getting-throttled-or-blocked-in-sharepoint-online

I especially like this part:

Why can’t you just tell me the exact throttling limits?

Setting and publishing exact throttling limits sounds very straightforward, but in fact, it’s not the best way to go. We continually monitor resource usage on SharePoint Online.
Depending on usage, we fine-tune thresholds so users can consume the maximum number of resources without degrading the reliability and performance of SharePoint Online.

In the provided code samples they are waiting 30 secs after each throttled request, and doubling the timeout each throttle received after that.

There appears to be some discussion on the OneDrive GitHub page:


#11

Hi, I’m not in a position to do builds so have to rely on developers’ expertise.

However, it looks increasingly likely that clone is currently optimistic, as you suggest.

I’d be happy to try a beta version with changes to those parameters taking into account weropol’s additional information below about 30 secs wait time. (Maybe could be shorter!?)


#12

Thanks for the useful links.

I looked through lots of sample code and there didn’t seem to be a consensus on what exact values to use. One bit of code used 200ms, another 30s!

Rclone uses the same backoff strategy for any retriable error (eg some internet error) so I don’t want to make the initial sleep too long. The above seems like a reasonable compromise hopefully.

I’ve made a test build here.

This

  • immediately on retry bumps the time up to 1s
  • will double that time to a max of 30s on futher errors

https://beta.rclone.org/v1.41-061-g51f3432a-onedrive-pacer/ (uploaded in 15-30 mins)

It be interested to see logs with -vv of it in action.

I note also that the -vv --dump bodies log indicates onedrive doesn’t send Retry-After: headers like it says in the docs :frowning:


#13

I can see the retry logic is working properly which is good, however it doesn’t seem to have made it work a lot better :frowning:

I’m not decorating the User-Agent according to the specs.

This is something you could try to see if it makes a difference using the --user-agent flag.


#14

I can’t find anything in rclone Docs about any --user-agent flag or how to apply it? Sorry, if I appear a bit thick!


#15

Here is the help for it

  --user-agent string                   Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.41-061-g51f3432a-onedrive-pacer")

so just do --user-agent "user/agent" in the fancy form the Microsoft specify and maybe it will work better!


#16

2018/05/15 15:32:32 DEBUG : rclone: Version “v1.41-061-g51f3432a-onedrive-pacerβ” starting with parameters [“rclone” “copy” “OneDriveforBusiness:/xxx Infor reports” “StratusWebdav:/alfresco/webdav/Shared” “-vv” “–transfers” “2” “–user-agent” “rclone/v1.41-061-g51f3432a-onedrive-pacerβ”]
2018/05/15 15:32:32 DEBUG : Using config file from “/Users/admin/.config/rclone/rclone.conf”
2018/05/15 15:32:33 DEBUG : pacer: Rate limited, sleeping for 1s (1 consecutive low level retries)
2018/05/15 15:32:33 DEBUG : pacer: low level retry 1/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2018/05/15 15:32:34 DEBUG : pacer: Rate limited, sleeping for 2s (2 consecutive low level retries)
2018/05/15 15:32:34 DEBUG : pacer: low level retry 2/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2018/05/15 15:32:35 DEBUG : pacer: Rate limited, sleeping for 4s (3 consecutive low level retries)
2018/05/15 15:32:35 DEBUG : pacer: low level retry 3/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2018/05/15 15:32:37 DEBUG : pacer: Rate limited, sleeping for 8s (4 consecutive low level retries)
2018/05/15 15:32:37 DEBUG : pacer: low level retry 4/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2018/05/15 15:32:41 DEBUG : pacer: Rate limited, sleeping for 16s (5 consecutive low level retries)
2018/05/15 15:32:41 DEBUG : pacer: low level retry 5/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2018/05/15 15:32:49 DEBUG : pacer: Rate limited, sleeping for 30s (6 consecutive low level retries)
2018/05/15 15:32:49 DEBUG : pacer: low level retry 6/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2018/05/15 15:33:05 DEBUG : pacer: Rate limited, sleeping for 30s (7 consecutive low level retries)
2018/05/15 15:33:05 DEBUG : pacer: low level retry 7/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2018/05/15 15:33:35 DEBUG : pacer: Rate limited, sleeping for 30s (8 consecutive low level retries)
2018/05/15 15:33:35 DEBUG : pacer: low level retry 8/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2018/05/15 15:34:05 DEBUG : pacer: Rate limited, sleeping for 30s (9 consecutive low level retries)
2018/05/15 15:34:05 DEBUG : pacer: low level retry 9/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2018/05/15 15:34:35 DEBUG : pacer: Rate limited, sleeping for 30s (10 consecutive low level retries)
2018/05/15 15:34:35 DEBUG : pacer: low level retry 10/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2018/05/15 15:34:35 Failed to create file system for “OneDriveforBusiness:/xxx Infor reports”: failed to get root: activityLimitReached: throttledRequest: The request has been throttled

:cry:


#17

Try using something like --user-agent "ISV|rclone.org|rclone/v1.41" like it recommends in the docs.

That might help… Maybe!


#18

Applying the recommended format has resulted in a success but it took 47 minutes to transfer just 4 small documents. Of course, there are many factors that could be at play here so I’m not able to say that this was the defining factor. But at least I have a proof-of-concept to work with now, so I won’t give up at this stage and can now begin to try transfers from ODB to some other end-points, to compare results.

Thanks again for your support.


#19

I don’t think that is working properly somehow!

Let me know what you discover :smile: