Error reading destination directory: couldn't list files: 413 Request Entity Too Large

pcf000 · July 20, 2023, 11:54pm

What is the problem you are having with rclone?

When syncing with OneDrive via WebDAV, I have "413 Request Entity Too Large" errors.

Specifically, I have them when trying to read directories with more than 5000 items during the sync process. Writing them works okay, reading gets the error. The 5000-item limit is a hard-coded limit in webdav for onedrive (see Manage large lists and libraries - Microsoft Support) and, since it's the response to PROPFIND, may require some creative workarounds.

I believe this is the same problem reported in Couldn't list files: 413 Request Entity Too Large on copying .

Run the command 'rclone version' and share the full output of the command.

rclone v1.63.0

os/version: ubuntu 22.04 (64 bit)
os/kernel: 5.15.0-76-generic (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.20.5
go/linking: static
go/tags: none

Which cloud storage system are you using? (eg Google Drive)

OneDrive for business, via webdav. The webdav part is important, because I'm stuck with it. My company has bought heavily into the Microsoft ecosystem, even though my department develops Linux software. The official support response to my request to authorise rclone for OneDrive access was "use www.office.com to access OneDrive from a browser in Linux and backup using that website."

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone --quiet sync ~/ chunked: --user-agent="ISV|rclone.org|rclone/v1.56.2" --links --webdav-encoding=Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,LeftSpace,LeftTilde,RightSpace,RightPeriod,InvalidUtf8,Dot --compress-level 5 --exclude-from $fn --transfers 1 --tpslimit 10 --onedrive-list-chunk 100 --retries 1 --ignore-errors

The rclone config contents with secrets removed.

[onedrive]
type = webdav
url = https://COMPANY.sharepoint.com/personal/USER_COMPANY_com/Documents
vendor = sharepoint
user = USER@COMPANY.com
pass = <stuff>

[chunked]
type = chunker
remote = onedrive:backup
chunk_size = 256Mi

A log from the command with the `-vv --dump bodies` flags

2023/07/20 18:39:58 INFO  : Starting transaction limiter: max 10 transactions/s with burst 1
2023/07/20 18:39:58 DEBUG : rclone: Version "v1.63.0" starting with parameters ["rclone" "sync" "/home/pf/Maildir/COMPANY/inbox/cur" "chunked:Maildir/COMPANY/inbox/cur" "--user-agent=ISV|rclone.org|rclone/v1.56.2" "--links" "--webdav-encoding=Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,LeftSpace,LeftTilde,RightSpace,RightPeriod,InvalidUtf8,Dot" "--compress-level" "5" "--transfers" "1" "--tpslimit" "10" "--retries" "1" "-vv" "--dump" "bodies"]
2023/07/20 18:39:58 DEBUG : Creating backend with remote "/home/pf/Maildir/COMPANY/inbox/cur"
2023/07/20 18:39:58 DEBUG : Using config file from "/home/pf/.config/rclone/rclone.conf"
2023/07/20 18:39:58 DEBUG : local: detected overridden config - adding "{b6816}" suffix to name
2023/07/20 18:39:58 DEBUG : fs cache: renaming cache item "/home/pf/Maildir/COMPANY/inbox/cur" to be canonical "local{b6816}:/home/pf/Maildir/COMPANY/inbox/cur"
2023/07/20 18:39:58 DEBUG : Creating backend with remote "chunked:Maildir/COMPANY/inbox/cur"
2023/07/20 18:39:58 DEBUG : Creating backend with remote "onedrive:backup/Maildir/COMPANY/inbox/cur"
2023/07/20 18:39:58 DEBUG : onedrive: detected overridden config - adding "{CXKwz}" suffix to name
2023/07/20 18:39:58 DEBUG : found headers:
2023/07/20 18:39:58 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2023/07/20 18:39:58 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/07/20 18:39:58 DEBUG : HTTP REQUEST (req 0xc000bdc300)
2023/07/20 18:39:58 DEBUG : POST /extSTS.srf HTTP/1.1
Host: login.microsoftonline.com
User-Agent: ISV|rclone.org|rclone/v1.56.2
Content-Length: 1358
Accept-Encoding: gzip

<s:Envelope xmlns:s="http://www.w3.org/2003/05/soap-envelope"
xmlns:a="http://www.w3.org/2005/08/addressing"
xmlns:u="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">
<s:Header>
<a:Action s:mustUnderstand="1">http://schemas.xmlsoap.org/ws/2005/02/trust/RST/Issue</a:Action>
<a:ReplyTo>
<a:Address>http://www.w3.org/2005/08/addressing/anonymous</a:Address>
</a:ReplyTo>
<a:To s:mustUnderstand="1">https://login.microsoftonline.com/extSTS.srf</a:To>
<o:Security s:mustUnderstand="1"
 xmlns:o="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd">
<o:UsernameToken>
  <o:Username>USER@COMPANY.com</o:Username>
  <o:Password>not-the-password</o:Password>
</o:UsernameToken>
</o:Security>
</s:Header>
<s:Body>
<t:RequestSecurityToken xmlns:t="http://schemas.xmlsoap.org/ws/2005/02/trust">
<wsp:AppliesTo xmlns:wsp="http://schemas.xmlsoap.org/ws/2004/09/policy">
  <a:EndpointReference>
    <a:Address>https://COMPANY.sharepoint.com/personal/USER_COMPANY_com/Documents/</a:Address>
  </a:EndpointReference>
</wsp:AppliesTo>
<t:KeyType>http://schemas.xmlsoap.org/ws/2005/05/identity/NoProofKey</t:KeyType>
<t:RequestType>http://schemas.xmlsoap.org/ws/2005/02/trust/Issue</t:RequestType>
<t:TokenType>urn:oasis:names:tc:SAML:1.0:assertion</t:TokenType>
</t:RequestSecurityToken>
</s:Body>
</s:Envelope>
2023/07/20 18:39:58 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/07/20 18:39:59 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/07/20 18:39:59 DEBUG : HTTP RESPONSE (req 0xc000bdc300)
2023/07/20 18:39:59 DEBUG : HTTP/1.1 200 OK
Content-Length: 3568
Cache-Control: no-store, no-cache
Content-Type: application/soap+xml; charset=utf-8
Date: Thu, 20 Jul 2023 23:39:58 GMT
Expires: -1
P3p: CP="DSP CUR OTPi IND OTRi ONL FIN"
Pragma: no-cache
Set-Cookie: fpc=Aqk8GsndNT5Crz9SMQdC5YOzuFdjAQAAAM68S9wOAAAA; expires=Sat, 19-Aug-2023 23:39:59 GMT; path=/; secure; HttpOnly; SameSite=None
Set-Cookie: x-ms-gateway-slice=estsfd; path=/; secure; samesite=none; httponly
Set-Cookie: stsservicecookie=estsfd; path=/; secure; samesite=none; httponly
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
X-Ms-Ests-Server: 2.1.15868.4 - WUS2 ProdSlices
X-Ms-Request-Id: 10bd5348-7824-4396-adb0-b16c7ed54500
X-Xss-Protection: 0

<?xml version="1.0" encoding="utf-8"?><S:Envelope xmlns:wsa="http://www.w3.org/2005/08/addressing" xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd" xmlns:wsp="http://schemas.xmlsoap.org/ws/2004/09/policy" xmlns:wst="http://schemas.xmlsoap.org/ws/2005/02/trust" xmlns:S="http://www.w3.org/2003/05/soap-envelope"><S:Header><wsa:Action S:mustUnderstand="1" wsu:Id="Action">http://schemas.xmlsoap.org/ws/2005/02/trust/RSTR/Issue</wsa:Action><wsa:To S:mustUnderstand="1" wsu:Id="To">http://schemas.xmlsoap.org/ws/2004/08/addressing/role/anonymous</wsa:To><wsse:Security S:mustUnderstand="1"><wsu:Timestamp wsu:Id="TS" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd"><wsu:Created>2023-07-20T23:39:59.1614195Z</wsu:Created><wsu:Expires>2023-07-20T23:44:59.1614195Z</wsu:Expires></wsu:Timestamp></wsse:Security></S:Header><S:Body xmlns:S="http://www.w3.org/2003/05/soap-envelope"><wst:RequestSecurityTokenResponse xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd" xmlns:wsp="http://schemas.xmlsoap.org/ws/2004/09/policy" xmlns:wst="http://schemas.xmlsoap.org/ws/2005/02/trust"><wst:TokenType>urn:passport:compact</wst:TokenType><wsp:AppliesTo><wsa:EndpointReference xmlns:wsa="http://www.w3.org/2005/08/addressing"><wsa:Address>https://COMPANY.sharepoint.com/personal/USER_COMPANY_com/Documents/</wsa:Address></wsa:EndpointReference></wsp:AppliesTo><wst:Lifetime><wsu:Created>2023-07-20T23:39:58Z</wsu:Created><wsu:Expires>2023-07-21T23:39:58Z</wsu:Expires></wst:Lifetime><wst:RequestedSecurityToken><wsse:BinarySecurityToken Id="Compact0" xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd">t=EwDwAk6hBwAU7VLCn7giL6WPlB2UHeyTOgkXkIkAAYZYuIiN1uDtql1vCLqaHeG52fdMTLuajOecP2Mgl40WSPuYWGQ2h997LXBIA0CTwL0BZs4B5wVOpdul67lhs6soTy6tlleHH6gG2hXsFpI+JbJBv3dN0iW3jkXNHUsZaT5XWBGVvYk3j2SM6pC5oURdPpAsPZOc0buxT5uO503hZRv0yi3jp5BisPoVBxHi7nlcU8EAjo8Zoq4oARANjWQ9MJxUG5MtGE0hVEta6AKSCo4pAdg9drHpn8qgGcbVoQNrUUL5GyK6TdKYKW47tJ8dOEMNIizZwjCHQCzKQ0IgHHPh/0SYw7MGPr+D/fl8FqN8Hk1XSWoyLYdLpbSRhmkDZgAACHwiUdGGYiYVwAG12saCMShBIcI849t1Xt/Pd/UphM+ADgDCXV1eTnBatagnpNJFEHpPpvIUDBAb7hhHKZJLyLTIMQpOVYcslHCYIBld39zTaNRefYbBbWQnHG3o3xiRnlwzc4qfNKnbWpx6uR0j01+b5afJzyDP1UK1tevdDLCr/ZuTqxD1qE+l701MI8A7iLT0dPs/do+qeJ7vsakNZSGVF8qZc/ad+sDZk715DvEVg9YA2Hdnu2KwFIuTD5goANIqST3lpXcbcjxfr0q9foz17jY7DU+acDlygu8/fZ8535DFxUbSm2l/tl8pvJWEfg9Ao0XFDFjZhDEOd4yqzqW9R/aGNw64l+2L7k1kUrgMBzKKjqASYX7qXQoRV3GJikR5vJWtHlLsbhyn6dGtdArReR4j29sKj9wGF8of0XbFtJxJmAv0SpWyqFQYFSENUn5Z7I9y/xu/jJNrzjs5R7gtx4EsMJNTGKgm64BCJXRhbhNqzP+1KV6MVE9laiasodB9TrtwR7upVNXVrf48LHwjJFI6kVkYWfh9lIH+YoszpUIJ2o9j4gGvKz0/U1+JUuAnDuByeYkoEucelKRd87Mn70LRfXm5MllKGQI=&amp;p=</wsse:BinarySecurityToken></wst:RequestedSecurityToken><wst:RequestedAttachedReference><wsse:SecurityTokenReference xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd"><wsse:Reference URI="Vz5Dg/rQg2/4eP80yF92OKNx/Ak="></wsse:Reference></wsse:SecurityTokenReference></wst:RequestedAttachedReference><wst:RequestedUnattachedReference><wsse:SecurityTokenReference xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd"><wsse:Reference URI="Vz5Dg/rQg2/4eP80yF92OKNx/Ak="></wsse:Reference></wsse:SecurityTokenReference></wst:RequestedUnattachedReference></wst:RequestSecurityTokenResponse></S:Body></S:Envelope>
2023/07/20 18:39:59 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/07/20 18:39:59 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/07/20 18:39:59 DEBUG : HTTP REQUEST (req 0xc000d56900)
2023/07/20 18:39:59 DEBUG : PROPFIND /personal/USER_COMPANY_com/Documents/backup/Maildir/COMPANY/inbox/cur HTTP/1.1
Host: COMPANY.sharepoint.com
User-Agent: ISV|rclone.org|rclone/v1.56.2
Cookie: FedAuth=77u/PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0idXRmLTgiPz48U1A+VjEyLDBoLmZ8bWVtYmVyc2hpcHwxMDAzMjAwMTIxNDkyMjk2QGxpdmUuY29tLDAjLmZ8bWVtYmVyc2hpcHxwYXVmdXF1YUBhbWQuY29tLDEzMzM0MzY5OTk5MDAwMDAwMCwxMzMyOTc5NjU1MjAwMDAwMDAsMTMzMzQ0NTYzOTkzMzI3NDE4LDAuMC4wLjAsMTA0ODU3OCwzZGQ4OTYxZi1lNDg4LTRlNjAtOGUxMS1hODJkOTk0ZTE4M2QsLCw5ODdkYzhhMC02MDlhLTMwMDAtZGU4Ni03NWYyZjRhNmFkNDEsOTg3ZGM4YTAtNjA5YS0zMDAwLWRlODYtNzVmMmY0YTZhZDQxLCwwLDAsMCwsLCwyNjUwNDY3NzQzOTk5OTk5OTk5LDAsLCwsLCwsMCwsbThqK0xxU2psMEtDbE1RWWlpWmVoRHFhazhlM2NHTmpBZ2hlWG96ZGgrbUowT0Y1K05tVk96SXB6VTRaNmc2RWlwRGNkaFVhbm9Qc21hZW4xUUp4dHlOUFF1UDRCcXBQMnhRdXlMd3NDZUx4TWltZjJ1WUlyMk9jRm5JY1cxaHNmbjZ6ZTl0dXNueWhNRW5NSW9PU3VQMlhmcHdkcUY0cEFzL25QNVZ1bnV5K2R4SE0xY0p0OGJ4dW9zM0JoaVpvYTZtN3pqWGJlaGF6WnduQzR4R1dLcEcvQ2ljeHRucjlTMU10ZURaZ2U3KzRkM0o1d1VRbTc4VUFWQVljN1E5THFySDZwYy9zcGNvNm5Qd1FDdnl3UTB4eElpNUt6ODJCQnRqcmRpN1cxUmFzcG5GN2xmY283MFdJNFdSN2ZpSCs5Uy90YlhlRTlVdW5pVk9ON1ZJVi9BPT08L1NQPg==; rtFa=g8sVkxlL6zXrGFDHNUyWg4uJVRwJ+vhYoqsG5spY7rcmM0REODk2MUYtRTQ4OC00RTYwLThFMTEtQTgyRDk5NEUxODNEIzEzMzM0MzY5OTk5MzQ4NDM0NSM5ODdEQzhBMC02MDlBLTMwMDAtREU4Ni03NUYyRjRBNkFENDEjUEFVRlVRVUElNDBBTUQuQ09NdjBkl9BVv6HXy6LW8qDYb5cwMGYEgvDDwAXbxlKcKbMcXcqy1hCuYVtu9pJsidXHY1TeCukHfLIgDH5SiAJJD8Aayz3hkuStknqvH3cqrM2+GtSGVmfgciVTL9mi5KDlrC8fvhfCGIovouhjpQl8K5zyl7k662Ub5rzJegNOgflDYYnkF4Uu64QAhc/KdOoqAZwTcTMzUEDgy9XiquBY58GuuKaHKL1Rv7uh/J0blbZ0BDEQF5rH5EMbEySmwjFJChKRobROQX1OHOioFsBHWwDSF6RU49tImYw4I80+xbz7opVPqHQJkPiSxFbDyMO2MPe5BsTVliwU4Ht1kJV2G5AAAAA=
Depth: 1
Referer: https://COMPANY.sharepoint.com/personal/USER_COMPANY_com/Documents/
Accept-Encoding: gzip

2023/07/20 18:39:59 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/07/20 18:39:59 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/07/20 18:39:59 DEBUG : HTTP RESPONSE (req 0xc000d56900)
2023/07/20 18:39:59 DEBUG : HTTP/2.0 413 Request Entity Too Large
Accept-Ranges: bytes
Cache-Control: private,max-age=0
Content-Security-Policy: frame-ancestors 'self' teams.microsoft.com *.teams.microsoft.com *.skype.com *.teams.microsoft.us local.teams.office.com teams.microsoftonline.cn *.powerapps.com *.yammer.com *.officeapps.live.com *.office.com *.stream.azure-test.net *.microsoftstream.com *.dynamics.com *.microsoft.com onedrive.live.com *.onedrive.live.com securebroker.sharepointonline.com;
Date: Thu, 20 Jul 2023 23:39:59 GMT
Expires: Wed, 05 Jul 2023 23:39:59 GMT
Microsoftsharepointteamservices: 16.0.0.23905
Ms-Cv: oMh9mLsQADDehn8Cq3xTqw.0
P3p: CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI"
Public-Extension: http://schemas.microsoft.com/repl-2
Request-Id: 987dc8a0-10bb-3000-de86-7f02ab7c53ab
Set-Cookie: rtFa=g8sVkxlL6zXrGFDHNUyWg4uJVRwJ+vhYoqsG5spY7rcmM0REODk2MUYtRTQ4OC00RTYwLThFMTEtQTgyRDk5NEUxODNEIzEzMzM0MzY5OTk5MzQ4NDM0NSM5ODdEQzhBMC02MDlBLTMwMDAtREU4Ni03NUYyRjRBNkFENDEjUEFVRlVRVUElNDBBTUQuQ09NdjBkl9BVv6HXy6LW8qDYb5cwMGYEgvDDwAXbxlKcKbMcXcqy1hCuYVtu9pJsidXHY1TeCukHfLIgDH5SiAJJD8Aayz3hkuStknqvH3cqrM2+GtSGVmfgciVTL9mi5KDlrC8fvhfCGIovouhjpQl8K5zyl7k662Ub5rzJegNOgflDYYnkF4Uu64QAhc/KdOoqAZwTcTMzUEDgy9XiquBY58GuuKaHKL1Rv7uh/J0blbZ0BDEQF5rH5EMbEySmwjFJChKRobROQX1OHOioFsBHWwDSF6RU49tImYw4I80+xbz7opVPqHQJkPiSxFbDyMO2MPe5BsTVliwU4Ht1kJV2G5AAAAA=; domain=sharepoint.com; path=/; SameSite=None; secure; HttpOnly
Set-Cookie: FedAuth=77u/PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0idXRmLTgiPz48U1A+VjEyLDBoLmZ8bWVtYmVyc2hpcHwxMDAzMjAwMTIxNDkyMjk2QGxpdmUuY29tLDAjLmZ8bWVtYmVyc2hpcHxwYXVmdXF1YUBhbWQuY29tLDEzMzM0MzY5OTk5MDAwMDAwMCwxMzMyOTc5NjU1MjAwMDAwMDAsMTMzMzQ0NTYzOTkzMzI3NDE4LDAuMC4wLjAsMTA0ODU3OCwzZGQ4OTYxZi1lNDg4LTRlNjAtOGUxMS1hODJkOTk0ZTE4M2QsLCw5ODdkYzhhMC02MDlhLTMwMDAtZGU4Ni03NWYyZjRhNmFkNDEsOTg3ZGM4YTAtNjA5YS0zMDAwLWRlODYtNzVmMmY0YTZhZDQxLCwwLDAsMCwsLCwyNjUwNDY3NzQzOTk5OTk5OTk5LDAsLCwsLCwsMCwsbThqK0xxU2psMEtDbE1RWWlpWmVoRHFhazhlM2NHTmpBZ2hlWG96ZGgrbUowT0Y1K05tVk96SXB6VTRaNmc2RWlwRGNkaFVhbm9Qc21hZW4xUUp4dHlOUFF1UDRCcXBQMnhRdXlMd3NDZUx4TWltZjJ1WUlyMk9jRm5JY1cxaHNmbjZ6ZTl0dXNueWhNRW5NSW9PU3VQMlhmcHdkcUY0cEFzL25QNVZ1bnV5K2R4SE0xY0p0OGJ4dW9zM0JoaVpvYTZtN3pqWGJlaGF6WnduQzR4R1dLcEcvQ2ljeHRucjlTMU10ZURaZ2U3KzRkM0o1d1VRbTc4VUFWQVljN1E5THFySDZwYy9zcGNvNm5Qd1FDdnl3UTB4eElpNUt6ODJCQnRqcmRpN1cxUmFzcG5GN2xmY283MFdJNFdSN2ZpSCs5Uy90YlhlRTlVdW5pVk9ON1ZJVi9BPT08L1NQPg==; path=/; SameSite=None; secure; HttpOnly
Set-Cookie: rtFa=g8sVkxlL6zXrGFDHNUyWg4uJVRwJ+vhYoqsG5spY7rcmM0REODk2MUYtRTQ4OC00RTYwLThFMTEtQTgyRDk5NEUxODNEIzEzMzM0MzY5OTk5MzQ4NDM0NSM5ODdEQzhBMC02MDlBLTMwMDAtREU4Ni03NUYyRjRBNkFENDEjUEFVRlVRVUElNDBBTUQuQ09NdjBkl9BVv6HXy6LW8qDYb5cwMGYEgvDDwAXbxlKcKbMcXcqy1hCuYVtu9pJsidXHY1TeCukHfLIgDH5SiAJJD8Aayz3hkuStknqvH3cqrM2+GtSGVmfgciVTL9mi5KDlrC8fvhfCGIovouhjpQl8K5zyl7k662Ub5rzJegNOgflDYYnkF4Uu64QAhc/KdOoqAZwTcTMzUEDgy9XiquBY58GuuKaHKL1Rv7uh/J0blbZ0BDEQF5rH5EMbEySmwjFJChKRobROQX1OHOioFsBHWwDSF6RU49tImYw4I80+xbz7opVPqHQJkPiSxFbDyMO2MPe5BsTVliwU4Ht1kJV2G5AAAAA=; domain=sharepoint.com; path=/; SameSite=None; secure; HttpOnly
Set-Cookie: FedAuth=77u/PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0idXRmLTgiPz48U1A+VjEyLDBoLmZ8bWVtYmVyc2hpcHwxMDAzMjAwMTIxNDkyMjk2QGxpdmUuY29tLDAjLmZ8bWVtYmVyc2hpcHxwYXVmdXF1YUBhbWQuY29tLDEzMzM0MzY5OTk5MDAwMDAwMCwxMzMyOTc5NjU1MjAwMDAwMDAsMTMzMzQ0NTYzOTkzMzI3NDE4LDAuMC4wLjAsMTA0ODU3OCwzZGQ4OTYxZi1lNDg4LTRlNjAtOGUxMS1hODJkOTk0ZTE4M2QsLCw5ODdkYzhhMC02MDlhLTMwMDAtZGU4Ni03NWYyZjRhNmFkNDEsOTg3ZGM4YTAtNjA5YS0zMDAwLWRlODYtNzVmMmY0YTZhZDQxLCwwLDAsMCwsLCwyNjUwNDY3NzQzOTk5OTk5OTk5LDAsLCwsLCwsMCwsbThqK0xxU2psMEtDbE1RWWlpWmVoRHFhazhlM2NHTmpBZ2hlWG96ZGgrbUowT0Y1K05tVk96SXB6VTRaNmc2RWlwRGNkaFVhbm9Qc21hZW4xUUp4dHlOUFF1UDRCcXBQMnhRdXlMd3NDZUx4TWltZjJ1WUlyMk9jRm5JY1cxaHNmbjZ6ZTl0dXNueWhNRW5NSW9PU3VQMlhmcHdkcUY0cEFzL25QNVZ1bnV5K2R4SE0xY0p0OGJ4dW9zM0JoaVpvYTZtN3pqWGJlaGF6WnduQzR4R1dLcEcvQ2ljeHRucjlTMU10ZURaZ2U3KzRkM0o1d1VRbTc4VUFWQVljN1E5THFySDZwYy9zcGNvNm5Qd1FDdnl3UTB4eElpNUt6ODJCQnRqcmRpN1cxUmFzcG5GN2xmY283MFdJNFdSN2ZpSCs5Uy90YlhlRTlVdW5pVk9ON1ZJVi9BPT08L1NQPg==; path=/; SameSite=None; secure; HttpOnly
Spiislatency: 1
Sprequestduration: 19
Sprequestguid: 987dc8a0-10bb-3000-de86-7f02ab7c53ab
Strict-Transport-Security: max-age=31536000
X-1dscollectorurl: https://mobile.events.data.microsoft.com/OneCollector/1.0/
X-Ariacollectorurl: https://browser.pipe.aria.microsoft.com/Collector/3.0/
X-Cache: CONFIG_NOCACHE
X-Content-Type-Options: nosniff
X-Databoundary: NONE
X-Frame-Options: SAMEORIGIN
X-Ms-Invokeapp: 1; RequireReadOnly
X-Msdavext_error: 1966171; The%20attempted%20operation%20is%20prohibited%20because%20it%20exceeds%20the%20list%20view%20threshold%2e
X-Msedge-Ref: Ref A: 72CF3FD39BFD4E69B33DD93233CE8111 Ref B: DFW311000108027 Ref C: 2023-07-20T23:39:59Z
X-Networkstatistics: 0,4204800,0,145,906458,0,2726079
X-Powered-By: ASP.NET
X-Sharepointhealthscore: 2
Content-Length: 0

2023/07/20 18:39:59 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/07/20 18:39:59 Failed to create file system for "chunked:Maildir/COMPANY/inbox/cur": failed to make remote "onedrive:backup/Maildir/COMPANY/inbox/cur" to wrap: read metadata failed: 413 Request Entity Too Large

kapitainsky · July 21, 2023, 1:04pm

It sounds like there is no no way around it.

This offers some glimpse of hope...

You could try to use http remote. It would require to get office session cookie from browser and then using --set-cookie flag to pass it to rclone. Never tried it on office.com but it can be worth to try. But it would only allow reading files and I see you want to sync from local to remote. Unfortunately it is only option I can think about.

kapitainsky · July 21, 2023, 1:11pm

Other thing you could try is to use:

--no-traverse Don't traverse destination file system on copy

so you could blindly attempt to copy files to destination using rclone copy instead of rclone sync

ncw · July 22, 2023, 8:45am

I'm pretty sure this will work via the ondrive protocol as I remember adding this to help with listing very large directories with 50,000 entries.

  --onedrive-list-chunk int   Size of listing chunk (default 1000)

Any ideas? I don't think you can page the webdav listings so I can't think of any.

pcf000 · July 24, 2023, 5:52pm

Note that "creative" in this case means "non-trivial and probably a pain in the ass." Mostly I wanted to get the limitation on record.

One thought is to automagically break large directories into smaller ones, analogous to what chunker does to files. Possibly that should even be in chunker. Large directories that are already on the server are hopeless, though.

Another is to use whatever API commands happen at browser level to do the infinite-scrolling trick, to fetch the list in chunks, but I don't know if that allows for fetching properties or doing anything useful besides display. That also means another specialised OneDrive backend, ugh. Is there any way to do a webdav connection but send onedrive commands? Does that question even make sense?

For my own part, I'm exploring backing up to an internal SMB server (which is s-l-o-w), or grabbing subdirectories of varying granularity and tarring them up to hide the large directories (at a cost of incrementality). I'm also able to push some of the problem projects to github and then exclude them from rclone.

ncw · July 25, 2023, 1:42am

That would be possible with an overlay backend. It would be a bit of work to write one though.

This isn't possible, the API doesn't allow it. (This is what I call paged listings)

I can't remember if there is a search in WebDAV. If there is then you could search for files starting with A then B etc...

pcf000 · July 28, 2023, 5:02pm

I may make a run at adapting the chunker code into a directory-chunking overlay. I have some concepts in mind, but need to learn enough Go and rclone-implementation to tell if they're feasible. It does seem to be something I can play with here in my corner without disturbing anyone else.

ncw · July 28, 2023, 11:22pm

It is certainly feasible

I'd probably start from the combine backend which does something similar and is much simpler than chunker.

system · August 27, 2023, 11:23pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.