ShareFile Limited to 2GB?

Help and Support
1

What is the problem you are having with rclone?

When attempting to upload/copy to Citrix ShareFile using rclone, I am unable to upload anything larger than 2GB. To confirm, I have advanced, which allows for up to 100GB to be uploaded at a time. (Tested same 10GB.dat file with Citrix Desktop App, Citrix Files Migration Tool, and Citrix ShareFile web client all were successful.)
Tested the same rclone command on both Windows and Linux, and am receiving the same error.
Have attempted to reach out to ShareFile, but they keep redirecting me to their API forms.

Run the command 'rclone version' and share the full output of the command.

PS C:\Users\Administrator\Downloads\rclone-v1.61.0-windows-amd64> .\rclone.exe version
rclone v1.61.1
- os/version: Microsoft Windows Server 2022 Standard Evaluation 21H2 (64 bit)
- os/kernel: 10.0.20348.1547 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.19.4
- go/linking: static
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

ShareFile

The command you were trying to run (eg rclone copy /tmp remote:tmp)

\rclone.exe copy "C:\TEST\10Gb.dat" "SF:" -P -vv

The rclone config contents with secrets removed.

[SF]
type = sharefile
endpoint = https://XXX.sharefile.com
token = XXXXX

A log from the command with the -vv flag

PS C:\Users\Administrator\Downloads\rclone-v1.61.0-windows-amd64> .\rclone.exe copy "C:\TEST\10Gb.dat" "SF:" -P -vv
2023/02/27 19:51:33 DEBUG : rclone: Version "v1.61.1" starting with parameters ["C:\\Users\\Administrator\\Downloads\\rclone-v1.61.0-windows-amd64\\rclone.exe" "copy" "C:\\TEST\\10Gb.dat" "SF:" "-P" "-vv"]
2023/02/27 19:51:33 DEBUG : Creating backend with remote "C:\\TEST\\10Gb.dat"
2023/02/27 19:51:33 DEBUG : Using config file from "C:\\Users\\Administrator\\AppData\\Roaming\\rclone\\rclone.conf"
2023/02/27 19:51:33 DEBUG : fs cache: adding new entry for parent of "C:\\TEST\\10Gb.dat", "//?/C:/TEST"
2023/02/27 19:51:33 DEBUG : Creating backend with remote "SF:"
2023-02-27 19:51:33 DEBUG : 10Gb.dat: Need to transfer - File not found at Destination
2023-02-27 19:51:34 ERROR : 10Gb.dat: Failed to copy: upload failed: System error occurred (-2)
2023-02-27 19:51:34 ERROR : Attempt 1/3 failed with 1 errors and: upload failed: System error occurred (-2)
2023-02-27 19:51:34 DEBUG : 10Gb.dat: Need to transfer - File not found at Destination
2023-02-27 19:51:34 ERROR : 10Gb.dat: Failed to copy: upload failed: System error occurred (-2)
2023-02-27 19:51:34 ERROR : Attempt 2/3 failed with 1 errors and: upload failed: System error occurred (-2)
2023-02-27 19:51:34 DEBUG : 10Gb.dat: Need to transfer - File not found at Destination
2023-02-27 19:51:34 ERROR : 10Gb.dat: Failed to copy: upload failed: System error occurred (-2)
2023-02-27 19:51:34 ERROR : Attempt 3/3 failed with 1 errors and: upload failed: System error occurred (-2)
Transferred:       97.801 MiB / 97.801 MiB, 100%, 65.147 MiB/s, ETA 0s
Errors:                 1 (retrying may help)
Elapsed time:         1.3s
2023/02/27 19:51:34 INFO  :
Transferred:       97.801 MiB / 97.801 MiB, 100%, 65.147 MiB/s, ETA 0s
Errors:                 1 (retrying may help)
Elapsed time:         1.3s

2023/02/27 19:51:34 DEBUG : 5 go routines active
2023/02/27 19:51:34 Failed to copy: upload failed: System error occurred (-2)
2

Can you do a run with -vv --dump responses --retries 1 and post the result here please? Check the output doesn't have anything sensitive in it.

3

Of course!

PS C:\Users\Administrator\Downloads\rclone-v1.61.0-windows-amd64> .\rclone.exe copy "C:\TEST\10Gb.dat" "SF:" -P -vv --dump responses --retries 1
2023/02/28 07:48:19 DEBUG : rclone: Version "v1.61.1" starting with parameters ["C:\\Users\\Administrator\\Downloads\\rclone-v1.61.0-windows-amd64\\rclone.exe" "copy" "C:\\TEST\\10Gb.dat" "SF:" "-P" "-vv" "--dump" "responses" "--retries" "1"]
2023/02/28 07:48:19 DEBUG : Creating backend with remote "C:\\TEST\\10Gb.dat"
2023/02/28 07:48:19 DEBUG : Using config file from "C:\\Users\\Administrator\\AppData\\Roaming\\rclone\\rclone.conf"
2023/02/28 07:48:19 DEBUG : fs cache: adding new entry for parent of "C:\\TEST\\10Gb.dat", "//?/C:/TEST"
2023/02/28 07:48:19 DEBUG : Creating backend with remote "SF:"
2023/02/28 07:48:19 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2023/02/28 07:48:19 DEBUG : sharefile root '': Token expired but no uploads in progress - doing nothing
2023/02/28 07:48:19 DEBUG : SF: Loaded invalid token from config file - ignoring
2023/02/28 07:48:19 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/28 07:48:19 DEBUG : HTTP REQUEST (req 0xc000918100)
2023/02/28 07:48:19 DEBUG : POST /oauth/token HTTP/1.1
Host: xxx.sharefile.com
User-Agent: rclone/v1.61.1
Content-Length: 117
Authorization: XXXX
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: gzip

2023/02/28 07:48:19 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/28 07:48:19 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/28 07:48:19 DEBUG : HTTP RESPONSE (req 0xc000918100)
2023/02/28 07:48:19 DEBUG : HTTP/2.0 200 OK
Content-Length: 427
Cache-Control: no-store, no-cache
Citrix-Transactionid: fd393fd7-a5dc-45c3-8332-b425f8e23222
Content-Type: application/json;charset=UTF-8
Correlationid: zor3CDCKuUiKAEFq3VdLxA
Date: Tue, 28 Feb 2023 13:48:19 GMT
Expires: Mon, 27 Feb 2023 13:48:19 GMT
Set-Cookie: AWSALBTG=jxcIKmL1LytYBJAWHmLGVJONcxbPvZA/wDroVif1mm0MGUXSDrU9O79+GS0ngB5tsMQiJ6tvKN3csrpQ9DgjvnS5vcx4ybnqAvcf2JVA28t30FKkvCrP8/XKKRZ04YvSjiyFxoHLKy4R14LfxhU4CRjCdCVjnYB56XvpLAIbB1A1; Expires=Tue, 07 Mar 2023 13:48:19 GMT; Path=/
Set-Cookie: AWSALBTGCORS=jxcIKmL1LytYBJAWHmLGVJONcxbPvZA/wDroVif1mm0MGUXSDrU9O79+GS0ngB5tsMQiJ6tvKN3csrpQ9DgjvnS5vcx4ybnqAvcf2JVA28t30FKkvCrP8/XKKRZ04YvSjiyFxoHLKy4R14LfxhU4CRjCdCVjnYB56XvpLAIbB1A1; Expires=Tue, 07 Mar 2023 13:48:19 GMT; Path=/; SameSite=None; Secure
Set-Cookie: AWSALB=uDc8RjI+CMpE+msyp6FnS6eQ+DCKxBbWi/kib8LF4OYpSteA1qnWMAnclpzHk20FPsMOO4ci6F2Du8BTLlnQRvdnD80WFDrOT2Hlk3xaoI1bBbMG2QJPtGyg3cP1; Expires=Tue, 07 Mar 2023 13:48:19 GMT; Path=/
Set-Cookie: AWSALBCORS=uDc8RjI+CMpE+msyp6FnS6eQ+DCKxBbWi/kib8LF4OYpSteA1qnWMAnclpzHk20FPsMOO4ci6F2Du8BTLlnQRvdnD80WFDrOT2Hlk3xaoI1bBbMG2QJPtGyg3cP1; Expires=Tue, 07 Mar 2023 13:48:19 GMT; Path=/; SameSite=None; Secure
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Robots-Tag: noindex
X-Sfapi-Requestid: UZkyPB1JLEeizhOLzpoNXQ
X-Xss-Protection: 1; mode=block

{"access_token":"XXX"appcp":"sharefile.com","apicp":"sf-api.com","subdomain":"turn","access_files_folders":true,"modify_files_folders":true,"admin_users":true,"admin_accounts":true,"change_my_settings":true,"web_app_login":true}
2023/02/28 07:48:19 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/28 07:48:19 DEBUG : Saving config "token" in section "SF" of the config file
2023/02/28 07:48:19 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2023/02/28 07:48:19 DEBUG : SF: Saved new token in config file
2023/02/28 07:48:19 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/28 07:48:19 DEBUG : HTTP REQUEST (req 0xc000166100)
2023/02/28 07:48:19 DEBUG : GET /sf/v3/Items?%24select=odata.count%2CFileCount%2CName%2CFileName%2CCreationDate%2CIsHidden%2CFileSizeBytes%2Codata.type%2CId%2CHash%2CClientModifiedDate HTTP/1.1
Host: xxx.sharefile.com
User-Agent: rclone/v1.61.1
Authorization: XXXX
Accept-Encoding: gzip

2023/02/28 07:48:19 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/02/28 07:48:20 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/02/28 07:48:20 DEBUG : HTTP RESPONSE (req 0xc000166100)
2023/02/28 07:48:20 DEBUG : HTTP/2.0 200 OK
Content-Length: 353
Cache-Control: no-store, no-cache
Citrix-Transactionid: afc30d2d-c675-4b3a-a718-c7ed477f6525
Content-Type: application/json; charset=utf-8
Correlationid: aHc3kCEDw0K2pSot8ivSww
Date: Tue, 28 Feb 2023 13:48:20 GMT
Expires: Mon, 27 Feb 2023 13:48:19 GMT
Set-Cookie: AWSALBTG=Gbw5WJlrvB0pEHb1yISx7XRjB9kWdfGnX/wjLqJCm03L8bdMNcG9pI8UUV0L34YBO+8h8TQRZSV+bgMy8DmX7CR0wjlyLV/ymKffkdxgGB8BCya61pd+VXmKRTKT+A27wkRshkFAt737QIO35tEBmenT1gSiVABBxPuYBN9UQtd5; Expires=Tue, 07 Mar 2023 13:48:19 GMT; Path=/
Set-Cookie: AWSALBTGCORS=Gbw5WJlrvB0pEHb1yISx7XRjB9kWdfGnX/wjLqJCm03L8bdMNcG9pI8UUV0L34YBO+8h8TQRZSV+bgMy8DmX7CR0wjlyLV/ymKffkdxgGB8BCya61pd+VXmKRTKT+A27wkRshkFAt737QIO35tEBmenT1gSiVABBxPuYBN9UQtd5; Expires=Tue, 07 Mar 2023 13:48:19 GMT; Path=/; SameSite=None; Secure
Set-Cookie: AWSALB=2EcHEpSJjJ6S675C2u9uHo54QMFo+rZZOqaElC3i9prcE3A9Ex1hcIcHenOmtHWpERRW53Ds7lX2Wl64XuyExcv+ZH2Pwq17hNV3AoT+AWLveMBnVkONKTdE+xLb; Expires=Tue, 07 Mar 2023 13:48:19 GMT; Path=/
Set-Cookie: AWSALBCORS=2EcHEpSJjJ6S675C2u9uHo54QMFo+rZZOqaElC3i9prcE3A9Ex1hcIcHenOmtHWpERRW53Ds7lX2Wl64XuyExcv+ZH2Pwq17hNV3AoT+AWLveMBnVkONKTdE+xLb; Expires=Tue, 07 Mar 2023 13:48:19 GMT; Path=/; SameSite=None; Secure
Set-Cookie: SFAPI_AuthID=d09f1cff-3bdf-41e8-9ea4-4f83a705a431; domain=turn.sharefile.com; path=/; secure; httponly
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Robots-Tag: noindex
X-Sfapi-Accountid: a12fa41d-cab1-bc52-1a36-fc980a523c3b
X-Sfapi-Appcode: ThirdParty
X-Sfapi-Authenticateduserid: ?
X-Sfapi-Oauthclientid: ?
X-Sfapi-Requestid: _iKhGlgf-0eAR8fvwvM50Q
X-Sfapi-Userisimpersonated: False
X-Xss-Protection: 1; mode=block

{"FileCount":110630,"Name":"Personal Folders","FileName":"Personal Folders","CreationDate":"2023-02-06T20:37:08Z","IsHidden":false,"FileSizeBytes":38895074814770,"odata.metadata":"https://xxx.sharefile.com/sf/v3/$metadata#Items/ShareFile.Api.Models.Folder@Element","odata.type":"ShareFile.Api.Models.Folder","Id":"fohacd85-ee7d-4488-ac18-90d0d9a05636"}
2023/02/28 07:48:20 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-02-28 07:48:20 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-02-28 07:48:20 DEBUG : HTTP REQUEST (req 0xc000166200)
2023-02-28 07:48:20 DEBUG : GET /sf/v3/Items(fohacd85-ee7d-4488-ac18-90d0d9a05636)/ByPath?%24select=odata.count%2CFileCount%2CName%2CFileName%2CCreationDate%2CIsHidden%2CFileSizeBytes%2Codata.type%2CId%2CHash%2CClientModifiedDate&path=%2F10Gb.dat HTTP/1.1
Host: xxx.sharefile.com
User-Agent: rclone/v1.61.1
Authorization: XXXX
Accept-Encoding: gzip
2023-02-28 07:48:20 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-02-28 07:48:20 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-02-28 07:48:20 DEBUG : HTTP RESPONSE (req 0xc000166200)
2023-02-28 07:48:20 DEBUG : HTTP/2.0 404 Not Found
Content-Length: 117
Cache-Control: no-store, no-cache
Citrix-Transactionid: 599b4ad9-80fb-4c21-8845-6b6c4908d1db
Content-Type: application/json; charset=utf-8
Correlationid: 0o3uelTxQ02Dde2ZkL6eEA
Date: Tue, 28 Feb 2023 13:48:20 GMT
Expires: Mon, 27 Feb 2023 13:48:20 GMT
Set-Cookie: AWSALBTG=ChlVo6sxhizY+fCAFty4IvTRjFuQ+NEmNDKeiG4K4P/8pBLW/qZAxLWzSfq6kIE7mRD73P8BI1Kobuq8BuHxc7K2Ec+Jl+PvE6pCchaWT+rTz5uSsOIBvP6RJP7WEMBYO0xPHEx6iV4NAz+XzHhqEPYLJy/8FgBViiWTTXIX0nLP; Expires=Tue, 07 Mar 2023 13:48:20 GMT; Path=/
Set-Cookie: AWSALBTGCORS=ChlVo6sxhizY+fCAFty4IvTRjFuQ+NEmNDKeiG4K4P/8pBLW/qZAxLWzSfq6kIE7mRD73P8BI1Kobuq8BuHxc7K2Ec+Jl+PvE6pCchaWT+rTz5uSsOIBvP6RJP7WEMBYO0xPHEx6iV4NAz+XzHhqEPYLJy/8FgBViiWTTXIX0nLP; Expires=Tue, 07 Mar 2023 13:48:20 GMT; Path=/; SameSite=None; Secure
Set-Cookie: AWSALB=ND1TY+9UAKyYZPPa1zv+t3mS0hjC4UbGla3TugQ02KGzmKrIPkBhfF1ahlsk49WK8Yv8YnXQJTcv9CUTDzm/lndsPtphAs265wA9O0ExWchd49Eh9+PZvcol4QUO; Expires=Tue, 07 Mar 2023 13:48:20 GMT; Path=/
Set-Cookie: AWSALBCORS=ND1TY+9UAKyYZPPa1zv+t3mS0hjC4UbGla3TugQ02KGzmKrIPkBhfF1ahlsk49WK8Yv8YnXQJTcv9CUTDzm/lndsPtphAs265wA9O0ExWchd49Eh9+PZvcol4QUO; Expires=Tue, 07 Mar 2023 13:48:20 GMT; Path=/; SameSite=None; Secure
Set-Cookie: SFAPI_AuthID=d09f1cff-3bdf-41e8-9ea4-4f83a705a431; domain=turn.sharefile.com; path=/; secure; httponly
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Robots-Tag: noindex
X-Sfapi-Accountid: a12fa41d-cab1-bc52-1a36-fc980a523c3b
X-Sfapi-Appcode: ThirdParty
X-Sfapi-Authenticateduserid: efa0984d-9663-4af0-ac54-65e2485ba672
X-Sfapi-Oauthclientid: ?
X-Sfapi-Requestid: sPOUVFS130GlaUznl0UQsg
X-Sfapi-Userisimpersonated: False
X-Xss-Protection: 1; mode=block

{"code":"NotFound","message":{"lang":"en-US","value":"Item: Path element '10Gb.dat' not found."},"reason":"NotFound"}
2023-02-28 07:48:20 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-02-28 07:48:20 DEBUG : 10Gb.dat: Need to transfer - File not found at Destination
2023-02-28 07:48:20 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-02-28 07:48:20 DEBUG : HTTP REQUEST (req 0xc000166600)
2023-02-28 07:48:20 DEBUG : GET /sf/v3/Items(fohacd85-ee7d-4488-ac18-90d0d9a05636)/ByPath?%24select=odata.count%2CFileCount%2CName%2CFileName%2CCreationDate%2CIsHidden%2CFileSizeBytes%2Codata.type%2CId%2CHash%2CClientModifiedDate&path=%2F10Gb.dat HTTP/1.1
Host: xxx.sharefile.com
User-Agent: rclone/v1.61.1
Authorization: XXXX
Accept-Encoding: gzip
2023-02-28 07:48:20 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-02-28 07:48:20 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-02-28 07:48:20 DEBUG : HTTP RESPONSE (req 0xc000166600)
2023-02-28 07:48:20 DEBUG : HTTP/2.0 404 Not Found
Content-Length: 117
Cache-Control: no-store, no-cache
Citrix-Transactionid: 646ed541-8c5b-4cd8-b378-99d783ebe29e
Content-Type: application/json; charset=utf-8
Correlationid: p2JC--caaUuoSb8EygNXBg
Date: Tue, 28 Feb 2023 13:48:20 GMT
Expires: Mon, 27 Feb 2023 13:48:20 GMT
Set-Cookie: AWSALBTG=wU4tKlXvRS4j8OF4dNFlemovpcvougD4bB6gKDjw4jaKs3WKTEGFTiAwvPeQGbUJknnbSO2UXVKafz9K8CcIXxqteCesLUqUfkA7ZcYJnN8M3SKLJY7Z3sjdor6z3SM/ZPJBZMKpPbrtOpJQXcQzSN/qZYjRlS4PSNmS4evVDn4u; Expires=Tue, 07 Mar 2023 13:48:20 GMT; Path=/
Set-Cookie: AWSALBTGCORS=wU4tKlXvRS4j8OF4dNFlemovpcvougD4bB6gKDjw4jaKs3WKTEGFTiAwvPeQGbUJknnbSO2UXVKafz9K8CcIXxqteCesLUqUfkA7ZcYJnN8M3SKLJY7Z3sjdor6z3SM/ZPJBZMKpPbrtOpJQXcQzSN/qZYjRlS4PSNmS4evVDn4u; Expires=Tue, 07 Mar 2023 13:48:20 GMT; Path=/; SameSite=None; Secure
Set-Cookie: AWSALB=2tB4MDdauTOXIWdwyMPyAdyROjswvt0yRI6LdXUbhj/ioqccQTYCDXl9l7ETCYo8rqq8knae47ojfa5ok/gOvTZHmdmNEDs2VzAiS5YhHOv3WIhAYC7UwaFIZ58K; Expires=Tue, 07 Mar 2023 13:48:20 GMT; Path=/
Set-Cookie: AWSALBCORS=2tB4MDdauTOXIWdwyMPyAdyROjswvt0yRI6LdXUbhj/ioqccQTYCDXl9l7ETCYo8rqq8knae47ojfa5ok/gOvTZHmdmNEDs2VzAiS5YhHOv3WIhAYC7UwaFIZ58K; Expires=Tue, 07 Mar 2023 13:48:20 GMT; Path=/; SameSite=None; Secure
Set-Cookie: SFAPI_AuthID=d09f1cff-3bdf-41e8-9ea4-4f83a705a431; domain=turn.sharefile.com; path=/; secure; httponly
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Robots-Tag: noindex
X-Sfapi-Accountid: a12fa41d-cab1-bc52-1a36-fc980a523c3b
X-Sfapi-Appcode: ThirdParty
X-Sfapi-Authenticateduserid: efa0984d-9663-4af0-ac54-65e2485ba672
X-Sfapi-Oauthclientid: ?
X-Sfapi-Requestid: kwKzk_CDz0C52Yi5PElYUA
X-Sfapi-Userisimpersonated: False
X-Xss-Protection: 1; mode=block

{"code":"NotFound","message":{"lang":"en-US","value":"Item: Path element '10Gb.dat' not found."},"reason":"NotFound"}
2023-02-28 07:48:20 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-02-28 07:48:20 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-02-28 07:48:20 DEBUG : HTTP REQUEST (req 0xc000166100)
2023-02-28 07:48:20 DEBUG : POST /sf/v3/Items(fohacd85-ee7d-4488-ac18-90d0d9a05636)/Upload2 HTTP/1.1
Host: xxx.sharefile.com
User-Agent: rclone/v1.61.1
Content-Length: 208
Authorization: XXXX
Content-Type: application/json
Accept-Encoding: gzip
2023-02-28 07:48:20 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-02-28 07:48:20 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-02-28 07:48:21 DEBUG : HTTP RESPONSE (req 0xc000166100)
2023-02-28 07:48:21 DEBUG : HTTP/2.0 200 OK
Content-Length: 603
Cache-Control: no-store, no-cache
Citrix-Transactionid: 9c6d0d6c-b0d1-4f60-a6d0-729dd5951997
Content-Type: application/json; charset=utf-8
Correlationid: mxtYlrIlpUCXGvHGB25fFg
Date: Tue, 28 Feb 2023 13:48:20 GMT
Expires: Mon, 27 Feb 2023 13:48:20 GMT
Set-Cookie: AWSALBTG=Cusn9JOmJg65rTMhQ0h7LUq3/5AZd0t7VoJxPk879gIzqqvSUZ4zqkFNdpnLKHgBCPvwV3u7ajef8YfPTUN/tLz1VNpa2kxMR0AwArkaanW7QMBcKfzYlkiigEOafUGzflN2kU2+jeAP/k6qhbAPDl3L/l6/1gyY6Z8tHZzuPO0O; Expires=Tue, 07 Mar 2023 13:48:20 GMT; Path=/
Set-Cookie: AWSALBTGCORS=Cusn9JOmJg65rTMhQ0h7LUq3/5AZd0t7VoJxPk879gIzqqvSUZ4zqkFNdpnLKHgBCPvwV3u7ajef8YfPTUN/tLz1VNpa2kxMR0AwArkaanW7QMBcKfzYlkiigEOafUGzflN2kU2+jeAP/k6qhbAPDl3L/l6/1gyY6Z8tHZzuPO0O; Expires=Tue, 07 Mar 2023 13:48:20 GMT; Path=/; SameSite=None; Secure
Set-Cookie: AWSALB=qiGIsL4+IM4m8qAF2TGz55MBfUGHcqiSw+pqMDW5NVWkktqbVarPUQokye0eRDw2ac2wxQNXeM24sN6qRHUO9rNsNRb8hAn1HOXSRs6f/S/pqJZ0VCVEUvkgGjSU; Expires=Tue, 07 Mar 2023 13:48:20 GMT; Path=/
Set-Cookie: AWSALBCORS=qiGIsL4+IM4m8qAF2TGz55MBfUGHcqiSw+pqMDW5NVWkktqbVarPUQokye0eRDw2ac2wxQNXeM24sN6qRHUO9rNsNRb8hAn1HOXSRs6f/S/pqJZ0VCVEUvkgGjSU; Expires=Tue, 07 Mar 2023 13:48:20 GMT; Path=/; SameSite=None; Secure
Set-Cookie: SFAPI_AuthID=d09f1cff-3bdf-41e8-9ea4-4f83a705a431; domain=turn.sharefile.com; path=/; secure; httponly
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Robots-Tag: noindex
X-Sfapi-Accountid: a12fa41d-cab1-bc52-1a36-fc980a523c3b
X-Sfapi-Appcode: ThirdParty
X-Sfapi-Authenticateduserid: efa0984d-9663-4af0-ac54-65e2485ba672
X-Sfapi-Oauthclientid: ?
X-Sfapi-Requestid: u6s4x4Vp002afedH0FN--g
X-Sfapi-Userisimpersonated: False
X-Xss-Protection: 1; mode=block

{"Method":"Standard","ChunkUri":"https://sf-apiadapter-sharefile-useast.sharefile.com/service/apiadapter/api/standardupload?uploadid=0eb4265f-2f5b-42df-a4d8-ceb0ec07ffd3&accountid=a12fa41d-cab1-bc52-1a36-fc980a523c3b&zoneid=zpfed2b3f5-fbbf-4ed5-9a58-f1bd888f01&zsid=ef&h=cEmdu27c%2F4kfUR5BI6yqJTc40oDbBvJ%2BZd9Br1cUia8%3D","IsResume":false,"ResumeIndex":0,"ResumeOffset":0,"ResumeFileHash":"","MaxNumberOfThreads":4,"odata.metadata":"https://xxx.sharefile.com/sf/v3/$metadata#UploadSpecification/ShareFile.Api.Models.UploadSpecification@Element","odata.type":"ShareFile.Api.Models.UploadSpecification"}
2023-02-28 07:48:21 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-02-28 07:49:20 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-02-28 07:49:20 DEBUG : HTTP REQUEST (req 0xc000294200)
2023-02-28 07:49:20 DEBUG : POST /service/apiadapter/api/standardupload?uploadid=0eb4265f-2f5b-42df-a4d8-ceb0ec07ffd3&accountid=a12fa41d-cab1-bc52-1a36-fc980a523c3b&zoneid=zpfed2b3f5-fbbf-4ed5-9a58-f1bd888f01&zsid=ef&h=cEmdu27c%2F4kfUR5BI6yqJTc40oDbBvJ%2BZd9Br1cUia8%3D&fmt=json HTTP/1.1
Host: sf-apiadapter-sharefile-useast.sharefile.com
User-Agent: rclone/v1.61.1
Content-Length: 10737418240
Authorization: XXXX
Accept-Encoding: gzip
2023-02-28 07:49:20 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-02-28 07:49:20 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-02-28 07:49:20 DEBUG : HTTP RESPONSE (req 0xc000294200)
2023-02-28 07:49:20 DEBUG : HTTP/1.1 200 OK
Content-Length: 68
Citrix-Transactionid: 9668f98c-2b0c-466f-9098-899d2598237b
Connection: keep-alive
Content-Language: en
Content-Type: application/json; charset=utf-8
Date: Tue, 28 Feb 2023 13:49:20 GMT
Requestid: 0HMOJCPDOJ9Q2:00000001
X-Sfapi-Requestid: 0HMOJCPDOJ9Q2:00000001

{"errorMessage":"System error occurred","errorCode":-2,"error":true}
2023-02-28 07:49:20 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-02-28 07:49:20 ERROR : 10Gb.dat: Failed to copy: upload failed: System error occurred (-2)
2023-02-28 07:49:20 ERROR : Attempt 1/1 failed with 1 errors and: upload failed: System error occurred (-2)
Transferred:       60.027 MiB / 60.027 MiB, 100%, 3.586 MiB/s, ETA 0s
Errors:                 1 (retrying may help)
Elapsed time:       1m0.7s
2023/02/28 07:49:20 INFO  :
Transferred:       60.027 MiB / 60.027 MiB, 100%, 3.586 MiB/s, ETA 0s
Errors:                 1 (retrying may help)
Elapsed time:       1m0.7s

2023/02/28 07:49:20 DEBUG : 5 go routines active
2023/02/28 07:49:20 Failed to copy: upload failed: System error occurred (-2)
4

That seemed to be going OK ... Rclone gets the upload URL and posts the 10G file to it

2023-02-28 07:49:20 DEBUG : HTTP REQUEST (req 0xc000294200)
2023-02-28 07:49:20 DEBUG : POST /service/apiadapter/api/standardupload?uploadid=0eb4265f-2f5b-42df-a4d8-ceb0ec07ffd3&accountid=a12fa41d-cab1-bc52-1a36-fc980a523c3b&zoneid=zpfed2b3f5-fbbf-4ed5-9a58-f1bd888f01&zsid=ef&h=cEmdu27c%2F4kfUR5BI6yqJTc40oDbBvJ%2BZd9Br1cUia8%3D&fmt=json HTTP/1.1
Host: sf-apiadapter-sharefile-useast.sharefile.com
User-Agent: rclone/v1.61.1
Content-Length: 10737418240
Authorization: XXXX
Accept-Encoding: gzip

However almost immediately it gets an error back

2023-02-28 07:49:20 DEBUG : HTTP RESPONSE (req 0xc000294200)
2023-02-28 07:49:20 DEBUG : HTTP/1.1 200 OK
Content-Length: 68
Citrix-Transactionid: 9668f98c-2b0c-466f-9098-899d2598237b
Connection: keep-alive
Content-Language: en
Content-Type: application/json; charset=utf-8
Date: Tue, 28 Feb 2023 13:49:20 GMT
Requestid: 0HMOJCPDOJ9Q2:00000001
X-Sfapi-Requestid: 0HMOJCPDOJ9Q2:00000001

{"errorMessage":"System error occurred","errorCode":-2,"error":true}

I don't understand why rclone wasn't trying to do a chunked upload though as I suspect 10G is too big for a single part.

You don't appear to have set --sharefile-upload-cutoff so it should be uploading in 64M chunks like this

$ rclone copy /tmp/10G TestSharefile: -vv
2023/02/28 16:54:20 DEBUG : rclone: Version "v1.62.0-beta.6745.9e1443799" starting with parameters ["rclone" "copy" "/tmp/10G" "TestSharefile:" "-vv"]
2023/02/28 16:54:20 DEBUG : Creating backend with remote "/tmp/10G"
2023/02/28 16:54:20 DEBUG : Using config file from "/home/ncw/.rclone.conf"
2023/02/28 16:54:20 DEBUG : fs cache: adding new entry for parent of "/tmp/10G", "/tmp"
2023/02/28 16:54:20 DEBUG : Creating backend with remote "TestSharefile:"
2023/02/28 16:54:20 DEBUG : 10G: Need to transfer - File not found at Destination
2023/02/28 16:54:21 DEBUG : 10G: Starting upload of large file in 160 chunks
2023/02/28 16:54:21 DEBUG : 10G: Sending chunk 0 length 67108864
2023/02/28 16:54:22 DEBUG : 10G: Sending chunk 1 length 67108864
2023/02/28 16:54:22 DEBUG : 10G: Sending chunk 2 length 67108864
2023/02/28 16:54:23 DEBUG : 10G: Sending chunk 3 length 67108864
...

Are you sure you don't have upload_cutoff in your config file? If you do, try removing it and trying again.

5

I apologize, I did add that option as part of troubleshooting. I have removed it. Still failing, however, updated end of paste is below. (I canceled out after the 6th retry)

{"value":"OK","error":false}
2023-02-28 11:04:30 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-02-28 11:04:30 DEBUG : 10Gb.dat: Done sending chunk 154
2023-02-28 11:04:31 DEBUG : 10Gb.dat: Sending chunk 158 length 67108864
2023-02-28 11:04:31 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-02-28 11:04:31 DEBUG : HTTP RESPONSE (req 0xc000a18f00)
2023-02-28 11:04:31 DEBUG : HTTP/1.1 200 OK
Content-Length: 28
Citrix-Transactionid: c1c5de35-a77f-460a-a8df-abb476caedea
Connection: keep-alive
Content-Language: en
Content-Type: application/json; charset=utf-8
Date: Tue, 28 Feb 2023 17:04:31 GMT
Requestid: 0HMOK4F3E1MFS:00000002
X-Sfapi-Requestid: 0HMOK4F3E1MFS:00000002

{"value":"OK","error":false}
2023-02-28 11:04:31 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-02-28 11:04:31 DEBUG : 10Gb.dat: Done sending chunk 156
2023-02-28 11:04:31 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-02-28 11:04:31 DEBUG : HTTP REQUEST (req 0xc000a19200)
2023-02-28 11:04:31 DEBUG : POST /service/apiadapter/api/threadedupload?uploadid=b008a43a-e81f-4618-bc86-ace7cbbdb6fd&accountid=a12fa41d-cab1-bc52-1a36-fc980a523c3b&zoneid=zpfed2b3f5-fbbf-4ed5-9a58-f1bd888f01&zsid=ef&h=bCGWGDK5P5C0ypvKUetj4PPUmTBOBMr5VAHYPYBvV1g%3D&fmt=json&index=158&byteOffset=10603200512&hash=e269faca3dd13e240166313bdff36305&fmt=json HTTP/1.1
Host: sf-apiadapter-sharefile-useast.sharefile.com
User-Agent: rclone/v1.61.1
Content-Length: 67108864
Authorization: XXXX
Accept-Encoding: gzip
2023-02-28 11:04:31 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-02-28 11:04:31 DEBUG : 10Gb.dat: Sending chunk 159 length 67108864
2023-02-28 11:04:31 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-02-28 11:04:31 DEBUG : HTTP RESPONSE (req 0xc000b86200)
2023-02-28 11:04:31 DEBUG : HTTP/1.1 200 OK
Content-Length: 28
Citrix-Transactionid: 0ab3e890-dfe0-412b-ac6f-64825ac81ddb
Connection: keep-alive
Content-Language: en
Content-Type: application/json; charset=utf-8
Date: Tue, 28 Feb 2023 17:04:31 GMT
Requestid: 0HMOMEV2ON8TG:00000005
X-Sfapi-Requestid: 0HMOMEV2ON8TG:00000005

{"value":"OK","error":false}
2023-02-28 11:04:31 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-02-28 11:04:31 DEBUG : 10Gb.dat: Done sending chunk 157
2023-02-28 11:04:31 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-02-28 11:04:31 DEBUG : HTTP REQUEST (req 0xc00085e400)
2023-02-28 11:04:31 DEBUG : POST /service/apiadapter/api/threadedupload?uploadid=b008a43a-e81f-4618-bc86-ace7cbbdb6fd&accountid=a12fa41d-cab1-bc52-1a36-fc980a523c3b&zoneid=zpfed2b3f5-fbbf-4ed5-9a58-f1bd888f01&zsid=ef&h=bCGWGDK5P5C0ypvKUetj4PPUmTBOBMr5VAHYPYBvV1g%3D&fmt=json&index=159&byteOffset=10670309376&hash=732c51b93974b10c7a9c205da89a0dc3&fmt=json&finish=true&fileSize=10737418240&fileHash=8e806ab1423efecab868964641b8c937 HTTP/1.1
Host: sf-apiadapter-sharefile-useast.sharefile.com
User-Agent: rclone/v1.61.1
Content-Length: 67108864
Authorization: XXXX
Accept-Encoding: gzip
2023-02-28 11:04:31 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-02-28 11:04:32 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-02-28 11:04:32 DEBUG : HTTP RESPONSE (req 0xc000a19200)
2023-02-28 11:04:32 DEBUG : HTTP/1.1 200 OK
Content-Length: 28
Citrix-Transactionid: 0578fb38-a76c-4659-9df8-921b8645aedd
Connection: keep-alive
Content-Language: en
Content-Type: application/json; charset=utf-8
Date: Tue, 28 Feb 2023 17:04:32 GMT
Requestid: 0HMOK4VAH6BDP:00000002
X-Sfapi-Requestid: 0HMOK4VAH6BDP:00000002

{"value":"OK","error":false}
2023-02-28 11:04:32 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-02-28 11:04:32 DEBUG : 10Gb.dat: Done sending chunk 158
2023-02-28 11:04:33 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-02-28 11:04:33 DEBUG : HTTP RESPONSE (req 0xc000b00400)
2023-02-28 11:04:33 DEBUG : HTTP/1.1 504 Gateway Time-out
Content-Length: 132
Connection: keep-alive
Content-Type: text/html
Date: Tue, 28 Feb 2023 17:04:33 GMT
Server: awselb/2.0

<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
</body>
</html>
2023-02-28 11:04:33 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-02-28 11:04:33 DEBUG : 10Gb.dat: Error sending chunk 21: <html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
</body>
</html>
: 504: 504 Gateway Time-out
2023-02-28 11:04:33 DEBUG : pacer: low level retry 1/10 (error <html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
</body>
</html>
: 504: 504 Gateway Time-out)
2023-02-28 11:04:33 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2023-02-28 11:04:33 DEBUG : 10Gb.dat: Sending chunk 21 length 67108864
2023-02-28 11:04:33 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-02-28 11:04:33 DEBUG : HTTP REQUEST (req 0xc000b00b00)
2023-02-28 11:04:33 DEBUG : POST /service/apiadapter/api/threadedupload?uploadid=b008a43a-e81f-4618-bc86-ace7cbbdb6fd&accountid=a12fa41d-cab1-bc52-1a36-fc980a523c3b&zoneid=zpfed2b3f5-fbbf-4ed5-9a58-f1bd888f01&zsid=ef&h=bCGWGDK5P5C0ypvKUetj4PPUmTBOBMr5VAHYPYBvV1g%3D&fmt=json&index=21&byteOffset=1409286144&hash=cefead245e629452ae839b4868437844&fmt=json HTTP/1.1
Host: sf-apiadapter-sharefile-useast.sharefile.com
User-Agent: rclone/v1.61.1
Content-Length: 67108864
Authorization: XXXX
Accept-Encoding: gzip
2023-02-28 11:04:33 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-02-28 11:04:33 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-02-28 11:04:33 DEBUG : HTTP RESPONSE (req 0xc00085e400)
2023-02-28 11:04:33 DEBUG : HTTP/1.1 200 OK
Content-Length: 28
Citrix-Transactionid: 17de37bd-463d-49d8-8a30-bc49ecf82797
Connection: keep-alive
Content-Language: en
Content-Type: application/json; charset=utf-8
Date: Tue, 28 Feb 2023 17:04:33 GMT
Requestid: 0HMOMFD385AI4:00000005
X-Sfapi-Requestid: 0HMOMFD385AI4:00000005

{"value":"OK","error":false}
2023-02-28 11:04:33 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-02-28 11:04:33 DEBUG : pacer: Reducing sleep to 15ms
2023-02-28 11:04:33 DEBUG : 10Gb.dat: Done sending chunk 159
2023-02-28 11:04:33 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-02-28 11:04:33 DEBUG : HTTP RESPONSE (req 0xc000b00b00)
2023-02-28 11:04:33 DEBUG : HTTP/1.1 200 OK
Content-Length: 68
Citrix-Transactionid: 35859266-cfcc-4951-9273-1478e76ebdb4
Connection: keep-alive
Content-Language: en
Content-Type: application/json; charset=utf-8
Date: Tue, 28 Feb 2023 17:04:33 GMT
Requestid: 0HMOJB2LPD9N7:00000013
X-Sfapi-Requestid: 0HMOJB2LPD9N7:00000013

{"errorMessage":"System error occurred","errorCode":-2,"error":true}
2023-02-28 11:04:33 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-02-28 11:04:33 DEBUG : pacer: Reducing sleep to 11.25ms
2023-02-28 11:04:33 DEBUG : 10Gb.dat: Done sending chunk 21
2023-02-28 11:04:33 DEBUG : 10Gb.dat: Finishing large file upload
2023-02-28 11:04:33 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-02-28 11:04:33 DEBUG : HTTP REQUEST (req 0xc00085e300)
2023-02-28 11:04:33 DEBUG : POST /service/apiadapter/api/threadedupload/finish?uploadid=b008a43a-e81f-4618-bc86-ace7cbbdb6fd&accountid=a12fa41d-cab1-bc52-1a36-fc980a523c3b&zoneid=zpfed2b3f5-fbbf-4ed5-9a58-f1bd888f01&zsid=ef&h=ksy%2BFK2fAo5tofHBk6B%2FduWzX4InchMWwuL09%2BY9AqA%3D&fmt=json HTTP/1.1
Host: sf-apiadapter-sharefile-useast.sharefile.com
User-Agent: rclone/v1.61.1
Content-Length: 0
Authorization: XXXX
Accept-Encoding: gzip
2023-02-28 11:04:33 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-02-28 11:06:33 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-02-28 11:06:33 DEBUG : HTTP RESPONSE (req 0xc00085e300)
2023-02-28 11:06:33 DEBUG : HTTP/1.1 504 Gateway Time-out
Content-Length: 132
Connection: keep-alive
Content-Type: text/html
Date: Tue, 28 Feb 2023 17:06:33 GMT
Server: awselb/2.0

<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
</body>
</html>
2023-02-28 11:06:33 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-02-28 11:06:33 DEBUG : pacer: low level retry 1/10 (error <html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
</body>
</html>
: 504: 504 Gateway Time-out)
2023-02-28 11:06:33 DEBUG : pacer: Rate limited, increasing sleep to 22.5ms
2023-02-28 11:06:33 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-02-28 11:06:33 DEBUG : HTTP REQUEST (req 0xc00085e200)
2023-02-28 11:06:33 DEBUG : POST /service/apiadapter/api/threadedupload/finish?uploadid=b008a43a-e81f-4618-bc86-ace7cbbdb6fd&accountid=a12fa41d-cab1-bc52-1a36-fc980a523c3b&zoneid=zpfed2b3f5-fbbf-4ed5-9a58-f1bd888f01&zsid=ef&h=ksy%2BFK2fAo5tofHBk6B%2FduWzX4InchMWwuL09%2BY9AqA%3D&fmt=json HTTP/1.1
Host: sf-apiadapter-sharefile-useast.sharefile.com
User-Agent: rclone/v1.61.1
Content-Length: 0
Authorization: XXXX
Accept-Encoding: gzip
2023-02-28 11:06:33 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-02-28 11:08:33 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-02-28 11:08:33 DEBUG : HTTP RESPONSE (req 0xc00085e200)
2023-02-28 11:08:33 DEBUG : HTTP/1.1 504 Gateway Time-out
Content-Length: 132
Connection: keep-alive
Content-Type: text/html
Date: Tue, 28 Feb 2023 17:08:33 GMT
Server: awselb/2.0

<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
</body>
</html>
2023-02-28 11:08:33 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-02-28 11:08:33 DEBUG : pacer: low level retry 2/10 (error <html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
</body>
</html>
: 504: 504 Gateway Time-out)
2023-02-28 11:08:33 DEBUG : pacer: Rate limited, increasing sleep to 45ms
2023-02-28 11:08:33 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-02-28 11:08:33 DEBUG : HTTP REQUEST (req 0xc00085e300)
2023-02-28 11:08:33 DEBUG : POST /service/apiadapter/api/threadedupload/finish?uploadid=b008a43a-e81f-4618-bc86-ace7cbbdb6fd&accountid=a12fa41d-cab1-bc52-1a36-fc980a523c3b&zoneid=zpfed2b3f5-fbbf-4ed5-9a58-f1bd888f01&zsid=ef&h=ksy%2BFK2fAo5tofHBk6B%2FduWzX4InchMWwuL09%2BY9AqA%3D&fmt=json HTTP/1.1
Host: sf-apiadapter-sharefile-useast.sharefile.com
User-Agent: rclone/v1.61.1
Content-Length: 0
Authorization: XXXX
Accept-Encoding: gzip
2023-02-28 11:08:33 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-02-28 11:10:33 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-02-28 11:10:33 DEBUG : HTTP RESPONSE (req 0xc00085e300)
2023-02-28 11:10:33 DEBUG : HTTP/1.1 504 Gateway Time-out
Content-Length: 132
Connection: keep-alive
Content-Type: text/html
Date: Tue, 28 Feb 2023 17:10:33 GMT
Server: awselb/2.0

<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
</body>
</html>
2023-02-28 11:10:33 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-02-28 11:10:33 DEBUG : pacer: low level retry 3/10 (error <html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
</body>
</html>
: 504: 504 Gateway Time-out)
2023-02-28 11:10:33 DEBUG : pacer: Rate limited, increasing sleep to 90ms
2023-02-28 11:10:33 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-02-28 11:10:33 DEBUG : HTTP REQUEST (req 0xc00085e200)
2023-02-28 11:10:33 DEBUG : POST /service/apiadapter/api/threadedupload/finish?uploadid=b008a43a-e81f-4618-bc86-ace7cbbdb6fd&accountid=a12fa41d-cab1-bc52-1a36-fc980a523c3b&zoneid=zpfed2b3f5-fbbf-4ed5-9a58-f1bd888f01&zsid=ef&h=ksy%2BFK2fAo5tofHBk6B%2FduWzX4InchMWwuL09%2BY9AqA%3D&fmt=json HTTP/1.1
Host: sf-apiadapter-sharefile-useast.sharefile.com
User-Agent: rclone/v1.61.1
Content-Length: 0
Authorization: XXXX
Accept-Encoding: gzip
2023-02-28 11:10:33 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-02-28 11:12:33 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-02-28 11:12:33 DEBUG : HTTP RESPONSE (req 0xc00085e200)
2023-02-28 11:12:33 DEBUG : HTTP/1.1 504 Gateway Time-out
Content-Length: 132
Connection: keep-alive
Content-Type: text/html
Date: Tue, 28 Feb 2023 17:12:33 GMT
Server: awselb/2.0

<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
</body>
</html>
2023-02-28 11:12:33 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-02-28 11:12:33 DEBUG : pacer: low level retry 4/10 (error <html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
</body>
</html>
: 504: 504 Gateway Time-out)
2023-02-28 11:12:33 DEBUG : pacer: Rate limited, increasing sleep to 180ms
2023-02-28 11:12:33 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-02-28 11:12:33 DEBUG : HTTP REQUEST (req 0xc00085e300)
2023-02-28 11:12:33 DEBUG : POST /service/apiadapter/api/threadedupload/finish?uploadid=b008a43a-e81f-4618-bc86-ace7cbbdb6fd&accountid=a12fa41d-cab1-bc52-1a36-fc980a523c3b&zoneid=zpfed2b3f5-fbbf-4ed5-9a58-f1bd888f01&zsid=ef&h=ksy%2BFK2fAo5tofHBk6B%2FduWzX4InchMWwuL09%2BY9AqA%3D&fmt=json HTTP/1.1
Host: sf-apiadapter-sharefile-useast.sharefile.com
User-Agent: rclone/v1.61.1
Content-Length: 0
Authorization: XXXX
Accept-Encoding: gzip
2023-02-28 11:12:33 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-02-28 11:14:33 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-02-28 11:14:33 DEBUG : HTTP RESPONSE (req 0xc00085e300)
2023-02-28 11:14:33 DEBUG : HTTP/1.1 504 Gateway Time-out
Content-Length: 132
Connection: keep-alive
Content-Type: text/html
Date: Tue, 28 Feb 2023 17:14:33 GMT
Server: awselb/2.0

<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
</body>
</html>
2023-02-28 11:14:33 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-02-28 11:14:33 DEBUG : pacer: low level retry 5/10 (error <html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
</body>
</html>
: 504: 504 Gateway Time-out)
2023-02-28 11:14:33 DEBUG : pacer: Rate limited, increasing sleep to 360ms
2023-02-28 11:14:33 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-02-28 11:14:33 DEBUG : HTTP REQUEST (req 0xc000a18d00)
2023-02-28 11:14:33 DEBUG : POST /service/apiadapter/api/threadedupload/finish?uploadid=b008a43a-e81f-4618-bc86-ace7cbbdb6fd&accountid=a12fa41d-cab1-bc52-1a36-fc980a523c3b&zoneid=zpfed2b3f5-fbbf-4ed5-9a58-f1bd888f01&zsid=ef&h=ksy%2BFK2fAo5tofHBk6B%2FduWzX4InchMWwuL09%2BY9AqA%3D&fmt=json HTTP/1.1
Host: sf-apiadapter-sharefile-useast.sharefile.com
User-Agent: rclone/v1.61.1
Content-Length: 0
Authorization: XXXX
Accept-Encoding: gzip
2023-02-28 11:14:33 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-02-28 11:16:33 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-02-28 11:17:42 DEBUG : HTTP RESPONSE (req 0xc000a18d00)
2023-02-28 11:17:42 DEBUG : HTTP/1.1 504 Gateway Time-out
Content-Length: 132
Connection: keep-alive
Content-Type: text/html
Date: Tue, 28 Feb 2023 17:16:33 GMT
Server: awselb/2.0

<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
</body>
</html>
2023-02-28 11:17:42 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-02-28 11:17:42 DEBUG : pacer: low level retry 6/10 (error <html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
</body>
</html>
: 504: 504 Gateway Time-out)
2023-02-28 11:17:42 DEBUG : pacer: Rate limited, increasing sleep to 720ms
2023-02-28 11:17:42 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-02-28 11:17:42 DEBUG : HTTP REQUEST (req 0xc000a19000)
2023-02-28 11:17:42 DEBUG : POST /service/apiadapter/api/threadedupload/finish?uploadid=b008a43a-e81f-4618-bc86-ace7cbbdb6fd&accountid=a12fa41d-cab1-bc52-1a36-fc980a523c3b&zoneid=zpfed2b3f5-fbbf-4ed5-9a58-f1bd888f01&zsid=ef&h=ksy%2BFK2fAo5tofHBk6B%2FduWzX4InchMWwuL09%2BY9AqA%3D&fmt=json HTTP/1.1
Host: sf-apiadapter-sharefile-useast.sharefile.com
User-Agent: rclone/v1.61.1
Content-Length: 0
Authorization: XXXX
Accept-Encoding: gzip
2023-02-28 11:17:42 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Transferred:       10.039 GiB / 10 GiB, 100%, 0 B/s, ETA -
Transferred:            0 / 1, 0%
Elapsed time:     15m28.5s
Transferring:
 *                                      10Gb.dat:100% /10Gi, 0/s, -
6

That's a bug at Sharefile I think. 504 Gateway Timeout means that the server you were talking to at sharefile timed out talking to the next server along.

Rclone retried the chunk and all was well though, so let's move along to the end of the transfer.

And then tried to finish the transfer which gave another timeout after 2 minutes.

Rclone tries this a few times but it fails.

Was the file uploaded OK in the end?

So if you wanted to report this to Sharefile you'd say

The file gets uploaded correctly in parts using the threaded upload method but calling the FinishURI to complete the file gives a 504 Gateway Timeout after 2 minutes. Finishing a 10GB file seems to be taking longer than the 2 minute gateway timeout.

I don't think this is something rclone can solve - it needs sharefile to raise that timeout or suggest a different way of finishing the file.

I conjecture the reason finishing the file takes a long time is that it is likely calculating the MD5 hash which involves reading 10GB of data on a busy machine over a busy network. Or maybe it is joining the parts together, so reading and writing 10GB of data.

7

Nick, I greatly appreciate your troubleshooting assistance. I have reached back out to ShareFile with the information you provided, hopefully this goes... somewhere.

The file does not show in the end. :frowning:

If I get anything back from ShareFile, I will post it here.

1 Like
8

Update from ShareFile TechSupport - Storing beyond a few TB in their platform is not supported, and WILL cause bugs in the API.

9

:frowning:

I guess you could try the chunker backend?

10

That's what I did end up having to do, just wanted to get to the root of the non-chunked issue. Just glad to know it's not RClone's fault!

1 Like
11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.