Trying to pass remote HTTP parameter via question mark

Suspected Bug
1

What is the problem you are having with rclone?

I am trying to use rclone to download backup file from Atlassian Jira to a local folder.

Run the command 'rclone version' and share the full output of the command.

rclone v1.63.1

  • os/version: Microsoft Windows Server 2022 Datacenter 21H2 (64 bit)
  • os/kernel: 10.0.20348.1487 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.20.6
  • go/linking: static
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Atlassian Jira

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy atlassian:/plugins/servlet/export/download/?fileId=a7307c5e-9d3d-4710-595f-ce6f0d9ec6df C:\Backup\

The rclone config contents with secrets removed.

[atlassian]
type = http
url = https://mycompany.atlassian.net
headers = "Host","mycompany.atlassian.net","Authorization","Basic keyhere"

A log from the command with the -vv flag

C:\>rclone copy atlassian:plugins/servlet/export/download/?fileId=a7307c5e-9d3d-4710-595f-ce6f0d9ec6df C:\Backup -vv
2023/08/04 10:58:23 DEBUG : rclone: Version "v1.63.1" starting with parameters ["C:\\ProgramData\\chocolatey\\lib\\rclone.portable\\tools\\rclone-v1.63.1-windows-amd64\\rclone.exe" "copy" "atlassian:plugins/servlet/export/download/?fileId=7c30a75e-dd93-4071-95f5-6dfce9ec6f0d" "C:\\Backup" "-vv"]
2023/08/04 10:58:23 DEBUG : Creating backend with remote "atlassian:plugins/servlet/export/download/?fileId=a7307c5e-9d3d-4710-595f-ce6f0d9ec6df"
2023/08/04 10:58:23 DEBUG : Using config file from "C:\\Users\\me\\AppData\\Roaming\\rclone\\rclone.conf"
2023/08/04 10:58:23 DEBUG : Assuming path is a file as HEAD response is an error (400 Bad Request)
2023/08/04 10:58:23 DEBUG : If path is a directory you must add a trailing '/'
2023/08/04 10:58:23 DEBUG : Root: https://mycompany.atlassian.net/plugins/servlet/export/download/
2023/08/04 10:58:23 DEBUG : Creating backend with remote "C:\\Backup"
2023/08/04 10:58:23 DEBUG : fs cache: renaming cache item "C:\\Backup" to be canonical "//?/C:/Backup"
2023/08/04 10:58:23 ERROR : Attempt 1/3 failed with 1 errors and: failed to stat: HTTP Error: 400 Bad Request
2023/08/04 10:58:23 ERROR : Attempt 2/3 failed with 1 errors and: failed to stat: HTTP Error: 400 Bad Request
2023/08/04 10:58:24 ERROR : Attempt 3/3 failed with 1 errors and: failed to stat: HTTP Error: 400 Bad Request
2023/08/04 10:58:24 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         1.0s

2023/08/04 10:58:24 DEBUG : 3 go routines active
2023/08/04 10:58:24 Failed to copy: failed to stat: HTTP Error: 400 Bad Request

Thoughts

My suggestion is that rclone does not pass parameter fileId in url string which is after question mark. It is based on running rclone with adding --dump=bodies --retries=1 --low-level-retries=1 --log-level=DEBUG parameters to command line. I see in debug that Atlassian responds with error page which states: fileId parameter must be specified.

2

Could you try to put your src in quotation marks?

rclone copy "atlassian:/plugins/servlet/export/download/?fileId=a7307c5e-9d3d-4710-595f-ce6f0d9ec6df" C:\Backup\

3

Yes, I tried this already:

  • rclone copy "atlassian:/plugins/servlet/export/download/?fileId=a7307c5e-9d3d-4710-595f-ce6f0d9ec6df" C:\Backup\
  • rclone copy atlassian:"/plugins/servlet/export/download/?fileId=a7307c5e-9d3d-4710-595f-ce6f0d9ec6df" C:\Backup\

Result is the same http 400.

4

Thanks for confirmation

5

Also I tried to run with inline config:

rclone copy :http: --http-url "https://mycompany.atlassian.net/plugins/servlet/export/download/?fileId=a7307c5e-9d3d-4710-595f-ce6f0d9ec6df" --http-headers '"Host","mycompany.atlassian.net","Authorization","Basic keyhere"' C:\Backup --http-no-slash --dump=bodies --retries=1 --low-level-retries=1 --log-level=DEBUG

Result differs from previous http 400 but also fails:

C:\rclone copy :http: --http-url "https://mycompany.atlassian.net/plugins/servlet/export/download/?fileId=a7307c5e-9d3d-4710-595f-ce6f0d9ec6df" --http-headers '"Host","mycompany.atlassian.net","Authorization","Basic keyhere"' C:\Backup --http-no-slash --dump=bodies --retries=1 --low-level-retries=1 --log-level=DEBUG
2023/08/04 13:07:35 DEBUG : rclone: Version "v1.63.1" starting with parameters ["C:\\ProgramData\\chocolatey\\lib\\rclone.portable\\tools\\rclone-v1.63.1-windows-amd64\\rclone.exe" "copy" ":http:" "--http-url" "https://mycompany.atlassian.net/plugins/servlet/export/download/?fileId=a7307c5e-9d3d-4710-595f-ce6f0d9ec6df" "--http-headers" "'Host,mycompany.atlassian.net,Authorization,Basic keyhere'" "C:\\Backup" "--http-no-slash" "--dump=bodies" "--retries=1" "--low-level-retries=1" "--log-level=DEBUG"]
2023/08/04 13:07:35 DEBUG : Creating backend with remote ":http:"
2023/08/04 13:07:35 DEBUG : Using config file from "C:\\Users\\me\\AppData\\Roaming\\rclone\\rclone.conf"
2023/08/04 13:07:35 DEBUG : :http: detected overridden config - adding "{ZhEwX}" suffix to name
2023/08/04 13:07:35 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2023/08/04 13:07:35 DEBUG : Root: https://mycompany.atlassian.net/plugins/servlet/export/download/?fileId=a7307c5e-9d3d-4710-595f-ce6f0d9ec6df/
2023/08/04 13:07:35 DEBUG : fs cache: renaming cache item ":http:" to be canonical ":http{ZhEwX}:"
2023/08/04 13:07:35 DEBUG : Creating backend with remote "C:\\Backup"
2023/08/04 13:07:35 DEBUG : fs cache: renaming cache item "C:\\Backup" to be canonical "//?/C:/Backup"
2023/08/04 13:07:35 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/08/04 13:07:35 DEBUG : HTTP REQUEST (req 0xc000820b00)
2023/08/04 13:07:35 DEBUG : GET /plugins/servlet/export/download/?fileId=a7307c5e-9d3d-4710-595f-ce6f0d9ec6df/ HTTP/1.1
Host: mycompany.atlassian.net
User-Agent: rclone/v1.63.1
'host: mycompany.atlassian.net
Authorization: XXXX
Accept-Encoding: gzip

2023/08/04 13:07:35 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/08/04 13:07:35 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/08/04 13:07:35 DEBUG : HTTP RESPONSE (req 0xc000820b00)
2023/08/04 13:07:35 DEBUG : HTTP/2.0 302 Found
Atl-Traceid: 912c781b4ab03cac
Cache-Control: no-cache, no-store, must-revalidate
Content-Type: text/html;charset=UTF-8
Date: Fri, 04 Aug 2023 10:07:35 GMT
Expires: Wed, 31 Dec 1969 23:59:59 GMT
Location: https://api.media.atlassian.com/file/a7307c5e-9d3d-4710-595f-ce6f0d9ec6df//binary?token=tokenhere&client=c7cb6fea-3332-634e-6bd0-a09f5d1769a2&dl=true&name=jira-export.zip&max-age=82800
Nel: {"failure_fraction": 0.001, "include_subdomains": true, "max_age": 600, "report_to": "endpoint-1"}
Pragma: no-cache
Report-To: {"endpoints": [{"url": "https://dz8aopenkvv6s.cloudfront.net"}], "group": "endpoint-1", "include_subdomains": true, "max_age": 600}
Server: AtlassianEdge
Set-Cookie: atlassian.xsrf.token=tokenhere_lin; path=/; SameSite=None; Secure
Strict-Transport-Security: max-age=63072000; includeSubDomains; preload
Timing-Allow-Origin: *
Vary: Accept-Encoding
X-Aaccountid: 62bc04affa171a27239d6e29
X-Arequestid: c73bf4b0170019851391ad1886ea48fe
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block
Content-Length: 0

2023/08/04 13:07:35 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/08/04 13:07:35 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/08/04 13:07:35 DEBUG : HTTP REQUEST (req 0xc000821200)
2023/08/04 13:07:35 DEBUG : GET /file/a7307c5e-9d3d-4710-595f-ce6f0d9ec6df//binary?token=tokenhere&client=c7cb6fea-3332-634e-6bd0-a09f5d1769a2&dl=true&name=jira-export.zip&max-age=82800 HTTP/1.1
Host: api.media.atlassian.com
User-Agent: rclone/v1.63.1
'host: mycompany.atlassian.net
Referer: https://mycompany.atlassian.net/plugins/servlet/export/download/?fileId=a7307c5e-9d3d-4710-595f-ce6f0d9ec6df/
Accept-Encoding: gzip

2023/08/04 13:07:35 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/08/04 13:07:35 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/08/04 13:07:35 DEBUG : HTTP RESPONSE (req 0xc000821200)
2023/08/04 13:07:35 DEBUG : HTTP/2.0 403 Forbidden
Content-Length: 254
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Accept-Ranges, Content-Encoding, Content-Length, Content-Range, X-Media-Region, X-Media-Env
Atl-Traceid: 8cea9ed0834c3108
Baggage: atl-paas-enc-byok=false,atl-paas-sharding=keyhere
Content-Type: application/json; charset=utf-8
Date: Fri, 04 Aug 2023 10:07:35 GMT
Nel: {"failure_fraction": 0.001, "include_subdomains": true, "max_age": 600, "report_to": "endpoint-1"}
Report-To: {"endpoints": [{"url": "https://dz8aopenkvv6s.cloudfront.net"}], "group": "endpoint-1", "include_subdomains": true, "max_age": 600}
Server: AtlassianEdge
Strict-Transport-Security: max-age=63072000; preload
X-B3-Parentspanid: 9e177bd4637806d8
X-B3-Sampled: 0
X-B3-Spanid: f00e6421b28fa78d
X-B3-Traceid: 8cea9ed0834c3108
X-Content-Type-Options: nosniff
X-Dns-Prefetch-Control: off
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-Media-Env: prod-eucentral
X-Media-Region: eu-central-1
X-Xss-Protection: 1; mode=block

{"error":{"code":"JwtAuthoriser:AuthorisationError","title":"The token does not allow requested operation. \"urn:filestore:file:a7307c5e-9d3d-4710-595f-ce6f0d9ec6df\" needs \"read\" permission","href":"https://api.media.atlassian.com#UnauthorizedError"}}
2023/08/04 13:07:35 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/08/04 13:07:35 ERROR : https://mycompany.atlassian.net/plugins/servlet/export/download/?fileId=a7307c5e-9d3d-4710-595f-ce6f0d9ec6df/: error reading source root directory: error listing "": failed to readDir: HTTP Error: 403 Forbidden
2023/08/04 13:07:35 DEBUG : Local file system at //?/C:/Backup: Waiting for checks to finish
2023/08/04 13:07:35 DEBUG : Local file system at //?/C:/Backup: Waiting for transfers to finish
2023/08/04 13:07:35 ERROR : Attempt 1/1 failed with 1 errors and: error listing "": failed to readDir: HTTP Error: 403 Forbidden
2023/08/04 13:07:35 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.6s

2023/08/04 13:07:35 DEBUG : 4 go routines active
2023/08/04 13:07:35 Failed to copy: error listing "": failed to readDir: HTTP Error: 403 Forbidden

As a test I tried to download this file located at 'https://mycompany.atlassian.net/plugins/servlet/export/download/?fileId=a7307c5e-9d3d-4710-595f-ce6f0d9ec6df' with Windows Powershell System.Net.WebClient (DownloadFile) passing the same headers and it worked. Cant't figure out where is the problem using rclone.

6

That is nearly there you can see that from the http get. However there is a / on the end.

Are you just copying a single file? You might find rclone copyurl easier to use in that case.

7

Still no success. Tried that:

rclone copyurl https://mycompany.atlassian.net/plugins/servlet/export/download/?fileId=a7307c5e-9d3d-4710-595f-ce6f0d9ec6df --header "Host: mycompany.atlassian.net" --header "Authorization: Basic keyhere" C:\Backup -vv

With result:

2023/08/05 10:26:05 DEBUG : rclone: Version "v1.63.1" starting with parameters ["C:\\ProgramData\\chocolatey\\lib\\rclone.portable\\tools\\rclone-v1.63.1-windows-amd64\\rclone.exe" "copyurl" "https://mycompany.atlassian.net/plugins/servlet/export/download/?fileId=a7307c5e-9d3d-4710-595f-ce6f0d9ec6df" "--header" "Host: mycompany.atlassian.net" "--header" "Authorization: Basic keyhere" "C:\\Backup" "-vv"]
2023/08/05 10:26:05 DEBUG : Creating backend with remote "C:/"
2023/08/05 10:26:05 DEBUG : Using config file from "C:\\Users\\me\\AppData\\Roaming\\rclone\\rclone.conf"
2023/08/05 10:26:05 DEBUG : fs cache: renaming cache item "C:/" to be canonical "//?/C:/"
2023/08/05 10:26:05 ERROR : Backup: Post request put error: Access is denied.
2023/08/05 10:26:05 ERROR : Attempt 1/3 failed with 2 errors and: Access is denied.
2023/08/05 10:26:06 ERROR : Backup: Post request put error: Access is denied.
2023/08/05 10:26:06 ERROR : Attempt 2/3 failed with 2 errors and: Access is denied.
2023/08/05 10:26:06 ERROR : Backup: Post request put error: Access is denied.
2023/08/05 10:26:06 ERROR : Attempt 3/3 failed with 2 errors and: Access is denied.
2023/08/05 10:26:06 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 2 (retrying may help)
Elapsed time:         1.3s

2023/08/05 10:26:06 DEBUG : 5 go routines active
2023/08/05 10:26:06 Failed to copyurl with 2 errors: last error was: Access is denied.

Running the same with debug:

rclone copyurl https://pulse-insure.atlassian.net/plugins/servlet/export/download/?fileId=a7307c5e-9d3d-4710-595f-ce6f0d9ec6df --header "Host: mycompany.atlassian.net" --header "Authorization: Basic keyhere" C:\Backup --dump=bodies --retries=1 --low-level-retries=1 --log-level=DEBUG

Brings the following output:

C:\Scripts>rclone copyurl https://mycompany.atlassian.net/plugins/servlet/export/download/?fileId=a7307c5e-9d3d-4710-595f-ce6f0d9ec6df --header "Host: mycompany.atlassian.net" --header "Authorization: Basic tokenhere" C:\Backup --dump=bodies --retries=1 --low-level-retries=1 --log-level=DEBUG
2023/08/05 10:42:34 DEBUG : rclone: Version "v1.63.1" starting with parameters ["C:\\ProgramData\\chocolatey\\lib\\rclone.portable\\tools\\rclone-v1.63.1-windows-amd64\\rclone.exe" "copyurl" "https://mycompany.atlassian.net/plugins/servlet/export/download/?fileId=a7307c5e-9d3d-4710-595f-ce6f0d9ec6df" "--header" "Host: mycompany.atlassian.net" "--header" "Authorization: Basic tokenhere" "C:\\Backup" "--dump=bodies" "--retries=1" "--low-level-retries=1" "--log-level=DEBUG"]
2023/08/05 10:42:34 DEBUG : Creating backend with remote "C:/"
2023/08/05 10:42:34 DEBUG : Using config file from "C:\\Users\\me\\AppData\\Roaming\\rclone\\rclone.conf"
2023/08/05 10:42:34 DEBUG : fs cache: renaming cache item "C:/" to be canonical "//?/C:/"
2023/08/05 10:42:34 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2023/08/05 10:42:34 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/08/05 10:42:34 DEBUG : HTTP REQUEST (req 0xc000826a00)
2023/08/05 10:42:34 DEBUG : GET /plugins/servlet/export/download/?fileId=a7307c5e-9d3d-4710-595f-ce6f0d9ec6df HTTP/1.1
Host: mycompany.atlassian.net
User-Agent: rclone/v1.63.1
Authorization: XXXX
Accept-Encoding: gzip

2023/08/05 10:42:34 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/08/05 10:42:34 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/08/05 10:42:34 DEBUG : HTTP RESPONSE (req 0xc000826a00)
2023/08/05 10:42:34 DEBUG : HTTP/2.0 302 Found
Atl-Traceid: 8b19510075133463
Cache-Control: no-cache, no-store, must-revalidate
Content-Type: text/html;charset=UTF-8
Date: Sat, 05 Aug 2023 07:42:34 GMT
Expires: Wed, 31 Dec 1969 23:59:59 GMT
Location: https://api.media.atlassian.com/file/a7307c5e-9d3d-4710-595f-ce6f0d9ec6df/binary?token=tokenhere&client=c7cb6fea-3332-634e-6bd0-a09f5d1769a2&dl=true&name=jira-export.zip&max-age=82800
Nel: {"failure_fraction": 0.001, "include_subdomains": true, "max_age": 600, "report_to": "endpoint-1"}
Pragma: no-cache
Report-To: {"endpoints": [{"url": "https://dz8aopenkvv6s.cloudfront.net"}], "group": "endpoint-1", "include_subdomains": true, "max_age": 600}
Server: AtlassianEdge
Set-Cookie: atlassian.xsrf.token=69c7aae2-3be0-4216-b795-1d7926bffafa_2a68758352932f46670a28e3b086e4b7422059ca_lin; path=/; SameSite=None; Secure
Strict-Transport-Security: max-age=63072000; includeSubDomains; preload
Timing-Allow-Origin: *
Vary: Accept-Encoding
X-Aaccountid: 62bc04affa171a27239d6e29
X-Arequestid: d565b2e23ca2a43ac0cc6b0285d98392
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block
Content-Length: 0

2023/08/05 10:42:34 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/08/05 10:42:34 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/08/05 10:42:34 DEBUG : HTTP REQUEST (req 0xc000648400)
2023/08/05 10:42:34 DEBUG : GET /file/a7307c5e-9d3d-4710-595f-ce6f0d9ec6df/binary?token=tokenhere&client=c7cb6fea-3332-634e-6bd0-a09f5d1769a2&dl=true&name=jira-export.zip&max-age=82800 HTTP/1.1
Host: api.media.atlassian.com
User-Agent: rclone/v1.63.1
Authorization: XXXX
Referer: https://mycompany.atlassian.net/plugins/servlet/export/download/?fileId=a7307c5e-9d3d-4710-595f-ce6f0d9ec6df
Accept-Encoding: gzip

2023/08/05 10:42:34 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/08/05 10:42:34 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/08/05 10:42:34 DEBUG : HTTP RESPONSE (req 0xc000648400)
2023/08/05 10:43:34 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:       1m0.2s

2023/08/05 10:45:44 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:       3m5.3s

2023/08/05 10:45:49 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:      3m15.9s

2023/08/05 10:45:52 DEBUG :
2023/08/05 10:45:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/08/05 10:46:13 ERROR : Backup: Post request put error: Access is denied.
2023/08/05 10:46:16 ERROR : Attempt 1/1 failed with 2 errors and: Access is denied.
2023/08/05 10:46:26 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 2 (retrying may help)
Elapsed time:      3m52.2s

2023/08/05 10:46:27 DEBUG : 4 go routines active
2023/08/05 10:46:28 Failed to copyurl with 2 errors: last error was: Access is denied.

C:\Scripts>

Also when just run it burtst traffic usage to maximum for a couple minutes.

rclone works like a charm when downloading Atlassian Confluence backup. But something goes wrong when trying to deal with Atlassian Jira.

8

I think the second copyurl worked but rclone doesn't have permission to write in the C:\Backup directory. Possibly. That is a bit of a weird error!

9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.