How to pass HTTP headers in a Connection String

Help and Support
1

What is the problem you are having with rclone?

I am trying to upload Confluence backups from Atlassian Cloud to a Google Storage Bucket.
So far, I have successfully uploaded files using a configuration file.
Now, we need to optimize this and make it more dynamic in the sense that we don't want to use config files. Instead, we need to be able to pass the config parameters directly in the command with connections strings.

To begin with, I am trying to download the confluence backup to my local machine and I am failing with passing the headers in the connection string. Can you please advise if it's possible to pass HTTP headers with connection strings? If yes, how?

Run the command 'rclone version' and share the full output of the command.

rclone v1.62.2
- os/version: Microsoft Windows 10 Pro 21H2 (64 bit)
- os/kernel: 10.0.19044.2965 Build 19044.2965.2965 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.20.2
- go/linking: static
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Google Cloud Storage

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy ":http,url='https://<my_cloudsite>.atlassian.net/wiki/download/temp/filestore/':'ea99194c-5199-4568-bb84-6f7c177d889c',headers='Content-Type,application/json,Authorization,<my_authorization_token>'" C:\TEMP

The rclone config contents with secrets removed.

The following config file works fine

[gcs]
type = google cloud storage
object_acl = private
bucket_acl = private
location = europe-north1
storage_class = NEARLINE
env_auth = false
token = <my_token>
project_number = <my_gcp_project>

[atlas-conf]
type = http
url = https://<my_cloudsite>.atlassian.net/wiki/download/temp/filestore/
headers = Content-Type,application/json,Authorization,,<my_authorization_token>

A log from the command with the -vv flag

This response is from Atlassian Cloud. I think it doesn't receive the authorization header.

2023/05/22 15:50:39 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "copy" ":http,url='https://<my_cloudsite>.atlassian.net/wiki/download/temp/filestore/':'ea99194c-5199-4568-bb84-6f7c177d889c',headers='Content-Type,application/json,Authorization,<mytoken>'" "'C:\\TEMP'" "-vv"]
2023/05/22 15:50:39 DEBUG : Creating backend with remote ":http,url='https://<my_cloudsite>.atlassian.net/wiki/download/temp/filestore/':'ea99194c-5199-4568-bb84-6f7c177d889c',headers='Content-Type,application/json,Authorization,<mytoken>'"
2023/05/22 15:50:39 DEBUG : Using config file from "C:\\Users\\dimit\\AppData\\Roaming\\rclone\\rclone.conf"
2023/05/22 15:50:39 DEBUG : :http: detected overridden config - adding "{vOGmu}" suffix to name
2023/05/22 15:50:39 DEBUG : Assuming path is a file as HEAD response is an error (403 Forbidden)
2023/05/22 15:50:39 DEBUG : If path is a directory you must add a trailing '/'
2023/05/22 15:50:39 DEBUG : Root: https://<my_cloudsite>.atlassian.net/wiki/download/temp/filestore/%27ea99194c-5199-4568-bb84-6f7c177d889c%27,headers=%27Content-Type,application/
2023/05/22 15:50:39 DEBUG : fs cache: adding new entry for parent of ":http,url='https://<my_cloudsite>.atlassian.net/wiki/download/temp/filestore/':'ea99194c-5199-4568-bb84-6f7c177d889c',headers='Content-Type,application/json,Authorization,<>'", ":http{vOGmu}:'ea99194c-5199-4568-bb84-6f7c177d889c',headers='Content-Type,application/json,Authorization,<>'"
2023/05/22 15:50:39 DEBUG : Creating backend with remote "'C:\\TEMP'"
2023/05/22 15:50:39 Failed to create file system for "'C:\\TEMP'": config name contains invalid characters - may only contain numbers, letters, `_`, `-`, `.`, `+`, `@` and space, while not start with `-` or space, and not end with space
2

hello and welcome to the forum,

can you run the command with -vv and post the full output

not an expert with connections strings but the domain name and path should be separated by a colon
url='https://<my_cloudsite>.atlassian.net':wiki/download/temp/filestore/ea99194c-5199-4568-bb84-6f7c177d889c'

3

Thanks for your response and sorry for missing that. I've updated the log section above.

4

if you re-edit old posts, get confusing, ok?

to simplify this, test with rclone ls -vv and post the full debug output.

5

After your suggestion, I changed the connection string:

Command

rclone ls -vv ":http,url='https://<mycloudsite>.atlassian.net':'/wiki/download/temp/filestore/ea99194c-5199-4568-bb84-6f7c177d889c',headers='Content-Type,application/json,Authorization,<token>'"

Log
This is helpful cause I see the constructed url in the log below and it is wrong
2023/05/22 16:35:34 DEBUG : Root: https://<cloudsite>.atlassian.net/%27/wiki/download/temp/filestore/ea99194c-5199-4568-bb84-6f7c177d889c%27,headers=%27Content-Type,application/json,Authorization,<mytoken>%27/

In my understanding, the character single quote ' is replaced by %27 at the constructed URL

This is the full log from cmd

2023/05/22 16:35:33 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "ls" "-vv" ":http,url='https://<cloudsite>.atlassian.net':'/wiki/download/temp/filestore/ea99194c-5199-4568-bb84-6f7c177d889c',headers='Content-Type,application/json,Authorization,<mytoken>'"]
2023/05/22 16:35:33 DEBUG : Creating backend with remote ":http,url='https://<cloudsite>.atlassian.net':'/wiki/download/temp/filestore/ea99194c-5199-4568-bb84-6f7c177d889c',headers='Content-Type,application/json,Authorization,<mytoken>'"
2023/05/22 16:35:33 DEBUG : Using config file from "C:\\Users\\dimit\\AppData\\Roaming\\rclone\\rclone.conf"
2023/05/22 16:35:33 DEBUG : :http: detected overridden config - adding "{T4DlP}" suffix to name
2023/05/22 16:35:34 DEBUG : Assuming path is a directory as HEAD response is it does not exist as a file (404 Not Found)
2023/05/22 16:35:34 DEBUG : To avoid the initial HEAD request add a trailing '/' to the path
2023/05/22 16:35:34 DEBUG : Root: https://<cloudsite>.atlassian.net/%27/wiki/download/temp/filestore/ea99194c-5199-4568-bb84-6f7c177d889c%27,headers=%27Content-Type,application/json,Authorization,<mytoken>%27/
2023/05/22 16:35:34 DEBUG : fs cache: renaming cache item ":http,url='https://<cloudsite>.atlassian.net':'/wiki/download/temp/filestore/ea99194c-5199-4568-bb84-6f7c177d889c',headers='Content-Type,application/json,Authorization,<mytoken>'" to be canonical ":http{T4DlP}:'/wiki/download/temp/filestore/ea99194c-5199-4568-bb84-6f7c177d889c',headers='Content-Type,application/json,Authorization,<mytoken>'"
2023/05/22 16:35:34 ERROR : : error listing: error listing "": directory not found
2023/05/22 16:35:34 DEBUG : 3 go routines active
2023/05/22 16:35:34 Failed to ls with 2 errors: last error was: error listing "": directory not found

Do you know how can I escape this character in Windows?

6

using command line., cmd batch file, powershell or what?

try this format

rclone ls -vv :http: --http-headers="Content-Type,application/json,Authorization,'1//0h9C5MPJwUJo4CgYIARAAGBESNwF-L9Ir4NoazoXYt8B1fHhAegRcmjISmndFatlUt_SYstasLG_zDjILrRxoeql4UKhWSsaR02k'" --http-url="https://downloads.rclone.org":/v1.62.2 
DEBUG : Setting --config "C:\\data\\rclone\\rclone.conf" from environment variable RCLONE_CONFIG="C:\\data\\rclone\\rclone.conf"
DEBUG : rclone: Version "v1.62.2" starting with parameters ["C:\\data\\rclone\\rclone.exe" "ls" "-vv" ":http:" "--http-headers=Content-Type,application/json,Authorization,'1//0h9C5MPJwUJo4CgYIARAAGBESNwF-L9Ir4NoazoXYt8B1fHhAegRcmjISmndFatlUt_SYstasLG_zDjILrRxoeql4UKhWSsaR02k'" "--http-url=https://downloads.rclone.org:/v1.62.2"]
DEBUG : Creating backend with remote ":http:"
DEBUG : Using config file from "C:\\data\\rclone\\rclone.conf"
DEBUG : :http: detected overridden config - adding "{mFj--}" suffix to name
DEBUG : Root: https://downloads.rclone.org:/v1.62.2/
DEBUG : fs cache: renaming cache item ":http:" to be canonical ":http{mFj--}:"
     3228 MD5SUMS
     3596 SHA1SUMS
     4700 SHA256SUMS
 16738480 rclone-v1.62.2-freebsd-386.zip
<SNIP>
 16127843 rclone-v1.62.2.tar.gz
       15 version.txt

else down the rabbit hole
https://forum.rclone.org/t/unable-to-use-on-fly-backend-with-configuration-string-in-shell/26625

7

Just a thought. If your solution with config file works why not to make config dynamic part?

You can start rclone with --config

rclone --config=/path/to/temp_config_job1.config

You could package it all in some script where you would call it with required parameters and it would create config and invoke rclone.

8

hi, my examples do not use config file?

if the OP could use a script without config file, i would use environment variables, not connection string.
more secure that way.

9

ahh sorry I mixed up

@dimitrisd mentioned in the original post:

And yes environment variables are the way to go too:) There's more than one way to skin a cat .