Problem with parsing of environment variables

What is the problem you are having with rclone?

i am having issues with the parsing of environment variables.

in this case, remote-x: does not exist, but rclone will use it.

set RCLONE_CONFIG_REMOTE_X_TYPE=local 

rclone listremotes 
2021/09/15 11:26:49 NOTICE: Config file "b:\\dummy\\dummy.conf" not found - using defaults
remote_x:

rclone ls remote-x: -vv 
2021/09/15 11:26:49 DEBUG : Setting --config "dummy.conf" from environment variable RCLONE_CONFIG="dummy.conf"
2021/09/15 11:26:49 DEBUG : rclone: Version "v1.56.0" starting with parameters ["C:\\data\\rclone\\scripts\\rclone.exe" "ls" "remote-x:" "-vv"]
2021/09/15 11:26:49 DEBUG : Creating backend with remote "remote-x:"
2021/09/15 11:26:49 DEBUG : Setting type="local" for "remote-x" from environment variable RCLONE_CONFIG_REMOTE_X_TYPE
2021/09/15 11:26:49 NOTICE: Config file "b:\\dummy\\dummy.conf" not found - using defaults
2021/09/15 11:26:49 DEBUG : fs cache: renaming cache item "remote-x:" to be canonical "remote-x://?/b:/dummy"
        1 file.txt
2021/09/15 11:26:49 DEBUG : 2 go routines active

in this case, remote-x: does exist, but rclone will not use it.

set RCLONE_CONFIG_REMOTE-X_TYPE=local 

rclone listremotes 
2021/09/15 11:27:58 NOTICE: Config file "b:\\dummy\\dummy.conf" not found - using defaults
remote-x:

rclone ls remote-x: -vv 
2021/09/15 11:27:58 DEBUG : Setting --config "dummy.conf" from environment variable RCLONE_CONFIG="dummy.conf"
2021/09/15 11:27:58 DEBUG : rclone: Version "v1.56.0" starting with parameters ["C:\\data\\rclone\\scripts\\rclone.exe" "ls" "remote-x:" "-vv"]
2021/09/15 11:27:58 DEBUG : Creating backend with remote "remote-x:"
2021/09/15 11:27:58 NOTICE: Config file "b:\\dummy\\dummy.conf" not found - using defaults
2021/09/15 11:27:58 Failed to create file system for "remote-x:": didn't find section in config file

What is your rclone version (output from rclone version)

rclone version
rclone v1.56.0
- os/version: Microsoft Windows 10 Pro 2009 (64 bit)
- os/kernel: 10.0.19043.1165 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.16.5
- go/linking: dynamic
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

wasabi - s3 clone

The command you were trying to run (eg rclone copy /tmp remote:tmp)

please see the top of the post

The rclone config contents with secrets removed.

not using a config file

A log from the command with the -vv flag

please see the top of post

on linux, using WSL,

same as windows, remote-x: does not exist but rclone will use it.

export RCLONE_CONFIG_REMOTE_X_TYPE=local

rclone listremotes
2021/09/15 11:52:18 NOTICE: Config file "/mnt/c/data/rclone/scripts/rr/other/newtest/dummy.conf" not found - using defaults
remote_x:

rclone ls remote-x: -vv
2021/09/15 11:45:14 NOTICE: Config file "/mnt/c/data/rclone/scripts/rr/other/newtest/dummy.conf" not found - using defaults
remote_x:
2021/09/15 11:45:14 DEBUG : Setting --config "dummy.conf" from environment variable RCLONE_CONFIG="dummy.conf"
2021/09/15 11:45:14 DEBUG : rclone: Version "v1.56.0" starting with parameters ["rclone" "ls" "remote-x:" "-vv"]
2021/09/15 11:45:14 DEBUG : Creating backend with remote "remote-x:"
2021/09/15 11:45:14 DEBUG : Setting type="local" for "remote-x" from environment variable RCLONE_CONFIG_REMOTE_X_TYPE
2021/09/15 11:45:14 NOTICE: Config file "/mnt/c/data/rclone/scripts/rr/other/newtest/dummy.conf" not found - using defaults
2021/09/15 11:45:14 DEBUG : fs cache: renaming cache item "remote-x:" to be canonical "remote-x:/mnt/c/data/rclone/scripts/rr/other/newtest"
      276 file.txt
2021/09/15 11:45:14 DEBUG : 2 go routines active

not same as windows, as bash does not like the variable name

export RCLONE_CONFIG_REMOTE-X_TYPE=local
./onfly.remote.sh: line 16: export: `RCLONE_CONFIG_REMOTE-X_TYPE=local': not a valid identifier

Interesting..

Looking at the code, the following are relevant:

With rclone ls remote-x:, it will look up config for remote remote-x, and when considering environment variables it will look for RCLONE_CONFIG_REMOTE_X_TYPE and never RCLONE_CONFIG_REMOTE_X-TYPE. This also means rclone ls remote-x: and rclone ls remote_x: gives the same result. And rclone listremotes will use the name parsed from environment variable, not translating _ back to - (it wouldn't know which is "more" correct).

The listremotes command considers environment variables starting with RCLONE_CONFIG_ and ending with TYPE, and returns everything between converted to lowercase, as seen from the following code:

Does not answer all your questions, probably, but maybe some?

Seems you can use env, maybe something like this (untested):

env 'RCLONE_CONFIG_REMOTE-X_TYPE=local' rclone listremotes

thanks, very interesting.

in my testing that was true, but when i posted, i wanted to use the simplest example.

well, at this point, seems like a bug to me, do you agree?

  1. rclone allows the use of remotes that do not exist.
  2. rclone disallows the use of remotes that do exist.

what is the next step?

Not sure.. bug, or a (undocumented) limitation...

I guess using - in environment variable names is so inconvenient, perhaps not even properly supported everywhere. If you have a remote remote-x in configfile it is nice to be able to override its config with RCLONE_CONFIG_REMOTE_X_SOMETHING.

Another case: I see that with rclone config, I am able to create two remotes in my config file:

rclone listremotes
remote-x:
remote_x:

Then if both are FTP remotes, I can set the following environment variable to override the username to use:

set RCLONE_CONFIG_REMOTE_X_USER=otheruser

Now it will be considered when using any of the remotes.

So I'm a bit torn what to make of this, or how to improve it in a good way.

Hopefully some thoughts from other developers, perhaps even @ncw. Then maybe create an issue in GitHub. If nothing else then at least some documentation should be added, and in any case it is less likely to just be forgotten there.

”-” isn’t always possible in environment variables:
https://stackoverflow.com/questions/2821043/allowed-characters-in-linux-environment-variable-names

This probably explains the code in ConfigToEnv and OptionToEnv, so in this perspective there is a bug in FileSections. Fixing this will however make it impossible to differ between remote-x and remote_x.

I therefore propose a more relaxed approach where the remote name is used as is, and then leave it up to the user to keep remote names within the limits of their environment, e.g. free from "-" if using environment variables in Powershell. This will still allow using both remote-x and remote_x, if you are able to create the environment variables in your environment/shell.

I haven't tried or tested, but I guess it can be achieved with these simple code changes (that include a parameter renaming to make the code easier to understand):

// ConfigToEnv converts a config section and name, e.g. ("myremote",
// "ignore-size") into an environment name
// "RCLONE_CONFIG_MYREMOTE_IGNORE_SIZE"
func ConfigToEnv(option, name string) string {
	return "RCLONE_CONFIG_" + strings.ToUpper(strings.Replace(option, "-", "_", -1)+"_"+name)
}

          
// OptionToEnv converts an option name, e.g. "ignore-size" into an
// environment name "RCLONE_IGNORE_SIZE"
func OptionToEnv(option string) string {
	return "RCLONE_" + strings.ToUpper(strings.Replace(option, "-", "_", -1))
}

Do you like it?

Does anybody fancy trying, testing and making the PR?
(I am currently busy with a fix to the pacers handling of "Too many requests")

I was thinking about this as well, but had a concern that rclone was not only "reading" options from environment variables but also "writing" them to environment, and that it then could be issues with - in names not just when user wanted to override options from env. But it seems I was wrong, (don't know where I had it from)... Then I agree, the solution could be as simple as stop mangling with the remote name part of the environment variable names.

I can try it out, and make a PR for it.

PS: I'm a bit confused with your changes in ConfigToEnv:

Original:

func ConfigToEnv(section, name string) string {
	return "RCLONE_CONFIG_" + strings.ToUpper(strings.Replace(section+"_"+name, "-", "_", -1))
}

Your:

func ConfigToEnv(option, name string) string {
	return "RCLONE_CONFIG_" + strings.ToUpper(strings.Replace(option, "-", "_", -1)+"_"+name)
}

I think you meant something like this:

func ConfigToEnv(name, option string) string {
	return "RCLONE_CONFIG_" + name + "_" + strings.ToUpper(strings.Replace(option, "-", "_", -1))
}

Super!

Yes, thanks! This may be even better:

func ConfigToEnv(remote, option string) string {
	return "RCLONE_CONFIG_" + remote + "_" + strings.ToUpper(strings.Replace(option, "-", "_", -1))
}

I do however suggest you also check the naming conventions in the area of the code where it is being called, sometimes the code use a terminology that is a little different from the user guides.

thanks,

i have a python script that does the following, using the same text name for all operations.
given that rclone still does not accept a ., in a remote name, i am forced to use -
and since rclone ls remote-x: and rclone ls remote_x: gives the same result, i use that workaround

  • create bucket
  • create IAM user
  • create user policy locked to that bucket. require MFA TOTP and session token to access the bucket.
  • creates a on the fly rclone remote, using MFA TOTP, session token

any chance of allowing a rclone remote use .?

thanks

Seems to work fine. But there might be gotchas I'm not aware of...

thanks much, looking forward to testing it.

i see that the build failed, please let us know when/if the beta is uploaded.

Beta builds:

  • Support for dot character in remote names: Here.
  • Support for hyphen in remote name from environment variable: Here.

hi and thanks,

Support for dot character in remote names
seems to work

set RCLONE_CONFIG_REMOTE.X_TYPE=local 

rclone_v1.56.0.exe about remote.x: 
Failed to create file system for "remote.x:": config name contains invalid characters - may only contain `0-9`, `A-Z`, `a-z`, `_`, `-` and space

rclone-v1.57.0-beta.5659.cb3703d05.remote-name-dot.exe about remote.x: 
Total:   145.986 GiB
Used:    87.399 GiB
Free:    58.588 GiB

Support for hyphen in remote name from environment variable
seems to work

set RCLONE_CONFIG_REMOTE-X_TYPE=local 

rclone_v1.56.0.exe config show remote_x: 
[remote_x]
Couldn't find type of fs for "remote_x"

rclone_v1.56.0.exe config show remote-x: 
[remote-x]
Couldn't find type of fs for "remote-x"

rclone_v1.57.0-beta.5660.8aa273a9e.remote-name-env.exe config show remote_x: 
[remote_x]
Couldn't find type of fs for "remote_x"

rclone_v1.57.0-beta.5660.8aa273a9e.remote-name-env.exe config show remote-x: 
[remote-x]

-----------------------------

set RCLONE_CONFIG_REMOTE_X_TYPE=local 

rclone_v1.56.0.exe config show remote_x: 
[remote_x]

rclone_v1.56.0.exe config show remote-x: 
[remote-x]

rclone_v1.57.0-beta.5660.8aa273a9e.remote-name-env.exe config show remote_x: 
[remote_x]

rclone_v1.57.0-beta.5660.8aa273a9e.remote-name-env.exe config show remote-x: 
[remote-x]
Couldn't find type of fs for "remote-x"

Is there any method to export variable with dash (-) in linux?

The change broken all my scripts which work for long long time.
Before 1.57
hello-world:
export RCLONE_CONFIG_HELLO_WORLD_XXXXX=YYY

1.57
hello-world:
export RCLONE_CONFIG_HELLO_WORLD_XXXXX=YYY (work for hello_world: only, but not work for hello-world: anymore)
export RCLONE_CONFIG_HELLO-WORLD_XXXXX=YYY (cannot use dash on linux)

Hi, welcome to the forum.

Sorry to hear that. The existing behavior had some rather unfortunate side effects, and the tradeoff of fixing those was to break cases like yours..

Seems the env command is a common workaround, can it be used in your case?

env 'strange-name=some value' myprogram