i am having issues with the parsing of environment variables.
in this case, remote-x: does not exist, but rclone will use it.
set RCLONE_CONFIG_REMOTE_X_TYPE=local
rclone listremotes
2021/09/15 11:26:49 NOTICE: Config file "b:\\dummy\\dummy.conf" not found - using defaults
remote_x:
rclone ls remote-x: -vv
2021/09/15 11:26:49 DEBUG : Setting --config "dummy.conf" from environment variable RCLONE_CONFIG="dummy.conf"
2021/09/15 11:26:49 DEBUG : rclone: Version "v1.56.0" starting with parameters ["C:\\data\\rclone\\scripts\\rclone.exe" "ls" "remote-x:" "-vv"]
2021/09/15 11:26:49 DEBUG : Creating backend with remote "remote-x:"
2021/09/15 11:26:49 DEBUG : Setting type="local" for "remote-x" from environment variable RCLONE_CONFIG_REMOTE_X_TYPE
2021/09/15 11:26:49 NOTICE: Config file "b:\\dummy\\dummy.conf" not found - using defaults
2021/09/15 11:26:49 DEBUG : fs cache: renaming cache item "remote-x:" to be canonical "remote-x://?/b:/dummy"
1 file.txt
2021/09/15 11:26:49 DEBUG : 2 go routines active
in this case, remote-x: does exist, but rclone will not use it.
set RCLONE_CONFIG_REMOTE-X_TYPE=local
rclone listremotes
2021/09/15 11:27:58 NOTICE: Config file "b:\\dummy\\dummy.conf" not found - using defaults
remote-x:
rclone ls remote-x: -vv
2021/09/15 11:27:58 DEBUG : Setting --config "dummy.conf" from environment variable RCLONE_CONFIG="dummy.conf"
2021/09/15 11:27:58 DEBUG : rclone: Version "v1.56.0" starting with parameters ["C:\\data\\rclone\\scripts\\rclone.exe" "ls" "remote-x:" "-vv"]
2021/09/15 11:27:58 DEBUG : Creating backend with remote "remote-x:"
2021/09/15 11:27:58 NOTICE: Config file "b:\\dummy\\dummy.conf" not found - using defaults
2021/09/15 11:27:58 Failed to create file system for "remote-x:": didn't find section in config file
What is your rclone version (output from rclone version)
rclone version
rclone v1.56.0
- os/version: Microsoft Windows 10 Pro 2009 (64 bit)
- os/kernel: 10.0.19043.1165 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.16.5
- go/linking: dynamic
- go/tags: cmount
Which cloud storage system are you using? (eg Google Drive)
wasabi - s3 clone
The command you were trying to run (eg rclone copy /tmp remote:tmp)
With rclone ls remote-x:, it will look up config for remote remote-x, and when considering environment variables it will look for RCLONE_CONFIG_REMOTE_X_TYPE and never RCLONE_CONFIG_REMOTE_X-TYPE. This also means rclone ls remote-x: and rclone ls remote_x: gives the same result. And rclone listremotes will use the name parsed from environment variable, not translating _ back to - (it wouldn't know which is "more" correct).
The listremotes command considers environment variables starting with RCLONE_CONFIG_ and ending with TYPE, and returns everything between converted to lowercase, as seen from the following code:
Does not answer all your questions, probably, but maybe some?
I guess using - in environment variable names is so inconvenient, perhaps not even properly supported everywhere. If you have a remote remote-x in configfile it is nice to be able to override its config with RCLONE_CONFIG_REMOTE_X_SOMETHING.
Another case: I see that with rclone config, I am able to create two remotes in my config file:
rclone listremotes
remote-x:
remote_x:
Then if both are FTP remotes, I can set the following environment variable to override the username to use:
set RCLONE_CONFIG_REMOTE_X_USER=otheruser
Now it will be considered when using any of the remotes.
So I'm a bit torn what to make of this, or how to improve it in a good way.
Hopefully some thoughts from other developers, perhaps even @ncw. Then maybe create an issue in GitHub. If nothing else then at least some documentation should be added, and in any case it is less likely to just be forgotten there.
This probably explains the code in ConfigToEnv and OptionToEnv, so in this perspective there is a bug in FileSections. Fixing this will however make it impossible to differ between remote-x and remote_x.
I therefore propose a more relaxed approach where the remote name is used as is, and then leave it up to the user to keep remote names within the limits of their environment, e.g. free from "-" if using environment variables in Powershell. This will still allow using both remote-x and remote_x, if you are able to create the environment variables in your environment/shell.
I haven't tried or tested, but I guess it can be achieved with these simple code changes (that include a parameter renaming to make the code easier to understand):
// ConfigToEnv converts a config section and name, e.g. ("myremote",
// "ignore-size") into an environment name
// "RCLONE_CONFIG_MYREMOTE_IGNORE_SIZE"
func ConfigToEnv(option, name string) string {
return "RCLONE_CONFIG_" + strings.ToUpper(strings.Replace(option, "-", "_", -1)+"_"+name)
}
// OptionToEnv converts an option name, e.g. "ignore-size" into an
// environment name "RCLONE_IGNORE_SIZE"
func OptionToEnv(option string) string {
return "RCLONE_" + strings.ToUpper(strings.Replace(option, "-", "_", -1))
}
Do you like it?
Does anybody fancy trying, testing and making the PR?
(I am currently busy with a fix to the pacers handling of "Too many requests")
I was thinking about this as well, but had a concern that rclone was not only "reading" options from environment variables but also "writing" them to environment, and that it then could be issues with - in names not just when user wanted to override options from env. But it seems I was wrong, (don't know where I had it from)... Then I agree, the solution could be as simple as stop mangling with the remote name part of the environment variable names.
I can try it out, and make a PR for it.
PS: I'm a bit confused with your changes in ConfigToEnv:
I do however suggest you also check the naming conventions in the area of the code where it is being called, sometimes the code use a terminology that is a little different from the user guides.
i have a python script that does the following, using the same text name for all operations.
given that rclone still does not accept a ., in a remote name, i am forced to use -
and since rclone ls remote-x: and rclone ls remote_x: gives the same result, i use that workaround
create bucket
create IAM user
create user policy locked to that bucket. require MFA TOTP and session token to access the bucket.
creates a on the fly rclone remote, using MFA TOTP, session token
Support for dot character in remote names
seems to work
set RCLONE_CONFIG_REMOTE.X_TYPE=local
rclone_v1.56.0.exe about remote.x:
Failed to create file system for "remote.x:": config name contains invalid characters - may only contain `0-9`, `A-Z`, `a-z`, `_`, `-` and space
rclone-v1.57.0-beta.5659.cb3703d05.remote-name-dot.exe about remote.x:
Total: 145.986 GiB
Used: 87.399 GiB
Free: 58.588 GiB
Support for hyphen in remote name from environment variable
seems to work
set RCLONE_CONFIG_REMOTE-X_TYPE=local
rclone_v1.56.0.exe config show remote_x:
[remote_x]
Couldn't find type of fs for "remote_x"
rclone_v1.56.0.exe config show remote-x:
[remote-x]
Couldn't find type of fs for "remote-x"
rclone_v1.57.0-beta.5660.8aa273a9e.remote-name-env.exe config show remote_x:
[remote_x]
Couldn't find type of fs for "remote_x"
rclone_v1.57.0-beta.5660.8aa273a9e.remote-name-env.exe config show remote-x:
[remote-x]
-----------------------------
set RCLONE_CONFIG_REMOTE_X_TYPE=local
rclone_v1.56.0.exe config show remote_x:
[remote_x]
rclone_v1.56.0.exe config show remote-x:
[remote-x]
rclone_v1.57.0-beta.5660.8aa273a9e.remote-name-env.exe config show remote_x:
[remote_x]
rclone_v1.57.0-beta.5660.8aa273a9e.remote-name-env.exe config show remote-x:
[remote-x]
Couldn't find type of fs for "remote-x"
Is there any method to export variable with dash (-) in linux?
The change broken all my scripts which work for long long time. Before 1.57
hello-world:
export RCLONE_CONFIG_HELLO_WORLD_XXXXX=YYY
1.57
hello-world:
export RCLONE_CONFIG_HELLO_WORLD_XXXXX=YYY (work for hello_world: only, but not work for hello-world: anymore)
export RCLONE_CONFIG_HELLO-WORLD_XXXXX=YYY (cannot use dash on linux)