Using backend flags in remote's configuration in config file

What is the problem you are having with rclone?

I'm trying to configure per-remote backend flags in config, in particual --s3-chunk-size and --s3-upload-concurrency

What is your rclone version (output from rclone version)

checked with:
rclone v1.50.2 - Ubuntu 20.04
rclone v1.56.2 - release from Github page

Which cloud storage system are you using? (eg Google Drive)

Wasabi S3 in this example

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone -vv -P copy /var/lib/mysql/mysql-bin.001098 wasabis3:company-mysql-backup/blabla/

The rclone config contents with secrets removed.

RCLONE_CONFIG_WASABIS3_S3_CHUNK_SIZE = 6M
RCLONE_CONFIG_WASABIS3_S3_UPLOAD_CONCURRENCY = 1
[wasabis3]
type = s3
provider = Wasabi
env_auth = false
access_key_id = BXXXXXXXXX
secret_access_key = eZZZZZZZZZZZZZZZZZ
endpoint = s3.eu-central-1.wasabisys.com

A log from the command with the -vv flag

./rclone -vv -P copy /var/lib/mysql/mysql-bin.001098 wasabis3:mycompany-mysql-backup/blabla/
2021/10/08 21:03:01 DEBUG : rclone: Version "v1.56.2" starting with parameters ["./rclone" "-vv" "-P" "copy" "/var/lib/mysql/mysql-bin.001098" "wasabis3:mycompany-mysql-backup/blabla/"]
2021/10/08 21:03:01 DEBUG : Creating backend with remote "/var/lib/mysql/mysql-bin.001098"
2021/10/08 21:03:01 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/10/08 21:03:01 DEBUG : fs cache: adding new entry for parent of "/var/lib/mysql/mysql-bin.001098", "/var/lib/mysql"
2021/10/08 21:03:01 DEBUG : Creating backend with remote "wasabis3:mycompany-mysql-backup/blabla/"
2021/10/08 21:03:01 DEBUG : fs cache: renaming cache item "wasabis3:mycompany-mysql-backup/blabla/" to be canonical "wasabis3:mycompany-mysql-backup/blabla"
2021-10-08 21:03:02 DEBUG : mysql-bin.001098: Need to transfer - File not found at Destination
2021-10-08 21:03:02 INFO  : S3 bucket mycompany-mysql-backup path blabla: Bucket "mycompany-mysql-backup" created with ACL "private"
2021-10-08 21:03:04 DEBUG : mysql-bin.001098: multipart upload starting chunk 1 size 5Mi offset 0/1.002Gi
2021-10-08 21:03:04 DEBUG : mysql-bin.001098: multipart upload starting chunk 2 size 5Mi offset 5Mi/1.002Gi
2021-10-08 21:03:04 DEBUG : mysql-bin.001098: multipart upload starting chunk 3 size 5Mi offset 10Mi/1.002Gi

So RCLONE_CONFIG_WASABIS3_S3_CHUNK_SIZE = 6M is ignored.
I've tried to put it inside remote [wasabis3] section, i.e.

[wasabis3]
type = s3
provider = Wasabi
env_auth = false
access_key_id = BXXXXXXXXX
secret_access_key = eZZZZZZZZZZZZZZZZZ
endpoint = s3.eu-central-1.wasabisys.com
RCLONE_CONFIG_WASABIS3_S3_CHUNK_SIZE = 6M
RCLONE_CONFIG_WASABIS3_S3_UPLOAD_CONCURRENCY = 1

But had no difference.

From my PoV, documentation is very blurry on it, without config file snippet example. What I could understand, that in config parsing and environment variables parsing is done via the same mechanics, so variable name export RCLONE_CONFIG_WASABIS3_S3_CHUNK_SIZE should be correct for my case.

Tried to set it via ENV, no luck as well

root@node3 ~/tmp/rclone-v1.56.2-linux-amd64 # env|fgrep -i RCLONE_CONFIG_WASABIS3_S3_CHUNK_SIZE
RCLONE_CONFIG_WASABIS3_S3_CHUNK_SIZE=6M

Still chunk size of 5M was used.

I guess I do misread something, need a working config sample/direction on doing it right.

You can't put environment variables into the config file.

If you run rclone config and just edit the remote, you can change those options and that'll add the right values to your rclone.conf

If you want to use a environment variable, I override my rclone.conf location by dropping this in my .bashrc

# RClone Config file
RCLONE_CONFIG=/opt/rclone/rclone.conf
export RCLONE_CONFIG

As I don't want it stored in the normal spot.

felix@gemini:~$ env | grep RCLONE_CONFIG
RCLONE_CONFIG=/opt/rclone/rclone.conf
felix@gemini:~$ rclone config file
Configuration file is stored at:
/opt/rclone/rclone.conf

For chunk size, you can see what the value is the config file vs environment variable here:

https://rclone.org/s3/#s3-chunk-size

May be I was not clear enough - I wanted to keep all settings in config file, not an environment variables.

I was referring to this Documentation , which says:

Config file
You can set defaults for values in the config file on an individual remote basis. The names of the config items are documented in the page for each backend.
To find the name of the environment variable, you need to set, take RCLONE_CONFIG_ + name of remote + _ + name of config file option and make it all uppercase.

Was very confused by mix of config file & environment variables.

thanks for pointing out, this way works fine:

[wasabis3]
type = s3
provider = Wasabi
env_auth = false
access_key_id = BXXXXXXXXX
secret_access_key = eZZZZZZZZZZ
endpoint = s3.eu-central-1.wasabisys.com
chunk_size = 24M
upload_concurrency = 8

For the Documentation section, it still confuses me and I guess not only me.

Did you click on the link I shared?

image

yes, got variable names from your link and used in solution - Using backend flags in remote's configuration in config file - #5 by CoolCold

I agree the section can easily be misread if you happen to overlook that “Config file” in a subsection of the main section on (the use of) environment variables.

Here is a proposal to make this clearer by modifying the opening sentence to match the opening section under "Options":

Environment variables

Options

Every option in rclone can have its default set by environment variable.

Config file

Environment variables can also be used to set the value of config items for individual remotes in the config file. The names of the config items are documented in the page for each backend.

To find the name of the environment variable, you need to set, take RCLONE_CONFIG_ + name of remote + _ + name of config file option and make it all uppercase.

Do you fancy making a pull request to improve this the way you find best?

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.