Include stuff from other files in the rclone.conf file?

Feature
1

Is there a way to include stuff from other files in the rclone.conf file? I haven't found anything like it in the docs.

What I'd like this for specifically: I have several different machines that I run rclone from. I want them to have all the same remotes configured, but with different credentials (so that I can, for example, tell what machine a file was uploaded from). So it seems like this would be easiest if I could stick all of the common configuration in a file which I store in source control, have it reference a credentials file, and then create specific credentials files for specific machines. This would also have the benefit of not having potentially sensitive stuff (like "secret_access_key") stored in source control, which I'd prefer not to have if possible.

So, I imagine something like this:

Instead of today, where I have an rclone.conf that's something like:

[Dropbox]
type = dropbox
token = ...

[GSuite]
type = drive
client_id = ...
client_secret = ...
scope = drive
token = ...

I would instead something like this in rclone.conf:

#include credentials.conf

[Dropbox]
type = dropbox

[GSuite]
type = drive
scope = drive

And something like this in credentials.conf:

[Dropbox]
token = ...

[GSuite]
client_id = ...
client_secret = ...
token = ...
2

You don't have to store them in the file as you can use environment variables for many things if you want.

3

Thanks, but I'm not understanding. For example, I have two unrelated remotes that both have an "access_key_id" file defined in rclone.conf, and of course the values are not the same in one remote as in the other.

4

Right, so on each system, you can configure an environment variable with the key.

I use a few settings like:

RCLONE_CONFIG=/opt/rclone/rclone.conf
RCLONE_RC_PASS=felix
RCLONE_RC_USER=felix

as an example.

5

I'm sorry, I still think I'm not understanding. I don't mean that I currently have two different config files on two different machines that need different values for the same configuration keys (although that is true). Rather, I mean that on any one single machine, I have two different remotes defined that need different values for the same configuration keys.

For example, I use Amazon S3. It's defined in my rclone.conf as:

[Raw-S3]
type = s3
provider = AWS
env_auth = false
access_key_id = {aki-amazon}
secret_access_key = {sak-amazon}
region = us-east-1
acl = private
server_side_encryption = AES256
storage_class = DEEP_ARCHIVE
bucket_acl = private

The documentation seems to indicate that I could strip out (for example) the access_key_id from the config file, and instead set the environment variable RCLONE_S3_ACCESS_KEY_ID to {aki-amazon}. That's fine in and of itself, but the problem is that I also use Wasabi. It's defined in my rclone.conf -- the exact same rclone.conf, on the exact same machine -- as:

[Raw-Wasabi]
type = s3
provider = Wasabi
env_auth = false
access_key_id = {aki-wasabi}
secret_access_key = {sak-wasabi}
endpoint = s3.wasabisys.com

The docs seem to indicate that rclone also uses the environment variable RCLONE_S3_ACCESS_KEY_ID for the access key ID of a Wasabi remote. So if I set up my RCLONE_S3_ACCESS_KEY_ID to be {aki-amazon} so that I can access my stuff on Amazon, then I will be unable to access my stuff on Wasabi, since Wasabi is expecting my access key ID to be {aki-wasabi}, not {aki-amazon}.

6

You can also set per backend environment variables - see this part of the docs: https://rclone.org/docs/#config-file

Eg to set a variable for a remote called mys3:

export RCLONE_CONFIG_MYS3_SECRET_ACCESS_KEY=XXX
7

Oh, great! Thank you. I'm still a little concerned about something, though:

In some cases, rclone seems to update the config file during normal use (i.e. due to running a regular command, not due to running rclone config). For example, drive remotes have a "token" key in their config, which contains (among other things) a time at which the token expires. If you do something like rclone lsf MyDrive: after the token in the config file has expired, rclone gets a new token and updates the config file with it.

That behavior obviously wouldn't be good when the same config file is used on multiple machines, and even on a single machine it would be a little annoying from the point of view of Subversion (or whatever) constantly telling you that the file has been updated. So if remove the token from the config file and use an environment variable for it instead, does rclone understand that it should update the environment variable rather than the config file? Or, if not, is there some other way to accomplish this sort of thing?

And to be clear, I'm not asking about for "token" specifically - it's just an example for whatever things rclone might update the config file with.

8

Yes rclone does write to the config file - the token needs to be updated regularly.

No it doesn't write back to the environment variable - this wouldn't work at all on unix platforms where environment variables are strictly ephemeral.

I think what you really want is the idea of the split config file, with a read only portion and a read/write portion.

There is an issue about this here: https://github.com/rclone/rclone/issues/3655

However what I'd do in this case is have a template config file which I just copy to the places where it is needed and treat it as a secret. Note that the refresh token in the config will expire eventually if you are using google drive.

It isn't safe, generally, to share config files between machines, but each can have its own copy derived from some master.

I wouldn't check the rclone config file into subversion at all - I'd have another mechanism for making sure that the correct file is in the correct place.

9

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.