Read RCLONE_CONFIG_PASS from a URL

Feature
#1

I got nervous about storing all of my credentials and encryption keys on a remote server and decided to encrypt my rclone config file. At boot time I mount an encfs volume to /home interactively, so if the machine ever loses power everything is safe, in theory; however, as an exercise in paranoia I wanted to protect against someone with physical access to the server.

Encrypting the config file created a bit of a problem because I need rclone to be able to run from scripts and have systemd keep an endpoint mounted, but storing the password on the server would defeat the purpose of encrypted the config file.

The server is running Ubuntu, so one option I considered was writing it to /dev/shm, but I would image that someone with physical access to the machine could easily read that file and then have free-run of my encrypted files stored in the cloud. I already use gpg-agent for SSH, but I have found it to be a royal PITA to get working properly in scripts, so I don't consider that a viable option. Ditto for PAM-based solutions. Those are also difficult to roll up into the simple deployment scripts I use when I migrate servers.

The solution I landed on was to have the scripts set the RCLONE_CONFIG_PASS environmental variable by calling a URL that reads the password off of my server at home over an HTTPS connection that only allows the IP address of my remote server.

I then can easily shut off that URL, denying the remote server access to the password and, in theory, protecting the contents of my config file in response to any suspicious activity on the server.

This solution has worked very well so far. It is relatively simple and portable, but is also a bit cumbersome because I have to wrap everything rclone-related in a shell script to fetch the password and set RCLONE_CONFIG_PASS.

Would it be possible to have a switch like --rclone-config-pass-url that allows rclone to fetch the password directly? Or is there, perhaps, an easier way of separating the config password from the server for non-interactive tasks?

#2

There us a pull request open for a --config-pass-command which would use the stdout of the command. I think you could then make the command curl http://... which would satisfy this requirement.

What do you think?

#3

For my purposes, that is no different than setting an environmental variable because it still makes rclone dependant on cURL.

Perhaps this is a misuse of encrypted rclone configs (or a lack of understanding systemd), but an example is keeping an endpoint mounted via systemd. If I set RCLONE_CONFIG_PASS when I start the service, everything is fine at first, but the endpoint eventually unmounts and the log file fills up with rclone complaining that the config is encrypted before systemd kills it entirely for restarting too quickly.

Thus, I have to point systemd at an external script that calls cURL to setup the environment to call rclone mount. If rclone were able to fetch the password directly from a URL (or a key manager or whatever) directly, it would obviate the external script because systemd could just call rclone directly.

Ultimately, it is really a minor inconvenience, but I could imagine that making it easier for rclone to grab a complex password from a URL / key manager / whatever might make encourage people to use encrypted config files.

edit: I suppose something like ExecStart=bash -c 'rclone --config-pass-command $(curl -sf URL)' might work, but having rclone actually fetch the URL would also allow for a non-zero return value when curl failes, which is what the script that ExecStart currently points to takes care of. Otherwise systemd just keeps restarting the service forever.

#4

Hmm, I could add another flag to rclone copyurl say --stdout very easily so you could use rclone for this!

Wouldn't you just use rclone --config-pass-command "curl -sf URL"? Rclone would check the exit status of the curl when it ran it?

#5

If copyurl could just dump the output of a URL to stdout so that --config-pass-command could read it, that would be ideal. However, even if you have to run rclone twice, once to dump the URL to stdout and once to read it, it would be better than relying on cURL, since it is not always installed in lightweight and embedded OSes.

For example rclone copyurl URL --stdout | rclone mount --config-pass-command - would work right out of the box on just about anything rclone runs on.

This is where my knowledge of systemd is a problem. If ExecStart=rclone mount --config-pass-command $(curl -sf URL) or can be invoked without bash -c in front of it, then systemd would get the exit code of rclone; if rclone indeed passes the exit code of curl, then everything is fine. But with bash -c in front of it, I'm pretty sure systemd would get the return code of bash instead of rclone or curl. That is also pretty *nix-dependent; I think the aforementioned solution of piping copyurl to --config-pass-command would be much more portable.

If rclone mount --config-pass-command $(rclone copyurl --stdout) or rclone copyurl URL --stdout | rclone mount --config-pass-command - were able to produce non-zero exit codes when either (both?) rclone invocations fail, I think that would solve all of my problems!

#6

BTW this is just another example of how rclone is slowly replacing many, many other tools for mounting, backing up, verifying, and otherwise maintaining files spread across all the different services and computers I use for work and fun. I very gratefully donate (via bitcoin) when I can, since my go skills (and free time) leave something to be desired.

#7

I had a few moments to add --stdout to rclone copyurl

https://beta.rclone.org/branch/v1.50.2-091-g0cdf4ca6-fix-copyurl-stdout-beta/ (uploaded in 15-30 mins)

The intention with

rclone mount --config-pass-command "rclone copyurl --stdout http://example.com"

would be that rclone runs the --config-pass-command very early in startup and if it fails it will exit with a non zero exit code.

So I don't think you need systemd to execute it with the $(rclone ...) syntax

Much appreciated - thanks!

#8

hi,
so if someone had access to the script, they would see that the rclone password is stored in a webpage and then goto that webpage and see the password.
is that correct?

#9

Yes, but the URL is only stored inside encrypted volumes, so someone would have to gain access to the server while the encrypted volume was mounted to read the URL. The actual password is only stored on a server that is in my home that blocks access in response to a tripwire on the remote server and only serves to one IP address, over a Wireguard tunnel. The only other way I could figure out how to automate rclone with an encrypted config was using a key agent that unlocks with a password (like ssh-agent), but I found it very difficult to get working reliably and I'm not sure it offers any additional protection because someone with physical access would still be able to decrypt the config file.

With the setup as I have it, assuming I can disable the URL in time, the only way to decrypt it would be brute force, even if someone were able to crack the encrypted volume that the config file is stored on.

I'm no security expert, so I'm sure there are better ways to protect my encrypted volume keys, but this is the best solution I could come up with that works reliably.

#10

Oh, I misunderstood what --config-pass-command does; it actually executes a command and then reads from stdout. That is perfect!

I will give this a try when I have time --- thanks!

EDIT: The --stdout switch works perfectly, but I think I have to wait until the --config-pass-command pull request gets merged.

1 Like
#11

ok, that sounds reasonably secure.

here is what i do.
i have a python script that encrypts the rclone password.

i have another python script that:

  1. decrypts the rclone password.
  2. copies the rclone password to the windows registry.

i have another python script that:

  1. runs a batch file that reads the rclone password from windows registry and sets RCLONE_CONFIG_PASS
  2. deletes the windows registry entry
  3. runs rclone
1 Like
#12

Since rclone is also running periodically via cron jobs and whatnot on a remote server, I wanted a way to separate the password physically from the server. For my use case, encrypting the password wouldn't add any security since the decryption key would need to be stored on the server anyway.

The --config-pass-command solution above should work for both of our use cases though, since you can just have a single script decrypt the password and dump it to stdout, right?

#13

that does not apply to me, as there is no decryption key needed.
the python script that encrypts and decrypts the rclone password is a self-contained algorithm.
it is a custom three-step encryption algorithm that uses ROT and then XOR on the output of the ROT and then take the output of the XOR and reverses it.

so someone would have to have access the source code of the python script, and that script is not stored on the server.

i use pyinstaller to convert the script to a .exe
then i use UPX to compress/scramble the .exe

so to get the rclone password someone would have to

  1. know that i scrambled the .exe with UPX and somehow unscramble it to get the original .exe
  2. take the unscrambled .exe and de-compile it back to python code.
  3. then you would have to find the encrypted rclone password.
#14

My method works well for me because I already have a bunch of servers spread around connected via secure tunnels, so it's easy just to fetch things over them, but I'm curious: How to you run automated / non-interactive rclone jobs without leaving the compiled decryption executable on the server?

For instance, I can keep an rlone mount going by setting / unsetting RCLONE_CONFIG_PASS, but when an rclone job spools up to sync a backup or move some files around, it needs access to the password again.

Perhaps the proper way to is to run an rclone rcd once and then pass commands to it instead of calling rlcone... but that would require rewriting a ton of scripts.

#15

the compiled .exe is on the server.

the encrypted rclone password is not stored in a file, whether locally or web server, but it is hidden deep inside the windows registry on that same server.

in addition, on the windows computer, i create a user
that user:

  1. is not able to login.
  2. using bitlocker, i add yet another layer of encryption to encrypt
    a. the already compressed/scrambled/compiled .exe
    b. the underlying windows user profile
    c. all user files
    d. entire registry that stores the already encrypted rclone password.
    e. the scripts that run rclone are run non-interactively via task scheduler.
#16