Rclone.conf symlink destroyed

Suspected Bug
1

What is the problem you are having with rclone?

Every time the rclone.conf file is updated (even for refresh tokens) it is deleted and re-written instead of simply opened for writing. This causes a problem if it is a symlink to a file elsewhere. Using a symlink since I am saving the configuration to yadm (dot-files manager) with alternatives for different machines.

This was previously reported in Config breaks rclone.conf symlinks but the only tip there was to symlink the folder, not the config. However, alternative directories don't work as nicely as individual files.

Run the command 'rclone version' and share the full output of the command.

rclone v1.57.0-DEV

  • os/version: fedora 34 (64 bit)
  • os/kernel: 5.17.5-100.fc34.x86_64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.16.11
  • go/linking: dynamic
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive and others.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Any command that could indirectly recreate rclone.conf. In particular rclone config ... will definitely do if you make any changes.

The rclone config contents with secrets removed.

[google-drive]
type = drive
client_id = ...
client_secret = ...
token = ...
export_formats = desktop
team_drive = 

[moravian-google-drive]
type = drive
client_id = ...
client_secret = ...
token = ...
export_formats = desktop

[dropbox]
type = dropbox
token = ...
3

On Linux you can symlink $HOME/.config/rclone to your managed area. I use this for shared NFS where multiple machines may use the same config.

FWIW, this type of rewrite is common for applications that do updates. I don't know what rclone does (I haven't looked) but when I write programs I create a new file with a temp name, and verify the contents are correct, and then move the file over the original. Typically may breaks symlinks (it definitely breaks hardlinks). But it ensures you never end up with a corrupted file!

4

@sweh seems to be right (I looked at the code). And as I said, I could symlink the directory, but with the rest of my system that is more obnoxious than symlinking a file.

It seems like the code responsible for this behavior is in the fs.config.configfile.Save() function and goes through the following steps:

  1. Saves the new config to a temporary file
  2. Renames the old config to ".old"
  3. Renames the temporary file to the real config file
  4. Removes the old config

The rename operations there are causing the problem for the symlink (steps 2 and 3). To keep all the safety benefits of the current system, the renames cannot be entirely replaced by copies (specifically step 3 cannot).

The best solution (if rclone wanted to add this "feature" / fix this "bug") would be to acquire the real path to the config file at rclone/configfile.go at 72227a01519f4cc9b0a1ecd390785c9d1d09d5d0 · rclone/rclone · GitHub like:

configPath := Realpath(config.GetConfigPath())

(note: I have never programmed in Go before, I don't know if you can have two function calls on a line like that)

The Realpath() function/package itself is available in Go here, looks pretty straightforward, and has an MIT license like rclone.

For adding a single package and changing a single line of code it seems like this should be doable!

5

Why not ditch the symlink and just point to the location of the rclone.conf ?

I use an environment variable as I keep mine in a custom location so no need for any links.

6

Yeah, I agree. If you can't symlink directories then use --config in your invocations to point to your managed area (which can be defined with a variable if necessary).

7

Using yadm, I can have files like rclone.conf##hostname.machine1 or rclone.conf##os.Darwin or lots of other things and it automatically creates the symlink then to the correct one. Otherwise, I would have to integrate that logic into every other place instead of just one place. I would do the directory symlinking first.

Ultimately, I hope that 2 lines of code aren't that big of a deal for them to add...

8

@coder-for-life do you want to send a PR implementing this?

I'd probably detect if the config was a symlink and read the link rather than take another dependency.

9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.