Issue with config file - rclone is deleting and misformatting text

Help and Support
1

What is the problem you are having with rclone?

  1. rclone is deleting text from config file.
  2. rclone is misformatting remotes from config file.

same behavior on windows and linux
same behavior v1.56.2 + v1.57.0

steps to reproduce

  1. create a config file with this text
[01]
type = alias
remote = 01
#01

[02]
type = alias
remote = 02
#02
  1. using rclone config, create a new remote and the updated config file will look like
[01]
type = alias
remote = 01

#01
[02]
type = alias
remote = 02

[03]
type = alias
remote = 03
  1. notice that:
    --- #01 has moved from bottom of [01] to top of [02]
    --- #02 from [02] has been deleted

What is your rclone version (output from rclone version)

rclone v1.56.2
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.10.60.1-microsoft-standard-WSL2 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.16.8
- go/linking: static
- go/tags: none

rclone v1.57.0
- os/version: Microsoft Windows 10 Pro 2009 (64 bit)
- os/kernel: 10.0.22000.318 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.17.2
- go/linking: dynamic
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

none

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone config --config=comment.conf

The rclone config contents with secrets removed.

see above

A log from the command with the -vv flag

rclone config --config=comment.conf -vv
2021/11/12 18:53:40 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "config" "--config=comment.conf" "-vv"]
2021/11/12 18:53:40 DEBUG : Using config file from "C:\\data\\rclone\\scripts\\comment.conf"
Current remotes:

Name                 Type
====                 ====
01                   alias
02                   alias

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> n
name> 03
Option Storage.
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value.
Storage> alias
2021/11/12 18:53:46 DEBUG : 03: config in: state="*all", result=""
2021/11/12 18:53:46 DEBUG : 03: config out: out=&{State:*all-set,0,false Option: OAuth:<nil> Error: Result:}, err=<nil>
2021/11/12 18:53:46 DEBUG : 03: config: reading config parameter "remote"
Option remote.
Remote or path to alias.
Can be "myremote:path/to/dir", "myremote:bucket", "myremote:" or "/local/path".
Enter a string value. Press Enter for the default ("").
remote> 03
2021/11/12 18:53:47 DEBUG : 03: config in: state="*all-set,0,false", result="03"
2021/11/12 18:53:47 DEBUG : Saving config "remote" in section "03" of the config file
2021/11/12 18:53:47 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2021/11/12 18:53:47 DEBUG : 03: config out: out=&{State:*all,1,false Option:<nil> OAuth:<nil> Error: Result:}, err=<nil>
2021/11/12 18:53:47 DEBUG : 03: config in: state="*all,1,false", result=""
2021/11/12 18:53:47 DEBUG : 03: config out: out=&{State:*postconfig Option:<nil> OAuth:<nil> Error: Result:}, err=<nil>
2021/11/12 18:53:47 DEBUG : 03: config in: state="*postconfig", result=""
2021/11/12 18:53:47 DEBUG : 03: config in: state="", result=""
2021/11/12 18:53:47 DEBUG : 03: config out: out=<nil>, err=<nil>
2021/11/12 18:53:47 DEBUG : 03: config out: out=<nil>, err=<nil>
2021/11/12 18:53:47 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
--------------------
[03]
type = alias
remote = 03
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
2021/11/12 18:53:48 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
Current remotes:

Name                 Type
====                 ====
01                   alias
02                   alias
03                   alias

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q
2021/11/12 18:53:50 DEBUG : rclone: Version "v1.57.0" finishing with parameters ["rclone" "config" "--config=comment.conf" "-vv"]
1 Like
2

I can confirm problems reported here.

It stems from the ini parser library GitHub - unknwon/goconfig: Package goconfig is a fully functional and comments-support configuration file (.ini) parser. internally called by rclone.

The parser relocates blank lines around the first and last comments in sections.
What's worse, it drops comments after the last key in the last section, right at the file end.

The former quirk is not generally a problem at all.
The last one used to break one of my workflows where I leveraged ansible to assemble rclone configurations on multiple remote hosts from the central fact inventory. I used the ansible blockinfile mechanics for upgrading rclone configs, which produced layouts like

# BEGIN development remotes
[s3dev1]
key = val
[s3dev2]
key = val
# END development remotes

rclone dropped the last block end marker confusing the ansible block parser and forcing undefined behavior. However, I solved it by forcing a dummy section at the end of file

# BEGIN production remotes
[s3prod1]
key = val
# END production remotes
[dummy-eof-section]

Unfortunately the upstream library has not been updated for a while. Probably we can evaluate the succesdor library GitHub - go-ini/ini: Package ini provides INI file read and write functionality in Go from the same author to check whether it handles comments any better.

3

I believe "unknown" values are left untouched, so you possibly could just do comment = this is a comment

4

so i tried that, ran into the same problem.

# BEGIN production remotes
[s3prod1]
key = val
# END production remotes
[dummy-eof-section]

became

# BEGIN production remotes
[s3prod1]
key = val

# END production remotes
[dummy-eof-section]
5

It does not change the rclone behavior for me. I think an extra blank line is not a problem as long as significant lines stay in place. To prevent ansible from detecting a change due to extra blank lines I inserted them in advance in templates to match rclone config parser quirks.

6

hi, i did some testing, and that is not a solution.

given this config file

+ cat /home/user01/rclone/scripts/configtest/comment.conf
[01]
type = alias
comment = middle
remote = remote
comment = end

[02]
type = alias
remote = 02

for rclone config show

+ rclone config show 01 --config=/home/user01/rclone/scripts/configtest/comment.conf
--------------------
[01]
type = alias
comment = end
remote = remote
--------------------

for rclone config update

+ rclone config update 01 remote test --config=/home/user01/rclone/scripts/configtest/comment.conf
--------------------
[01]
type = alias
comment = end
remote = test
--------------------
+ cat /home/user01/rclone/scripts/configtest/comment.conf
[01]
type = alias
comment = end
remote = test

[02]
type = alias
remote = 02
7

This section has two values with the same key. Which one will be used is indeterminate. Current parser will eat one of duplicates without a warning. Agreed, it's a poor practice.

8

ok, good point

in the end, i agree, the issue is with the third-party .ini library and its quirks.
once known, that can be worked around.

thanks,

9

I would be grateful if you submitted a bug (or enhancement, to your taste) report on github.

10

i took some time to think this over.

in many ways, the rclone.conf file is the most vital part of rclone.

i do not want to change the current functionality.

imho, not worth the risks over some small quirks, that once known as be worked around.

thanks

11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.