[mega] type = s3 provider = Mega ... chunk_size = 16Mi upload_concurrency = 12
Override only this setting for this remote
[remoteXX] ... retries = 50000
The idea would be:
Global defaults automatically apply to all remotes.
Backend-specific globals (like s3_chunk_size) apply only to matching backend types.
Individual remotes can override specific values locally.
Reduce the need to repeat large sets of flags on every command or wrapper script.
From my testing, a literal [global] section does not seem to behave consistently for runtime flags such as vfs_cache_mode, transfers, etc., even though backend-native remote options work correctly.
Is there an officially supported mechanism for this kind of inheritance/config layering?
Hey, thank you for replying, but I already read that, and it’s not quite what I’m looking for because I would still need to configure the same settings for each remote individually.
What I’m looking for is a way to define those settings in a single place so they apply to every remote by default, with the option to override them only in specific remotes when needed.
This enables a "profile" which lets you set command line flags and arguments in the config file.
So you could make a profile with those flags in (you got the syntax pretty much right just change the title to [profile:global] and use --profile global to enable them.
We haven't got a default profile yet, but I was thinking of adding one as [profile:default].
Any testing or comments gratefully received. Look further down the thread for the latest binary - I linked to the instructions.
I'm following this issue, and the profile system is really great. Replacing a lot of parameters with a single profile is already a good trade-off, but it would be even better to have a default profile, so users don't need to specify anything and everything works out of the box. This would also improve compatibility with any rclone GUI wrapper or third-party tools that use rclone internally.
With this, the new priority order would be (lowest wins):
rclone defaults
default profile
[remotes]
environment variables
CLI flags (including profile-related flags)
I'm going to test the beta version over the next few days and let you know how it goes.
Since we’re in contact, while I was testing and trying to figure out on my own whether there was a way to configure default settings for remotes, I tried to make rclone display all parameters being used when executing commands such as mount and copy, in order to verify whether the values were actually being overridden or not.
Does something like this exist? Is there a way to display, in the final output, the parameters that are effectively being used after the entire parameter precedence/order resolution process? And of course, in cases where tokens or sensitive information exist, display them in a redacted form similar to rclone config redacted.
I think you get an idea of the config being applied from the DEBUG messages. Not sure about sensitive stuff - it would be relatively easy to redact it.
Maybe we should do another --dump option to dump the state.
Hey, I've been testing out the profile system, and it's amazing. It's way easier and cleaner, to the point that I'm almost always using the terminal instead of a wrapper GUI.
To make it perfect, I really wish there were a way to specify a default profile for remotes so we wouldn't need to worry about all of those flags. Whether that's a profile with a special name that's automatically applied to all remotes by default, or a profile parameter on each remote where you specify which profile should be used, either approach would work. It would reduce the amount of configuration needed while still making full use of the profile system.
About the debug messages: while using the beta version with verbose mode enabled, I was able to see the parameter overrides happening correctly, which is great and really put my mind at ease that everything is working exactly as it should.
Right now, I'm using --profile default for everything. The Drive, Dropbox, and S3 profiles were mostly just for testing. I don't really see a need for them to be exclusive profiles, but if something like profile:default were to exist, it should probably be applied to all remotes automatically.
[profile:default]
# couldn't find type of fs for "profile:default"
parent_profiles = shared,s3_profile,dropbox_profile,drive_profile
[profile:drive_profile]
# couldn't find type of fs for "profile:drive_profile"
drive_chunk_size = 256M
[profile:dropbox_profile]
# couldn't find type of fs for "profile:dropbox_profile"
dropbox_batch_mode = sync
dropbox_chunk_size = 128M
[profile:s3_profile]
# couldn't find type of fs for "profile:s3_profile"
s3_chunk_size = 128M
s3_upload_concurrency = 8
s3_max_upload_parts = 10000
[profile:shared]
# couldn't find type of fs for "profile:shared"
cache_dir = D:\rclone\cache
vfs_cache_mode = full
vfs_cache_max_age = 48h
vfs_cache_max_size = 100G
vfs_read_ahead = 256M
buffer_size = 64M
checkers = 32
contimeout = 60s
fast_list = true
low_level_retries = 10
multi_thread_streams = 8
retries = 3
timeout = 5m
transfers = 16
unknown flag: --stdio Fatal: unable to open repository at rclone:XXXXXXXXXXX (scaffolded my repo) error talking HTTP to rclone: exit status 2
If I remove --profile default, the problem goes away and everything works correctly.
Am I missing something, or is rclone serve not currently compatible with profiles?
Just letting you know after our discussion: since you're planning to implement the default profile feature, passing something like this to restic may not even be necessary in the future.