Store config in memory

Feature
1

I'm using rclone as a stateless service, meaning that before every file upload or download request, I ensure that the necessary configuration is available. This configuration is primarily stored in a dedicated, secure, and consistent database table.

I think it would be beneficial if rclone offered an option to store its configuration in memory instead of on the filesystem. This could potentially improve performance.

WDYT?

2

what about a ramdisk, sort of a compromise?

with my backup solution, in most cases, no config files is created or used.
the script create an on-the-fly remote using environment variables.

also, can use connections strings, tho that can leak private info

here is an example using environment variables.

RCLONE_CONFIG_REMOTE_TYPE=s3
RCLONE_CONFIG_REMOTE_ACCESS_KEY_ID=redacted
RCLONE_CONFIG_REMOTE_SECRET_ACCESS_KEY=redacted
RCLONE_CONFIG_REMOTE_ENDPOINT=s3.us-east-2.wasabisys.com

rclone lsd remote:
1 Like
3

This is very wild speculation:) Do you realise that rclone does not read config file 1000 times a second but only once?

But if you try as suggested RAM disk and do some testing showing what are these potential performance improvements than we can talk:)

4

what about a ramdisk, sort of a compromise?

with my backup solution, in most cases, no config files is created or used.
the script create an on-the-fly remote using environment variables.

also, can use connections strings, tho that can leak private info

At least my use case of rclone is a stateless container in kubernetes. So, usage of ramdisk or dynamic environment variables isn't really an option for me, unfortunately.

But thanks for the good ideas.

5

This is very wild speculation:) Do you realise that rclone does not read config file 1000 times a second but only once?

But if you try as suggested RAM disk and do some testing showing what are these potential performance improvements than we can talk:)

I wouldn't call it "speculation" it's too strong word for me, but could call it like "conjecture" or smth like that.

I understand that rclone reads configuration each time it have a request to some remote file, which is fine until you have a lot of requests per second and poor HDD disk :slight_smile: I'm a bit of exaggerating my case, I mean I don't really use a HDD disks, but there's no limit for the perfect, right? :slight_smile:

Also, I don't say that it's must have feature, I'm more then happy about current rclone perfomance, just had an idea and wanted to share it with you guys .:slight_smile:

6

If you are using any sort of modern OS then most likely subsequent reads come from memory cache anyway. Adding extra complexity to "speed up" reading few hundreds bytes from a filesystem is IMO not worth of any effort.

What is true is that multiple independent rclone invocations create substantial overhead (new connection has to be established and authenticated every time etc.). If you really care about squeezing as much performance as possible then this is sth you should try to optimise. The most straightforward solution is to use rclone mount. A bit more complex but with much greater control is RC interface.

7

I'm currently using rcd (rc server) and its API, essentially developed a client for it and using it in my service.

Regarding the operating system and read cache, thank you for the tip - I’ll definitely look into it. This is the first time I've encountered caches at that level. Hopefully it works within containers. :slight_smile:

1 Like