I'm working in a startup and we have microservices architecture, there are several worker servers which based on request download files from a shared s3 storage, process it, etc.
My goal is to only download files once per worker server and keep the files locally a well, current implementation is to have a directory on the disk and check if it's exists there first before downloading it.
it works now, but there are several problems:
- How much disk space is used (we can afford around 1TB local disk), not trivial to check size constantly.
- What if two different processes (requests) try to download same time and overwrite each other? etc.
I'm using rclone from personal goals for years now with s3 as a backend + crypt.
I was thinking to mount s3 storage as a local dir, we control cache size, expiration, etc. and just access files from the directory instead of directly using s3.
What you think is it good idea to use rclone in production in such way?