I originally dreamed up my problems 3 years ago, but I had heard of google cloud storage and I believe before rclone supported gzip?
https://forum.rclone.org/t/i-think-rclone-needs-some-sort-of-inverted-chunker-style-overlay/40550/7
I am wondering, if gzip will not work for my purposes at all. Maybe restic will? The issue is even if restic can group small files into bigger chunks to better fit s3 storage…. what happens to the metadata of those small files with restic?
Then there is rclone’s own compress features. Compress but it still marked as experimental. It has been experimental as far as I know for a couple years, so I am not sure it is an ideal fix for me either. Nor how will rclone’s own compress would achieve my goals for s3. I could try to compress every folder, BUT rclone compress appears to be about compressing files. NOT about fake compressing folders to generate metadata files for those folders, so that the folders metadata can be labeled hot and the folder’s contents can be labeled cold.
Compress does allow zero gzip compression which I assume is what I want and good for speed “- 0 — turns off compression.”
But again, what well… I’ll just ask it.
Compress would become TOTALLY FLAWLESS and AMAZING if I could tell it to compress every folder, place the compressed folder into cold s3 storage and then place the metadata file into hot s3 storage. That would work like a dream. It’s something I’ve wanted for years. But it seems slightly out of reach for me right now.
Again unless anyone knows what I mean and already has this solution working for themselves?