I don't think that compressing on-the-fly is the main problem here. There are third-party solutions that does that. Presumably we could fit the compressed version of any single of these small files into memory before upload if we had to in any case, rendering that a largely moot point.
And as we said, compression itself may not even be the issue. The main limiter is the file-access (2-3files/sec on a Gdrive). So just gluing files together into a single larger one would overcome that spesific problem without any compression. The real problems are in the logistics of keeping track of it all (where archives might inherently be of use as they have baked-in indexes). It is absolutely a solvable problem, just more complex than you might initially expect 