Before bothering to raise a Github feature request I thought I'll raise it here - I notice that as gdrive team drives become more popular that more people are hitting the 4k-files limit.
My suggestion is a feature where it is basically the reverse of the new chunker.
A remote could be populated by a abstraction layer, which can create several large "files" (containers?) on the remote which within them can contain a allocation-table (allocation files?) and map to position points within said huge "files" to the actual content being queried.
This would give the ability to store a significantly larger amount of files than what is artificially being limited.
The concerns are of course that this is probably excessively expensive to both pre-populate the larger files/containers, and expensive operations to retrieve data (having to look into a allocation-section, then reference correct range in another object).
Thanks, interesting, I did not realise updating a file is such a pain.
I was leaning towards a solution chunking/containering it on the remote to very large files (~100gb?) to essentially get rid of those (number of) file limits. I suspect 64k-1M 'chunks' will quickly get to 4K files for most uses.
The latter item about ext4 is a interesting idea - lots of ext4 images on a remote, somehow loopback mounted and striped - sounds like dangerous and slow fun
PS: Just re-read, its 400k file limit for gdrive team drives, still a issue but not as constrained
Hmm, yes... 400k files * 1MB is 400GB so big but not massive. Increasing the chunk size is possible of course but if you are running (ext4 say) on it then you'll spend a lot of time downloading the 1MB chunks to update 512 bytes in them before uploading them again.
Not being able to update existing files is a pain! You could use a log structured file system to work around it but then you'd need a compaction phase at some point which would be pretty expensive in terms of network operations.