What is the best method to upload many small files?

Hi,

I have about two million, but small files that I don’t have locally so I have to wget them first using a script from another server. Now I directly started downloading them to the mount but I kept getting ratelimit exceeded errors on some files. What is the best way to gain the smallest loss and upload them onto an encrypted storage?

Thanks

The best way would be to bundle them up before you send them.

At most you can do 10 files per second with current Google API limits so that’s like 55 hours if my math is right.

I did 4 per second yet I got the ratelimit exceeded error like once per five minutes.