Best settings for many small files to Google Drive?

Help and Support
1

I’m using rclone to transfer a few TBs from ‘local’ drive on a G Cloud Windows VM to Google Drive. It was flying (1-2 Gbps) when I was moving large files, but on the thousands of smaller files remaining, I barely get 50Mbps. I guess I’m running up against the G Drive API limits.

I’ve tried 10 at a time, 100 at a time, different timeouts, drive chunk sizes, etc. Also tried using my own client id/key. But nothing seems to help.

I also tried ExpanDrive (abysmal) and native G Drive client (awful).

Has anyone found the optimal settings for this scenario?

FWIW, https://console.cloud.google.com/iam-admin/quotas shows “68,671 / 1,000,000,000” queries for Drive API.

My Queries per 100 seconds keeps spiking at around 600-700 then dropping

You’d think there’d be a way to more directly access G Drive from within a G Cloud VM…

2

My experiments indicated that GDrive doesn’t take files at faster than 2 per second. I think that is a fundamental (undocumented) rate limit somewhere.

2 Likes
3

Interesting. Now that I look at it that way, I am seeing a fairly consistent 3-4 files per second when uploading, regardless of size or anything else.