Uploading to GDrive, any way to speed up upload of many small files?


Recently started using GDrive business to have another backup of my important data.

I’m using rclone crypt for the remote. My rclone command is rclone -vvv --bwlimit 10M --tpslimit 10 --low-level-retries 20 sync <local> drive-enc:/<remote_directory> to avoid running into GDrive API or daily bandwidth limits (I am using my own API key)

Problem is that for a test I’m copying my home directory over and I have directories with MANY subdirectories and files, and right now it’s only transferring a few every second which I assume is the GDrive IOPS limit, is there a good way around this, or is this a fact of life when using GDrive?


Try increasing your transfers ˋ–transfers 50` to have more parallel uploads.

I will give that a try and report back

Did you find a fix? I’m having the same problem.

It’s slightly faster, but it’s still not ideal.

Just frustrating having the bulk of my offsite backup slowed down by IOPS and not my upload bandwidth.

If you are doing a backup, you might consider using restic - that can use rclone to upload to drive but it will pack and compress files together so making small file uploads more efficient.

However you may prefer your files uploaded 1:1

I haven’t heard of restic. I was looking into building my own similar solution to pack and compress files together but their are often edge cases you need to watch out for.