Uploading to GDrive, any way to speed up upload of many small files?

Help and Support
1

Hi.

Recently started using GDrive business to have another backup of my important data.

I’m using rclone crypt for the remote. My rclone command is rclone -vvv --bwlimit 10M --tpslimit 10 --low-level-retries 20 sync <local> drive-enc:/<remote_directory> to avoid running into GDrive API or daily bandwidth limits (I am using my own API key)

Problem is that for a test I’m copying my home directory over and I have directories with MANY subdirectories and files, and right now it’s only transferring a few every second which I assume is the GDrive IOPS limit, is there a good way around this, or is this a fact of life when using GDrive?

Thanks

2

Try increasing your transfers ˋ–transfers 50` to have more parallel uploads.

3

I will give that a try and report back

4

Did you find a fix? I’m having the same problem.

5

It’s slightly faster, but it’s still not ideal.

Just frustrating having the bulk of my offsite backup slowed down by IOPS and not my upload bandwidth.

6

If you are doing a backup, you might consider using restic - that can use rclone to upload to drive but it will pack and compress files together so making small file uploads more efficient.

However you may prefer your files uploaded 1:1

1 Like
7

I haven’t heard of restic. I was looking into building my own similar solution to pack and compress files together but their are often edge cases you need to watch out for.