Google Drive Slow File Creation

Help and Support
1

What is the problem you are having with rclone?

Google drive file copies are very slow, seemingly due to slow file creation speed.

Generally this is not much of an issue but for example one of my containers does a daily backup that is only 1.1GB but 4700 files and it takes a very long time to complete the backup (around 1 hour)

I created a very basic script to create X number of files with Y file size and then delete those files.

Here are the results:
Files Created : 100
File Size : 1 KB
Time to Create: 60.50 seconds (1.65 files/sec)
Time to Delete: 11.72 seconds (8.53 files/sec)
Total Time : 72.22 seconds

Just to show that upload speed is not an issue here is a test of 1x500MB file:

Files Created : 1
File Size : 500000 KB
Time to Create: 13.74 seconds (0.07 files/sec)
Time to Delete: 1.35 seconds (0.74 files/sec)
Total Time : 15.09 seconds

In case its helpful my average ping to google.com is 5.6ms (server is colocated in a datacenter)

Run the command 'rclone version' and share the full output of the command.

rclone v1.69.1

  • os/version: ubuntu 20.04 (64 bit)
  • os/kernel: 5.4.0-216-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.24.0
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy /tmp/rclonetesting gdrive-crypt:rclonetesting

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[gdrive]
type = drive
client_id = XXX
client_secret = XXX
scope = drive
token = XXX
team_drive = XXX
root_folder_id =

[gdrive-crypt]
type = crypt
remote = gdrive:
password = XXX
password2 = XXX

A log from the command that you were trying to run with the -vv flag

https://privatebin.net/?145a241ec5f2b6a1#GuZmXbncHnGAwDxXwe6uvQ54ikjez6tpjEn9PnomV1G7
2

It is well known characteristic of Google Drive. You will find it discussed on this forum many times in the past.

You can try to improve it by increasing number of transfers:

rclone copy /tmp/rclonetesting gdrive-crypt:rclonetesting --transfers 8

But if set too high it can lead to throttling errors. Experiment a bit to find the best value for you.

3

Appreciate the reply!

Is this affected by tps-limit? Is there a generally recommended/safe tps-limit to use with google drive?

4

I stopped using Google Drive long time ago. I think if you search on this forum you will find some recommendations

5

There's really no way around it.

Google's API limits file creation to 1-3 a second so doing a lot of small files has been and unless something changes, awful.

6

Alright fair enough! I guess I could always start compressing everything locally and then transferring 1 zip instead.

1 Like
7

Hi,

I am also unsatisfied with my Google Drive. Rather than connecting with the default client ID and Secret I created my own client ID and Secret in Google and that increases the speed somewhat. You might want to look into trying that.

Here are instructions for creating the Google Drive client ID and Secret:

8

OP already has client ID/secret and that doesn't impact file creation limits.

9

@LeoW, hi, good suggestions

afiak, the OP is already using client_ id + client_secret

from the rclone docs
Making your own client_id

10

Ok,

In that case there's really only one other option and that is to move to a different Cloud Server that offers better performance.

I have a proton account and it is extremely fast. I also have a mega Cloud Server account and it's pretty fast too. I've also used Microsoft azure to create a client ID and secret for Microsoft OneDrive and that works really fast now too.

1 Like