I observed lackluster performance when transferring larger files (32 x 1 GiB) using the Google Cloud Storage backend (type = google cloud storage) JSON interface as compared to using the S3 backend with GCS provider (type = s3, provider = GCS) XML interface. I was surprised that the S3 backend was 7x faster!
Part of the improvement comes from XML multipart uploads, but taking that into account there still seems to be limited concurrency when using the google cloud storage backend. I am not proficient in Golang, but checking rclone's source I believe it uses legacy auto-generated libraries (googlecloudstorage) and not the optimized ones published by Google (cloud.google.com/go/storage).
Would there be interest in refactoring or updating the native GCS backend to use it? I suspect updating rclone to use the optimized ones would improve performance dramatically.
Or alternatively, adding other auth mechanisms (service account keys, application default credentials) with the s3 and GCS provider?
Bechmark Tests:
I ran tests from a n4a-standard-8 (8 ARM vCPUs, 32 GiB RAM) against a regional co-located bucket and copied 32 x 1 GiB files from one prefix to another in the same bucket. I disabled server-side copies as the goal is to test throughput of the client and not offload of the server.
rclone.conf
[gcs]
type = google cloud storage
anonymous = false
bucket_policy_only = true
[gcs-s3]
type = s3
provider = GCS
endpoint = storage.googleapis.com
access_key_id = XXXX
secret_access_key = XXXX
I wrote a small bash wrapper that first generates a file list and then runs up 16 concurrent rclone processes on a split of that list. With process level concurrency I could achieve 20 to 25 seconds elapsed time using the gcs or gcs-s3 configuration. This test further supports the liklihood of concurrency limitations in the google cloud storage provider.
I suspect the difference in speed is entirely down to the multipart chunk writer which s3 backend supports but gcs does not.
What that will mean in a network to network copy is that rclone will copy the chunks in parallel from source to destination. Without that the source file is effectively read sequentially before being uploaded sequentially.
You can test this theory by using --disable OpenChunkWriter on the gcs-s3 to gcs-s3 copy which will force it back to sequential reading. The uploads will still be parallel but since the chunks have to be read sequentially over the network first I suspect this will replicate the slowdown probably to the same as the gcs -> gcs-s3 level.
The gcs backend doesn't support OpenChunkWriter (the big win) or Multpart uploads so it is definitely lacking in concurrency.
Last time I looked at this (quite a few years ago) the cloud.google.com/go/storage were just a thin wrapper over the auto generated ones, but that seems to be different now, so this is definitely worth a look. We currently use google.golang.org/api/storage/v1 which is officially deprecated so it would be nice to move off that.
You'll probably find that using --transfers 64 will have the same effect. The usual advice is increase --transfers until you run out of bandwidth or CPU.
I was testing rclone in the azure cloud recently and it was comfortably filling 20Gbit/s pipes using a couple of Cores of CPU only, so we should be able to get the same performance for GCS.
With DEBUG logging I noticed in the s3 backend example a log message at the end of the transfer about 260 go routines active, but with the gcs backend it only says 4 go routines active. Is this an interesting clue?
