Issue:
I observed lackluster performance when transferring larger files (32 x 1 GiB) using the Google Cloud Storage backend (type = google cloud storage) JSON interface as compared to using the S3 backend with GCS provider (type = s3, provider = GCS) XML interface. I was surprised that the S3 backend was 7x faster!
Part of the improvement comes from XML multipart uploads, but taking that into account there still seems to be limited concurrency when using the google cloud storage backend. I am not proficient in Golang, but checking rclone's source I believe it uses legacy auto-generated libraries (googlecloudstorage) and not the optimized ones published by Google (cloud.google.com/go/storage).
Would there be interest in refactoring or updating the native GCS backend to use it? I suspect updating rclone to use the optimized ones would improve performance dramatically.
Or alternatively, adding other auth mechanisms (service account keys, application default credentials) with the s3 and GCS provider?
Bechmark Tests:
I ran tests from a n4a-standard-8 (8 ARM vCPUs, 32 GiB RAM) against a regional co-located bucket and copied 32 x 1 GiB files from one prefix to another in the same bucket. I disabled server-side copies as the goal is to test throughput of the client and not offload of the server.
rclone.conf
[gcs]
type = google cloud storage
anonymous = false
bucket_policy_only = true
[gcs-s3]
type = s3
provider = GCS
endpoint = storage.googleapis.com
access_key_id = XXXX
secret_access_key = XXXX
Command options
--transfers=16 --checkers=32 --disable=copy,move --buffer-size=64M --s3-chunk-size=64M --disable-http2
Performance results
| Source | Destination | Elapsed Time | Throughput | CPU Usage |
|---|---|---|---|---|
gcs-s3 |
gcs-s3 |
21.3s | 1.509 GiB/s | ~600% |
gcs-s3 |
gcs |
1m 06.8s | 496.48 MiB/s | <100% |
gcs |
gcs-s3 |
1m 31.5s | 470.02 MiB/s | 100% - 300% |
gcs |
gcs |
2m 27.5s | 244.18 MiB/s | <100% |
Workaround: Use multiple rclone processes
I wrote a small bash wrapper that first generates a file list and then runs up 16 concurrent rclone processes on a split of that list. With process level concurrency I could achieve 20 to 25 seconds elapsed time using the gcs or gcs-s3 configuration. This test further supports the liklihood of concurrency limitations in the google cloud storage provider.