Optimize upload to GCS bucket

Help and Support
1

What is the problem you are having with rclone?

While uploading a file to GCS rclone appears to not use the full bandwidth.
I wonder if there are some flags or command line options to improve the upload speed.

Run the command 'rclone version' and share the full output of the command.

rclone v1.65.0

  • os/version: Microsoft Windows 10 Pro 22H2 (64 bit)
  • os/kernel: 10.0.19045.3693 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.21.4
  • go/linking: static
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

GCS - Google cloud storage

The command you were trying to run

rclone copy ./VM196S003012-20220815_123255402.mp4 gcs_remote:upload-bucket-useast1 --gcs-service-account-file srvc_key.json --config rclone1.conf --gcs-no-check-bucket -P

The content of rclone1.conf:

[gcs_remote]
type = google cloud storage
service_account_file = srvc_key.json
anonymous = false

A log from the command that you were trying to run with the -vv flag

rclone-v1.65.0-windows-amd64>rclone copy ./VM196S003012-20220815_123255402.mp4 gcs_remote:upload-bucket-useast1 --gcs-service-account-file srvc_key.json --config rclone1.conf --gcs-no-check-bucket -P -vv
2023/12/13 23:39:51 DEBUG : rclone: Version "v1.65.0" starting with parameters ["rclone" "copy" "./VM196S003012-20220815_123255402.mp4" "gcs_remote:upload-bucket-useast1" "--gcs-service-account-file" "srvc_key.json" "--config" "rclone1.conf" "--gcs-no-check-bucket" "-P" "-vv"]
2023/12/13 23:39:51 DEBUG : Creating backend with remote "./VM196S003012-20220815_123255402.mp4"
2023/12/13 23:39:51 DEBUG : Using config file from "C:\\Users\\tdayan\\Downloads\\rclone-v1.65.0-windows-amd64\\rclone1.conf"
2023/12/13 23:39:51 DEBUG : fs cache: adding new entry for parent of "./VM196S003012-20220815_123255402.mp4", "//?/C:/Users/user/Downloads/rclone-v1.65.0-windows-amd64"
2023/12/13 23:39:51 DEBUG : Creating backend with remote "gcs_remote:upload-bucket-useast1"
2023/12/13 23:39:51 DEBUG : gcs_remote: detected overridden config - adding "{oIHtk}" suffix to name
2023/12/13 23:39:51 DEBUG : fs cache: renaming cache item "gcs_remote:upload-bucket-useast1" to be canonical "gcs_remote{oIHtk}:upload-bucket-useast1"
2023/12/13 23:39:52 DEBUG : VM196S003012-20220815_123255402.mp4: Need to transfer - File not found at Destination
Transferred:          192 MiB / 779.890 MiB, 25%, 1023.176 KiB/s, ETA 9m48s
Transferred:            0 / 1, 0%
Elapsed time:      5m10.9s
Transferring:
 *           VM196S003012-20220815_123255402.mp4: 24% /779.890Mi, 1.013Mi/s, 9m40s
2

What is your upload bandwidth? Rclone is making 1023.176 KiB/s which is about 8Mbit/s which seems quite slow to me!

3

Yes, its slower than it should be. My upload speed is 5MB/s.
On other networks too, I can't reach the max limit of bandwidth.
Is there some settings/config I can set with rclone to improve the upload performance ?
(in case of download, for example there is the parameter of --multi-thread-streams)
Thanks

4

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.