Endlessly repeating &userProject= parameter in GET to google storage: context canceled, got HTTP response code 400

What is the problem you are having with rclone?

Copying from a requester-pays bucket in Google Cloud Storage using user_project=xyz fails with:

context canceled, got HTTP response code 400

Run the command 'rclone version' and share the full output of the command.

rclone v1.64.2

  • os/version: redhat 9.2 (64 bit)
  • os/kernel: 5.14.0-284.30.1.el9_2.x86_64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.21.3
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Cloud Storage

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy redacted:redacted/redacted/redacted/redacted/redacted/redacted /redacted/redacted/redacted/redacted/redacted/redacted/redacted/redacted/redacted/redacted --progress

The rclone config contents with secrets removed.

[redacted]
type = google cloud storage
location = us-central1
token = {"access_token":"redacted","token_type":"Bearer","refresh_token":"redacted","expiry":"2023-11-01T07:10:00.448660107Z"}
user_project = redacted

A log from the command with the -vv flag

2023/11/01 06:59:39 DEBUG : redacted.gz: multi-thread copy: chunk 488/4282 (32682016768-32749125632) size 64Mi finished
2023/11/01 06:59:39 DEBUG : redacted.gz: multi-thread copy: chunk 492/4282 (32950452224-33017561088) size 64Mi starting
2023/11/01 06:59:41 DEBUG : redacted.gz: multi-thread copy: chunk 489/4282 (32749125632-32816234496) size 64Mi finished
2023/11/01 06:59:41 DEBUG : redacted.gz: multi-thread copy: chunk 493/4282 (33017561088-33084669952) size 64Mi starting
2023/11/01 06:59:41 DEBUG : redacted.gz: multi-thread copy: chunk 492/4282 failed: multi-thread copy: failed to open source: googleapi: got HTTP response code 400 with body: <!DOCTYPE html> <html lang=en> <meta charset=utf-8>
 <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
 <title>Error 400 (Bad Request)!!1</title> <style> *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-
resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px} </style> <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
 <p><b>400.</b> <ins>That’s an error.</ins>
 <p>Your client has issued a malformed or illegal request. <ins>That’s all we know.</ins>
2023/11/01 06:59:41 DEBUG : redacted.gz: multi-thread copy: chunk 494/4282 (33084669952-33151778816) size 64Mi starting
2023/11/01 06:59:41 DEBUG : redacted.gz: multi-thread copy: chunk 493/4282 failed: multi-thread copy: failed to open source: Get "https://storage.googleapis.com/download/storage/v1/b/bucket-name/o/releases%2F2022_v3release_1115%2Fredacted%2Fredacted%2Fredacted%2Fredacted.gz?generation=1669861802119834&alt=media&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted": context canceled
2023/11/01 06:59:41 DEBUG : redacted.gz: Reopening on read failure after 44972936 bytes: retry 1/10: context canceled
2023/11/01 06:59:41 DEBUG : redacted.gz: Reopening on read failure after 59030418 bytes: retry 1/10: context canceled
2023/11/01 06:59:41 DEBUG : redacted.gz: Reopen failed after 44972936 bytes read: Get "https://storage.googleapis.com/download/storage/v1/b/bucket-name/o/releases%2F2022_v3release_1115%2Fredacted%2Fredacted%2Fredacted%2Fredacted.gz?generation=1669861802119834&alt=media&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted": context canceled
2023/11/01 06:59:41 DEBUG : redacted.gz: multi-thread copy: chunk 491/4282 failed: multi-thread copy: failed to write chunk: context canceled

This happened with both --transfers 4 and --transfers 1. I have not tried a complete copy with --multi-thread-streams 0 as it would take days to complete.

Full debug logs (non or less redacted) available upon request.

The bug is easy to spot. The userProject is being added to the URL each time it is opened here

I'm not at my computer right now so I'll try to fix later or you could have a go

Thanks for the quick reply! I am not familiar enough to edit the code myself.

In the meantime I used gsutil to copy the data.

p.s.: For some reason gsutil was much faster. Rclone was stuck at about 60 MiB/s but gsutil 300MiB/s. I tried --transfers 1, 4, and 8. Always ~60 MiB/s with rclone.

I've attempted to make a fix here - please give it a try!

v1.65.0-beta.7469.29e26abad.fix-gcs-open on branch fix-gcs-open (uploaded in 15-30 mins)

I think you are copying 1 big file? Setting --transfers won't help in that case. but setting --multi-thread-streams will

I've attempted to make a fix here - please give it a try!
v1.65.0-beta.7469.29e26abad.fix-gcs-open on branch fix-gcs-open (uploaded in 15-30 mins)

Thanks. I'll try to test it soon.

I think you are copying 1 big file? Setting --transfers won't help in that case. but setting --multi-thread-streams will

These were about 20 files totalling about 7.5TB. Half the files were small metadata files and checksums, the other half large data files (>10GB). But, that's off-topic. Thanks for responding quickly!

1 Like