Endlessly repeating &userProject= parameter in GET to google storage: context canceled, got HTTP response code 400

Suspected Bug
1

What is the problem you are having with rclone?

Copying from a requester-pays bucket in Google Cloud Storage using user_project=xyz fails with:

context canceled, got HTTP response code 400

Run the command 'rclone version' and share the full output of the command.

rclone v1.64.2

  • os/version: redhat 9.2 (64 bit)
  • os/kernel: 5.14.0-284.30.1.el9_2.x86_64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.21.3
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Cloud Storage

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy redacted:redacted/redacted/redacted/redacted/redacted/redacted /redacted/redacted/redacted/redacted/redacted/redacted/redacted/redacted/redacted/redacted --progress

The rclone config contents with secrets removed.

[redacted]
type = google cloud storage
location = us-central1
token = {"access_token":"redacted","token_type":"Bearer","refresh_token":"redacted","expiry":"2023-11-01T07:10:00.448660107Z"}
user_project = redacted

A log from the command with the -vv flag

2023/11/01 06:59:39 DEBUG : redacted.gz: multi-thread copy: chunk 488/4282 (32682016768-32749125632) size 64Mi finished
2023/11/01 06:59:39 DEBUG : redacted.gz: multi-thread copy: chunk 492/4282 (32950452224-33017561088) size 64Mi starting
2023/11/01 06:59:41 DEBUG : redacted.gz: multi-thread copy: chunk 489/4282 (32749125632-32816234496) size 64Mi finished
2023/11/01 06:59:41 DEBUG : redacted.gz: multi-thread copy: chunk 493/4282 (33017561088-33084669952) size 64Mi starting
2023/11/01 06:59:41 DEBUG : redacted.gz: multi-thread copy: chunk 492/4282 failed: multi-thread copy: failed to open source: googleapi: got HTTP response code 400 with body: <!DOCTYPE html> <html lang=en> <meta charset=utf-8>
 <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
 <title>Error 400 (Bad Request)!!1</title> <style> *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-
resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px} </style> <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
 <p><b>400.</b> <ins>That’s an error.</ins>
 <p>Your client has issued a malformed or illegal request. <ins>That’s all we know.</ins>
2023/11/01 06:59:41 DEBUG : redacted.gz: multi-thread copy: chunk 494/4282 (33084669952-33151778816) size 64Mi starting
2023/11/01 06:59:41 DEBUG : redacted.gz: multi-thread copy: chunk 493/4282 failed: multi-thread copy: failed to open source: Get "https://storage.googleapis.com/download/storage/v1/b/bucket-name/o/releases%2F2022_v3release_1115%2Fredacted%2Fredacted%2Fredacted%2Fredacted.gz?generation=1669861802119834&alt=media&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted": context canceled
2023/11/01 06:59:41 DEBUG : redacted.gz: Reopening on read failure after 44972936 bytes: retry 1/10: context canceled
2023/11/01 06:59:41 DEBUG : redacted.gz: Reopening on read failure after 59030418 bytes: retry 1/10: context canceled
2023/11/01 06:59:41 DEBUG : redacted.gz: Reopen failed after 44972936 bytes read: Get "https://storage.googleapis.com/download/storage/v1/b/bucket-name/o/releases%2F2022_v3release_1115%2Fredacted%2Fredacted%2Fredacted%2Fredacted.gz?generation=1669861802119834&alt=media&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted&userProject=redacted": context canceled
2023/11/01 06:59:41 DEBUG : redacted.gz: multi-thread copy: chunk 491/4282 failed: multi-thread copy: failed to write chunk: context canceled

This happened with both --transfers 4 and --transfers 1. I have not tried a complete copy with --multi-thread-streams 0 as it would take days to complete.

Full debug logs (non or less redacted) available upon request.

2

The bug is easy to spot. The userProject is being added to the URL each time it is opened here

I'm not at my computer right now so I'll try to fix later or you could have a go

3

Thanks for the quick reply! I am not familiar enough to edit the code myself.

In the meantime I used gsutil to copy the data.

p.s.: For some reason gsutil was much faster. Rclone was stuck at about 60 MiB/s but gsutil 300MiB/s. I tried --transfers 1, 4, and 8. Always ~60 MiB/s with rclone.

4

I've attempted to make a fix here - please give it a try!

v1.65.0-beta.7469.29e26abad.fix-gcs-open on branch fix-gcs-open (uploaded in 15-30 mins)

I think you are copying 1 big file? Setting --transfers won't help in that case. but setting --multi-thread-streams will

5

I've attempted to make a fix here - please give it a try!
v1.65.0-beta.7469.29e26abad.fix-gcs-open on branch fix-gcs-open (uploaded in 15-30 mins)

Thanks. I'll try to test it soon.

I think you are copying 1 big file? Setting --transfers won't help in that case. but setting --multi-thread-streams will

These were about 20 files totalling about 7.5TB. Half the files were small metadata files and checksums, the other half large data files (>10GB). But, that's off-topic. Thanks for responding quickly!

1 Like
6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.