Copyurl fails with stream error (wget and curl works)

Looks like they use CDN so nothing unusual to have different IP addresses. For me it is (and rclone works):

$ host www.nass.usda.gov
www.nass.usda.gov is an alias for www.nass.usda.gov.edgekey.net.
www.nass.usda.gov.edgekey.net is an alias for e10552.dscx.akamaiedge.net.
e10552.dscx.akamaiedge.net has address 104.103.197.236
e10552.dscx.akamaiedge.net has IPv6 address 2a02:26f0:fd00:da6::2938
e10552.dscx.akamaiedge.net has IPv6 address 2a02:26f0:fd00:dab::2938

I fudged my /etc/host file so that rclone resolved the same IP

$ rclone copyurl -P https://www.nass.usda.gov/Research_and_Science/Cropland/Release/datasets/2023_30m_cdls.zip sss.zip
Transferred:        1.049 GiB / 1.918 GiB, 55%, 5.430 MiB/s, ETA 2m43s
Transferred:            0 / 1, 0%
Elapsed time:       3m0.0s
Transferring:
*                                       sss.zip: 62% /1.918Gi, 6.129Mi/s, 2m1s
tcp        0   2016 10.10.0.10:32808          23.6.101.171:443        ESTABLISHED 3380038/rclone      

And it works absolutely perfectly downloading at 6 MiB/s (about my internet connection's limit).

I don't think this can be an rclone problem, I think it must be a networking problem or possibly a policy problem (ie some kind of ban from AWS).

:frowning:

But then what is the difference between rclone, requests, curl and wget? Why rclone fails, and all the other work?

A good question!

Things which are different between rclone and curl could be

  • SSL negotiation
  • Networking stack (think packet sizes)
  • Policy at the other end (user agent, ip address)

Probably something to do with SSL is most likely so that could be a go standard library problem. Can you try with some older rclone versions and see if that helps?

Here are the results of trying different versions:

1.43   works !
1.47.0 works !
1.48.0 fail, but much longer timeouts and different error
1.49.5 fail, but much longer timeouts and different error
1.50.0 fail, same as latest
1.50.2 fail, same as latest
1.52.3 fail, same as latest
1.54.1 fail, same as latest
1.64.2 fail, same as latest

So it stops working at 1.48.0 and slightly changes in how it fails in 1.50.0. The output of the 1.48.0 and 1.49.5 is this (note 5 minutes between each line):

[ec2-user@ip-172-31-30-203 rclone-v1.49.5-linux-amd64]$ ./rclone copyurl https://www.nass.usda.gov/Research_and_Science/Cropland/Release/datasets/2023_30m_cdls.zip out
2024/02/11 08:08:17 NOTICE: Config file "/home/ec2-user/.config/rclone/rclone.conf" not found - using defaults
2024/02/11 08:13:17 ERROR : Attempt 1/3 failed with 1 errors and: Get https://www.nass.usda.gov/Research_and_Science/Cropland/Release/datasets/2023_30m_cdls.zip: net/http: timeout awaiting response headers
2024/02/11 08:18:17 ERROR : Attempt 2/3 failed with 1 errors and: Get https://www.nass.usda.gov/Research_and_Science/Cropland/Release/datasets/2023_30m_cdls.zip: net/http: timeout awaiting response headers
2024/02/11 08:23:17 ERROR : Attempt 3/3 failed with 1 errors and: Get https://www.nass.usda.gov/Research_and_Science/Cropland/Release/datasets/2023_30m_cdls.zip: net/http: timeout awaiting response headers
2024/02/11 08:23:17 Failed to copyurl: Get https://www.nass.usda.gov/Research_and_Science/Cropland/Release/datasets/2023_30m_cdls.zip: net/http: timeout awaiting response headers

Rclone 1.47 was compiled with go1.12.4 whereas rclone 1.48 was compiled with go1.12.6

So that is actuallly quite a narrow range of go versions.

There are only two versions changes to look through go1.12.5 and go1.12.6

the changelog for 1.12.6 looks interesting

go1.12.6 (released 2019-06-11) includes fixes to the compiler, the linker, the go command, and the crypto/x509 , net/http , and os packages. See the Go 1.12.6 milestone on our issue tracker for details.

With fixes for http and parts of TLS. The HTTP Issue looks relevant

net/http: make Transport ignore 408 timeout messages from server [1.12 backport]

I think that is probably a red herring (but not 100% certain).

Let's now have a look at the changes in rclone between 1.47 and 1.48

The only significant change is this one 72721f4c8d11 which is

copyurl: honor --no-check-certificate

Looking at the code it does do something significant - it changes rclone over from using Go's http transport to rclone's http transport for copyurl.

Rclone's http transport has a whole heap of other features which are very useful, but maybe one of these is interfering somehow.

Can you try this? It reverts that commit to make copyurl use Go's HTTP transport. That isn't a proper fix, but if it works we'll know we are on the right track!

v1.66.0-beta.7686.601947eeb.fix-copyurl on branch fix-copyurl (uploaded in 15-30 mins)

That didn't build, try this one

v1.66.0-beta.7686.73a9363ca.fix-copyurl on branch fix-copyurl (uploaded in 15-30 mins)

Bingo!

[ec2-user@ip-172-31-31-218 rclone-v1.66.0-beta.7686.73a9363ca.fix-copyurl-linux-amd64]$ ./rclone copyurl https://www.nass.usda.gov/Research_and_Science/Cropland/Release/datasets/2023_30m_cdls.zip out
2024/02/13 07:56:50 NOTICE: Config file "/home/ec2-user/.config/rclone/rclone.conf" not found - using defaults
[ec2-user@ip-172-31-31-218 rclone-v1.66.0-beta.7686.73a9363ca.fix-copyurl-linux-amd64]$ ls out -lah
-rw-r--r--. 1 ec2-user ec2-user 2.0G Jan 26 20:06 out

Great! We are getting somewhere :slightly_smiling_face:

Is there anything special about your ec2 instance? Next week I can try getting one and see if I can replicate the problem.

A problem in the core http client is definitely worth fixing so any hints on how to get the correct ec2 instance appreciated!

There seems to be some correlation and special handling of the user agent too.

I was able to reproduce this on a plain Lightsail VM from AWS and also a separate server from Hetzner.

The latest version of rclone with the user-agent set to curl/7.88.1 works fine but not with the user-agent set to Mozilla/5.0 (Windows NT 10.0; Win64; x64).

Even curl doesn't work when the user-agent is overridden to Mozilla/5.0 (Windows NT 10.0; Win64; x64).

Trying a few invalid user-agents also don't help but anything starting with curl or wget works fine.

1 Like

Looks like the default Go transport defaults to the Go-http-client user-agent which also works fine with the latest rclone so it definitely seems like it's a user-agent whitelist on the servers.

1 Like

It can't be only user-agent, as rclone works from local machines. Maybe a filter on (IP, user-agent) pair?

To answer @ncw question, no, nothing special at all. Brand new AWS account, us-west-2 region, create ec2 instance, all settings default.

edit: I confirm that this works:

rclone copyurl https://www.nass.usda.gov/Research_and_Science/Cropland/Release/datasets/2023_30m_cdls.zip out --user-agent "curl"

Yeah, it seems like they may possibly restrict known Datacenter IP ranges to known user-agents.

Can you also confirm that it works for you by setting the user-agent to Go-http-client (which is the default user-agent of the Go HTTP transport rather than the rclone transport).

Yep, works with Go-http-client as well.

1 Like

Seems very similar: https://community.akamai.com/customers/s/question/0D54R00007GjCANSA3/why-does-akamai-edge-services-sometime-just-not-send-any-response-leaving-the-connection-to-timeout?language=en_US

Ah, great - the mystery is solved!

The question is what should we do about it?

I don't really want to change rclone's user agent to be curl or rclone/v1.66 curl or whatever that way we get madness like Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.3 - anyone want to guess which browser that is?

We could put a note in the docs of copyurl maybe? Would that have helped @psarka ?

I don't like it a lot but maybe something like Go-http-client/rclone/v1.66 (It works btw)?

But I am not sure whether any cloud provider has special handling for the current rclone UA in a positive way which may break because of this.

copyurl docs are rather short, so yes, likely I would have seen the note. One another thing that would have saved your time, would be a small list of debugging steps. Like:

  1. Try to run with curl
  2. Try to run with http2 disabled
  3. Try user-agent curl

It doesn't need to go to docs, a pinned topic in this forum would be enough. I really liked the message about reporting rclone version correctly, and I would have gladly followed any other instructions that would make your life easier.