Looks like they use CDN so nothing unusual to have different IP addresses. For me it is (and rclone works):
$ host www.nass.usda.gov
www.nass.usda.gov is an alias for www.nass.usda.gov.edgekey.net.
www.nass.usda.gov.edgekey.net is an alias for e10552.dscx.akamaiedge.net.
e10552.dscx.akamaiedge.net has address 104.103.197.236
e10552.dscx.akamaiedge.net has IPv6 address 2a02:26f0:fd00:da6::2938
e10552.dscx.akamaiedge.net has IPv6 address 2a02:26f0:fd00:dab::2938
I fudged my /etc/host file so that rclone resolved the same IP
$ rclone copyurl -P https://www.nass.usda.gov/Research_and_Science/Cropland/Release/datasets/2023_30m_cdls.zip sss.zip
Transferred: 1.049 GiB / 1.918 GiB, 55%, 5.430 MiB/s, ETA 2m43s
Transferred: 0 / 1, 0%
Elapsed time: 3m0.0s
Transferring:
* sss.zip: 62% /1.918Gi, 6.129Mi/s, 2m1s
tcp 0 2016 10.10.0.10:32808 23.6.101.171:443 ESTABLISHED 3380038/rclone
And it works absolutely perfectly downloading at 6 MiB/s (about my internet connection's limit).
I don't think this can be an rclone problem, I think it must be a networking problem or possibly a policy problem (ie some kind of ban from AWS).
But then what is the difference between rclone, requests, curl and wget? Why rclone fails, and all the other work?
A good question!
Things which are different between rclone and curl could be
Probably something to do with SSL is most likely so that could be a go standard library problem. Can you try with some older rclone versions and see if that helps?
Here are the results of trying different versions:
1.43 works !
1.47.0 works !
1.48.0 fail, but much longer timeouts and different error
1.49.5 fail, but much longer timeouts and different error
1.50.0 fail, same as latest
1.50.2 fail, same as latest
1.52.3 fail, same as latest
1.54.1 fail, same as latest
1.64.2 fail, same as latest
So it stops working at 1.48.0 and slightly changes in how it fails in 1.50.0. The output of the 1.48.0 and 1.49.5 is this (note 5 minutes between each line):
[ec2-user@ip-172-31-30-203 rclone-v1.49.5-linux-amd64]$ ./rclone copyurl https://www.nass.usda.gov/Research_and_Science/Cropland/Release/datasets/2023_30m_cdls.zip out
2024/02/11 08:08:17 NOTICE: Config file "/home/ec2-user/.config/rclone/rclone.conf" not found - using defaults
2024/02/11 08:13:17 ERROR : Attempt 1/3 failed with 1 errors and: Get https://www.nass.usda.gov/Research_and_Science/Cropland/Release/datasets/2023_30m_cdls.zip: net/http: timeout awaiting response headers
2024/02/11 08:18:17 ERROR : Attempt 2/3 failed with 1 errors and: Get https://www.nass.usda.gov/Research_and_Science/Cropland/Release/datasets/2023_30m_cdls.zip: net/http: timeout awaiting response headers
2024/02/11 08:23:17 ERROR : Attempt 3/3 failed with 1 errors and: Get https://www.nass.usda.gov/Research_and_Science/Cropland/Release/datasets/2023_30m_cdls.zip: net/http: timeout awaiting response headers
2024/02/11 08:23:17 Failed to copyurl: Get https://www.nass.usda.gov/Research_and_Science/Cropland/Release/datasets/2023_30m_cdls.zip: net/http: timeout awaiting response headers
Rclone 1.47 was compiled with go1.12.4 whereas rclone 1.48 was compiled with go1.12.6
So that is actuallly quite a narrow range of go versions.
There are only two versions changes to look through go1.12.5 and go1.12.6
the changelog for 1.12.6 looks interesting
go1.12.6 (released 2019-06-11) includes fixes to the compiler, the linker, the go command, and the
crypto/x509,net/http, andospackages. See the Go 1.12.6 milestone on our issue tracker for details.
With fixes for http and parts of TLS. The HTTP Issue looks relevant
net/http: make Transport ignore 408 timeout messages from server [1.12 backport]
I think that is probably a red herring (but not 100% certain).
Let's now have a look at the changes in rclone between 1.47 and 1.48
The only significant change is this one 72721f4c8d11 which is
copyurl: honor --no-check-certificate
Looking at the code it does do something significant - it changes rclone over from using Go's http transport to rclone's http transport for copyurl.
Rclone's http transport has a whole heap of other features which are very useful, but maybe one of these is interfering somehow.
Can you try this? It reverts that commit to make copyurl use Go's HTTP transport. That isn't a proper fix, but if it works we'll know we are on the right track!
v1.66.0-beta.7686.601947eeb.fix-copyurl on branch fix-copyurl (uploaded in 15-30 mins)
That didn't build, try this one
v1.66.0-beta.7686.73a9363ca.fix-copyurl on branch fix-copyurl (uploaded in 15-30 mins)
Bingo!
[ec2-user@ip-172-31-31-218 rclone-v1.66.0-beta.7686.73a9363ca.fix-copyurl-linux-amd64]$ ./rclone copyurl https://www.nass.usda.gov/Research_and_Science/Cropland/Release/datasets/2023_30m_cdls.zip out
2024/02/13 07:56:50 NOTICE: Config file "/home/ec2-user/.config/rclone/rclone.conf" not found - using defaults
[ec2-user@ip-172-31-31-218 rclone-v1.66.0-beta.7686.73a9363ca.fix-copyurl-linux-amd64]$ ls out -lah
-rw-r--r--. 1 ec2-user ec2-user 2.0G Jan 26 20:06 out
Great! We are getting somewhere 
Is there anything special about your ec2 instance? Next week I can try getting one and see if I can replicate the problem.
A problem in the core http client is definitely worth fixing so any hints on how to get the correct ec2 instance appreciated!
There seems to be some correlation and special handling of the user agent too.
I was able to reproduce this on a plain Lightsail VM from AWS and also a separate server from Hetzner.
The latest version of rclone with the user-agent set to curl/7.88.1 works fine but not with the user-agent set to Mozilla/5.0 (Windows NT 10.0; Win64; x64).
Even curl doesn't work when the user-agent is overridden to Mozilla/5.0 (Windows NT 10.0; Win64; x64).
Trying a few invalid user-agents also don't help but anything starting with curl or wget works fine.
Looks like the default Go transport defaults to the Go-http-client user-agent which also works fine with the latest rclone so it definitely seems like it's a user-agent whitelist on the servers.
It can't be only user-agent, as rclone works from local machines. Maybe a filter on (IP, user-agent) pair?
To answer @ncw question, no, nothing special at all. Brand new AWS account, us-west-2 region, create ec2 instance, all settings default.
edit: I confirm that this works:
rclone copyurl https://www.nass.usda.gov/Research_and_Science/Cropland/Release/datasets/2023_30m_cdls.zip out --user-agent "curl"
Yeah, it seems like they may possibly restrict known Datacenter IP ranges to known user-agents.
Can you also confirm that it works for you by setting the user-agent to Go-http-client (which is the default user-agent of the Go HTTP transport rather than the rclone transport).
Yep, works with Go-http-client as well.
Ah, great - the mystery is solved!
The question is what should we do about it?
I don't really want to change rclone's user agent to be curl or rclone/v1.66 curl or whatever that way we get madness like Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.3 - anyone want to guess which browser that is?
We could put a note in the docs of copyurl maybe? Would that have helped @psarka ?
I don't like it a lot but maybe something like Go-http-client/rclone/v1.66 (It works btw)?
But I am not sure whether any cloud provider has special handling for the current rclone UA in a positive way which may break because of this.
copyurl docs are rather short, so yes, likely I would have seen the note. One another thing that would have saved your time, would be a small list of debugging steps. Like:
It doesn't need to go to docs, a pinned topic in this forum would be enough. I really liked the message about reporting rclone version correctly, and I would have gladly followed any other instructions that would make your life easier.
Good idea! I added a small section like this to the copyurl docs for the v1.66 release
If you can't get rclone copyurl to work then here are some things you can try:
--disable-http2 rclone will use HTTP2 if available - try disabling it--bind 0.0.0.0 rclone will use IPv6 if available - try disabling it--bind ::0 to disable IPv4--user agent curl - some sites have whitelists for curl's user-agent - try thatcurl directlyThis topic was automatically closed 30 days after the last reply. New replies are no longer allowed.