302 Header Issue on Rclone Checker Script

What is the problem you are having with rclone?

I am running a checker script and return 302 response missing Location header when I try to check the contents of this page.

What is your rclone version (output from rclone version)

v. 1.52.3

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Linux Centos 6.10

Which cloud storage system are you using? (eg Google Drive)

AWS (destination) and remote HTTPS server (source)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone check "NOMADs:" "AWS test:" --one-way

The rclone config contents with secrets removed.

[horelS3]
type = s3
env_auth = false
access_key_id = hidden
secret_access_key = very hidden
#region = other-v2-signature
endpoint = https://pando-rgw01.chpc.utah.edu
location_constraint =

[AWS test]
type = s3
provider = AWS
env_auth = false
access_key_id = secret
secret_access_key =extra secret
region = us-west-1

[Tom Hrrr]
type = google cloud storage
token = {"super secret"}
client_id = https://console.cloud.google.com/storage/browser/high-resolution-rapid-refresh

[NOMADs]
type = http
url = https://nomads.ncep.noaa.gov/pub/data/nccf/com/hrrr/prod/

A log from the command with the -vv flag

2020/11/05 12:21:46 DEBUG : rclone: Version "v1.52.3" starting with parameters ["rclone" "check" "NOMADs:hrrr.20201104/conus/" "AWS test:hrrr/sfc/20201104/" "--one-way" "-vv"]
2020/11/05 12:21:46 DEBUG : Using config file from "/uufs/chpc.utah.edu/common/home/u0845413/.config/rclone/rclone.conf"
2020/11/05 12:21:46 DEBUG : fs cache: renaming cache item "AWS test:hrrr/sfc/20201104/" to be canonical "AWS test:hrrr/sfc/20201104"
2020/11/05 12:21:46 DEBUG : S3 bucket hrrr path sfc/20201104: Waiting for checks to finish
2020/11/05 12:21:55 DEBUG : hrrr.t11z.wrfsfcf05.grib2.idx: skipping because of error: failed to stat: Head "https://nomads.ncep.noaa.gov/pub/data/nccf/com/hrrr/prod/hrrr.20201104/conus/hrrr.t11z.wrfsfcf05.grib2.idx": 302 response missing Location header

I tried the site with rclone and I can see the 302 errors look like this

2020/11/06 17:24:36 DEBUG : HTTP RESPONSE (req 0xc0005f7f00)
2020/11/06 17:24:36 DEBUG : HTTP/1.1 302 Your allowed limit has been reached. Please go to https://www.weather.gov/abusive-user-block for more info 

So I suspect you are hitting the service too hard. Try slowing down with --tpslimit 1 - you might be able to use a bigger number or need a smaller.

Oh excellent, that explains the error. I was able to run it with the tpslimit and it worked! Now I just need to find the best setting it will allow. Thank you!

1 Like

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.