Copyurl in v1.49.5 now breaks with a 302 response

Suspected Bug
1

What is the problem you are having with rclone?

  • Using rclone to copy codebase backups from GitHub to S3
  • Worked fine on my Mac when I was testing it as a proof-of-concept
  • Been trying to deploy this task on AWS, but it keeps failing with "404 not found" (see output below)
  • Turns out, my Mac was on v1.48.0 and the rclone/rclone Docker image is v1.49.5
  • And after updating my Mac to v1.49.5, this command also breaks with the same output as AWS

Which OS you are using and how many bits (eg Windows 7, 64 bit)

  • MacOS Mojave 10.14.6 64 bit, installed rclone through Homebrew
  • AWS ECS Fargate running rclone/rclone:latest

Which cloud storage system are you using? (eg Google Drive)

  • Source is the GitHub API
  • Destination is an S3 bucket, configured entirely through environment variables

The command you were trying to run (eg rclone copy /tmp remote:tmp)

$ RCLONE_CONFIG_S3_TYPE=s3 \
  RCLONE_CONFIG_S3_PROVIDER=AWS \
  RCLONE_CONFIG_S3_ENV_AUTH=true \
  RCLONE_CONFIG_S3_REGION=eu-west-2 \
  RCLONE_CONFIG_S3_LOCATION_CONSTRAINT=eu-west-2 \
  RCLONE_CONFIG_S3_ACL=private \
  RCLONE_HTTP_HEADERS="Authorization: token $GITHUB_TOKEN" \
  rclone copyurl \
  https://api.github.com/repos/jdrydn/yoem/zipball \
  s3:backup-bucket/yoem.zip

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

2019/10/12 09:41:08 DEBUG : rclone: Version "v1.49.5" starting with parameters ["rclone" "-vv" "copyurl" "https://api.github.com/repos/jdrydn/yoem/zipball" "s3:backup-bucket/yoem.zip"]
2019/10/12 09:41:08 NOTICE: Config file "/Users/james/.config/rclone/rclone.conf" not found - using defaults
2019/10/12 09:41:09 ERROR : Attempt 1/3 failed with 1 errors and: CopyURL failed: 404 Not Found
2019/10/12 09:41:11 ERROR : Attempt 2/3 failed with 1 errors and: CopyURL failed: 404 Not Found
2019/10/12 09:41:12 ERROR : Attempt 3/3 failed with 1 errors and: CopyURL failed: 404 Not Found
2019/10/12 09:41:12 Failed to copyurl: CopyURL failed: 404 Not Found
  • Could be related to a change made to the CopyURL method?
  • The GitHub URL returns a 302 Found that resolves to a 200 OK, could be relevant?
2

That kind of indicates that Copy URL tried to copy the file but got a 404 not found.

I tried your example on v1.49.5 and the latest beta and it worked fine on both, though I didn't use the HTTP_HEADERS I just did it without auth.

If you run with -vv --dump bodies --log-file test.log then you can examine all the http transactions in the log. Hopefully that should shed some light - if not post them (redacting sensitive info if any) and I can take a look.

3

Aha, --dump bodies is a great debug option :+1: Looks like RCLONE_HTTP_HEADERS doesn't affect copyurl, just the HTTP remote. That is to say, the Authorization header isn't present in the log at all, which explains the 404 result! I guess something changed regarding this option between v1.48.0 and v1.49.5.

However I just realised I can get this to work by putting the token in the URL:

https://$GITHUB_TOKEN@api.github.com/repos/jdrydn/yoem/zipball

And now it works smoothly!

4

Yes I should have noticed that - sorry.

I don't think the RCLONE_HTTP_HEADERS env var has ever worked on anything but the http remote... :confused:

You could do your copy using the HTTP remote instead of using copyurl but it seems you've made it work in a great way!

Note that I tested with exactly your url and I got the zip back without using the Authorization: header.

5

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.