What is the problem you are having with rclone?
I'm trying to download a presigned S3 url using rclone and the http-url + files-from options. This results in a bad request. I am able to download the signed url through copyurl instead of copy but I would love to leverage the files-from functionality.
What is your rclone version (output from rclone version)
1.55.0
Which OS you are using and how many bits (eg Windows 7, 64 bit)
MacOs Big Sur
Which cloud storage system are you using? (eg Google Drive)
Currently none, attempting to download the file locally
The command you were trying to run (eg rclone copy /tmp remote:tmp)
rclone --config rclone.conf copy --files-from=list.txt --no-traverse --dump headers --http-url=https://<bucket-name>.s3.eu-west-1.amazonaws.com :http: destination: -vvv
list.txt contains the remainder of the presigned S3 Url (with https://.s3.eu-west-1.amazonaws.com/ removed)
The rclone config contents with secrets removed.
[destination]
type = local
A log from the command with the -vv flag
2021/04/26 14:58:50 DEBUG : Using config file from "/Users/michieldhadamus/testrclone/rclone.conf"
2021/04/26 14:58:50 DEBUG : rclone: Version "v1.55.0" starting with parameters ["rclone" "--config" "rclone.conf" "copy" "--files-from=list.txt" "--no-traverse" "--dump" "headers" "--http-url=https://<buckt-name>.eu-west-1.amazonaws.com" ":http:" "destination:" "-vvv"]
2021/04/26 14:58:50 DEBUG : Creating backend with remote ":http:"
2021/04/26 14:58:50 DEBUG : :http: detected overridden config - adding "{oWmdZ}" suffix to name
2021/04/26 14:58:50 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/04/26 14:58:50 DEBUG : fs cache: renaming cache item ":http:" to be canonical ":http{oWmdZ}:"
2021/04/26 14:58:50 DEBUG : Creating backend with remote "destination:"
2021/04/26 14:58:50 DEBUG : fs cache: renaming cache item "destination:" to be canonical "destination:/Users/michieldhadamus/testrclone"
2021/04/26 14:58:50 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/04/26 14:58:50 DEBUG : HTTP REQUEST (req 0xc000504d00)
2021/04/26 14:58:50 DEBUG : HEAD /data/path_to_file.parquet.snappy.parquet%5C%3FX-Amz-Algorithm%5C=AWS4-HMAC-SHA256%5C&X-Amz-Credential%5C=ASIA542GQASCPTEV33O7%252F20210426%252Feu-west-1%252Fs3%252Faws4_request%5C&X-Amz-Date%5C=20210426T114330Z%5C&X-Amz-Expires%5C=604800%5C&X-Amz-SignedHeaders%5C=host%5C&X-Amz-Security-Token%5C=IQoJb3JpZ2luX2VjELT%252F%252F%252F%252F%252F%252F%252F%252F%252F%252FwEaCWV1LXdlc3QtMSJIMEYCIQDSoWLbS%252FPEe8M%252B0hSdKp2NaFwHWNHGc%252BNb4xGmRcaKwwIhAIjH48xhggaeKLG%252BoWLksMtypsu91cvBWQX5uEdPgLzaKqgCCC0QAhoMOTU1MjQxMjAyODIwIgyenxSnMRQGyj74j1UqhQJzwkoOTGMOIu3S0rDn%252BBsCfVAcknDupAc5q0TIvfcx6cdAeS6aDERaKNCBf50ue%252B1EpPCsZh6atY2%252Bq1lOqyI9gzw15Wd4c0ddkba%252F78Xz5P5MdF7a9ubv%252BGndnF3IPxtxRg53Wji7el7s4g3BmgiKOf9Fsx4ntbjQdW6841voBy7Lue50GXNiLjP%252B2fkovCuaERAW52BX1H4YvG5J3U7338MfkR%252BrfPgtbR9lHYchrKBDbpbLmh11Ybf8p2%252FDPongPOnVkzu1I58iTsM2NCH%252F15JoPYYf%252BbLmBQSS7R%252FNdsItqIpHDNt9JZem5OEgeR38gf4HirVYcNMZj0ibkgdBLDuyTH%252BrphVlnNA6T4U9h7HhCeEz1uIGtUAF8mxVJ%252BXkVrb1DMGLxfjqc%253D%5C&X-Amz-Signature%5C=3d4e68e9d25f7e1f58318325dc9c1af3570f3d708093f8096f1488c597422751 HTTP/1.1
Host: <bucket-name>.s3.eu-west-1.amazonaws.com
User-Agent: rclone/v1.55.0
2021/04/26 14:58:50 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/04/26 14:58:50 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/04/26 14:58:50 DEBUG : HTTP RESPONSE (req 0xc000504d00)
2021/04/26 14:58:50 DEBUG : HTTP/1.1 400 Bad Request
Connection: close
Content-Type: application/xml
Date: Mon, 26 Apr 2021 12:58:49 GMT
Server: AmazonS3
X-Amz-Id-2: chCKYGUNbd22oCLx3YveeHHA+IXOGZ/e8T5vkknfmx4z4LrgRtHNH0ALuIhGUZihC3aMpSsGBzA=
X-Amz-Request-Id: V32P6D3FHX819FDK
2021/04/26 14:58:50 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/04/26 14:58:50 ERROR : : error reading source directory: failed to stat: HTTP Error 400: 400 Bad Request
2021/04/26 14:58:50 DEBUG : Local file system at /Users/michieldhadamus/testrclone: Waiting for checks to finish
2021/04/26 14:58:50 DEBUG : Local file system at /Users/michieldhadamus/testrclone: Waiting for transfers to finish
2021/04/26 14:58:50 ERROR : Attempt 1/3 failed with 1 errors and: failed to stat: HTTP Error 400: 400 Bad Request
2021/04/26 14:58:50 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/04/26 14:58:50 DEBUG : HTTP REQUEST (req 0xc0006da300)
2021/04/26 14:58:50 DEBUG : HEAD /data/path_to_file.parquet.snappy.parquet%5C%3FX-Amz-Algorithm%5C=AWS4-HMAC-SHA256%5C&X-Amz-Credential%5C=ASIA542GQASCPTEV33O7%252F20210426%252Feu-west-1%252Fs3%252Faws4_request%5C&X-Amz-Date%5C=20210426T114330Z%5C&X-Amz-Expires%5C=604800%5C&X-Amz-SignedHeaders%5C=host%5C&X-Amz-Security-Token%5C=IQoJb3JpZ2luX2VjELT%252F%252F%252F%252F%252F%252F%252F%252F%252F%252FwEaCWV1LXdlc3QtMSJIMEYCIQDSoWLbS%252FPEe8M%252B0hSdKp2NaFwHWNHGc%252BNb4xGmRcaKwwIhAIjH48xhggaeKLG%252BoWLksMtypsu91cvBWQX5uEdPgLzaKqgCCC0QAhoMOTU1MjQxMjAyODIwIgyenxSnMRQGyj74j1UqhQJzwkoOTGMOIu3S0rDn%252BBsCfVAcknDupAc5q0TIvfcx6cdAeS6aDERaKNCBf50ue%252B1EpPCsZh6atY2%252Bq1lOqyI9gzw15Wd4c0ddkba%252F78Xz5P5MdF7a9ubv%252BGndnF3IPxtxRg53Wji7el7s4g3BmgiKOf9Fsx4ntbjQdW6841voBy7Lue50GXNiLjP%252B2fkovCuaERAW52BX1H4YvG5J3U7338MfkR%252BrfPgtbR9lHYchrKBDbpbLmh11Ybf8p2%252FDPongPOnVkzu1I58iTsM2NCH%252F15JoPYYf%252BbLmBQSS7R%252FNdsItqIpHDNt9JZem5OEgeR38gf4HirVYcNMZj0ibkgdBLDuyTH%252BrphVlnNA6T4U9h7HhCeEz1uIGtUAF8mxVJ%252BXkVrb1DMGLxfjqc%253D%5C&X-Amz-Signature%5C=3d4e68e9d25f7e1f58318325dc9c1af3570f3d708093f8096f1488c597422751 HTTP/1.1
Host: <bucket-name>.s3.eu-west-1.amazonaws.com
User-Agent: rclone/v1.55.0
(signing information has been changed and filename is bogus so url is not valid)