Upload files through Pre-signed URLs

Help and Support
1

What is the problem you are having with rclone?

Want to upload files through Pre-signed URLs

What is your rclone version (output from rclone version)

rclone-v1.55.1

Which OS you are using and how many bits (eg Windows 7, 64 bit)

  • os/type: darwin

  • os/arch: amd64

Which cloud storage system are you using? (eg Google Drive)

Minio with azure gateway

The command you were trying to run (eg rclone copy /tmp remote:tmp)

./rclone copy /Users/pk/Downloads/testtt :http: --http-url "http://localhost:9000/test2/uploadfromgo?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=a1testfiles%2F20210513%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210513T130940Z&X-Amz-Expires=60000&X-Amz-SignedHeaders=host&X-Amz-Signature=3e43c758b63a32fedf67a54c6e319da366a696d31ccb7bd82a89bca2dfd6bf26" -vvv --no-traverse

The rclone config contents with secrets removed.

using dynamic config :http: with blank config in file.

A log from the command with the -vv flag

2021/05/13 11:58:32 NOTICE: Config file "/Users/pk/.config/rclone/rclone.conf" not found - using defaults
2021/05/13 11:58:32 DEBUG : rclone: Version "v1.55.1" starting with parameters ["./rclone" "copy" "/Users/pk/Downloads/testtt" ":http:" "--http-url" "http://localhost:9000/test2/uploadfromgo?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=a1testfiles%2F20210513%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210513T130940Z&X-Amz-Expires=60000&X-Amz-SignedHeaders=host&X-Amz-Signature=3e43c758b63a32fedf67a54c6e319da366a696d31ccb7bd82a89bca2dfd6bf26" "-vvv" "--no-traverse"]
2021/05/13 11:58:32 DEBUG : Creating backend with remote "/Users/pk/Downloads/testtt"
2021/05/13 11:58:32 DEBUG : fs cache: adding new entry for parent of "/Users/pk/Downloads/testtt", "/Users/pk/Downloads"
2021/05/13 11:58:32 DEBUG : Creating backend with remote ":http:"
2021/05/13 11:58:32 DEBUG : :http: detected overridden config - adding "{PQ27K}" suffix to name
2021/05/13 11:58:32 DEBUG : fs cache: renaming cache item ":http:" to be canonical ":http{PQ27K}:"
2021/05/13 11:58:32 ERROR : Attempt 1/3 failed with 1 errors and: failed to stat: HTTP Error 403: 403 Forbidden
2021/05/13 11:58:32 ERROR : Attempt 2/3 failed with 1 errors and: failed to stat: HTTP Error 403: 403 Forbidden
2021/05/13 11:58:32 ERROR : Attempt 3/3 failed with 1 errors and: failed to stat: HTTP Error 403: 403 Forbidden

2

I added --http-no-head flag to avoid the stat call in rclone that uses head request. I now see this error
2021/05/13 22:02:44 ERROR : testtt: Failed to copy: http remotes are read only

Does rclone support put request to pre-signed url? this would greatly improve the performance with multiple transfers/multipart etc available in rclone than simple curl.

I'm using copyurl to download the file using pre-signed url which works fine.

3

No it doesn't at the moment.

That could maybe be an extension to rclone copyurl or maybe to the http backend.

4

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.