Rclone can't upload to S3 by Acronis Cyber Infrastructure

Maxim567890 · November 27, 2019, 1:59pm

What is the problem you are having with rclone?

Rclone can't upload to S3 by Acronis
s3cmd and cyberduck works but rclone doesn't

What is your rclone version (output from `rclone version`)

rclone v1.50.2-066-g8f33c932-beta

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Centos 7

Which cloud storage system are you using? (eg Google Drive)

S3 Acronis
https://dl.acronis.com/u/software-defined/html/AcronisCyberInfrastructure_3_admins_guide_en-US/exporting-storage/exporting-data-via-s3.html

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone copy /home/user/file buck:dir -vv

A log from the command with the `-vv` flag (eg output from `rclone -vv copy /tmp remote:tmp`)

rclone copy /home/zones/parse-to-cl.js zones:zones123 -vv --dump-headers --retries 1
2019/11/27 16:58:06 NOTICE: --dump-headers is obsolete - please use --dump headers instead
2019/11/27 16:58:06 DEBUG : rclone: Version "v1.50.2-066-g8f33c932-beta" starting with parameters ["rclone" "copy" "/home/zones/parse-to-cl.js" "zones:zones123" "-vv" "--dump-headers" "--retries" "1"]
2019/11/27 16:58:06 DEBUG : Using config file from "/home/zones/.config/rclone/rclone.conf"
2019/11/27 16:58:06 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/11/27 16:58:06 DEBUG : HTTP REQUEST (req 0xc00058c500)
2019/11/27 16:58:06 DEBUG : HEAD /zones123/parse-to-cl.js HTTP/1.1
Host: s3.active.domains
User-Agent: rclone/v1.50.2-066-g8f33c932-beta
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20191127T135806Z

2019/11/27 16:58:06 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/11/27 16:58:06 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/11/27 16:58:06 DEBUG : HTTP RESPONSE (req 0xc00058c500)
2019/11/27 16:58:06 DEBUG : HTTP/1.1 404 Not Found
Connection: close
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token,X-Amz-User-Agent,X-Amz-Object-Map,Range,Content-MD5,if-match
Access-Control-Allow-Methods: GET, DELETE, POST, PUT, OPTIONS, HEAD
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Content-Length,ETag,X-Amz-Meta-Custom-Header,X-Amz-Notary-Provider,X-Amz-Cert,X-Amz-Notary-Hash,X-Amz-Notary-Protocol
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 27 Nov 2019 13:58:06 GMT
Server: nginx
X-Amz-Err-Code: NoSuchKey
X-Amz-Err-Message: The specified key does not exist.
X-Amz-Req-Time-Micros: 501
X-Amz-Request-Id: 80000000000000150005ce27d18aacee

2019/11/27 16:58:06 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/11/27 16:58:06 DEBUG : parse-to-cl.js: Need to transfer - File not found at Destination
2019/11/27 16:58:06 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/11/27 16:58:06 DEBUG : HTTP REQUEST (req 0xc00014aa00)
2019/11/27 16:58:06 DEBUG : PUT /zones123 HTTP/1.1
Host: s3.active.domains
User-Agent: rclone/v1.50.2-066-g8f33c932-beta
Content-Length: 0
Authorization: XXXX
X-Amz-Acl: private
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20191127T135806Z
Accept-Encoding: gzip

2019/11/27 16:58:06 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/11/27 16:58:06 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/11/27 16:58:06 DEBUG : HTTP RESPONSE (req 0xc00014aa00)
2019/11/27 16:58:06 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token,X-Amz-User-Agent,X-Amz-Object-Map,Range,Content-MD5,if-match
Access-Control-Allow-Methods: GET, DELETE, POST, PUT, OPTIONS, HEAD
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Content-Length,ETag,X-Amz-Meta-Custom-Header,X-Amz-Notary-Provider,X-Amz-Cert,X-Amz-Notary-Hash,X-Amz-Notary-Protocol
Connection: keep-alive
Date: Wed, 27 Nov 2019 13:58:06 GMT
Location: /zones123
Server: nginx
X-Amz-Req-Time-Micros: 61456
X-Amz-Request-Id: 80000000000000160000c2983aabe11e

2019/11/27 16:58:06 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/11/27 16:58:06 INFO : S3 bucket zones123: Bucket "zones123" created with ACL "private"
2019/11/27 16:58:06 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/11/27 16:58:06 DEBUG : HTTP REQUEST (req 0xc00058c800)
2019/11/27 16:58:06 DEBUG : PUT /zones123/parse-to-cl.js?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=c49b53144e68fabePH6Z%2F20191127%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20191127T135806Z&X-Amz-Expires=900&X-Amz-SignedHeaders=content-md5%3Bcontent-type%3Bhost%3Bx-amz-acl%3Bx-amz-meta-mtime&X-Amz-Signature=fa6b441c2cfc1e80dcc4deb27f2b90a10474e5d5beb987c3db523694e92e3c3b HTTP/1.1
Host: s3.active.domains
User-Agent: rclone/v1.50.2-066-g8f33c932-beta
Content-Length: 25447
content-md5: iC6RjqxfhqujW2FT2jGy5g==
content-type: application/javascript
x-amz-acl: private
x-amz-meta-mtime: 1561201423
Accept-Encoding: gzip

2019/11/27 16:58:06 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/11/27 16:58:06 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/11/27 16:58:06 DEBUG : HTTP RESPONSE (req 0xc00058c800)
2019/11/27 16:58:06 DEBUG : HTTP/1.1 400 Bad Request
Transfer-Encoding: chunked
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token,X-Amz-User-Agent,X-Amz-Object-Map,Range,Content-MD5,if-match
Access-Control-Allow-Methods: GET, DELETE, POST, PUT, OPTIONS, HEAD
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Content-Length,ETag,X-Amz-Meta-Custom-Header,X-Amz-Notary-Provider,X-Amz-Cert,X-Amz-Notary-Hash,X-Amz-Notary-Protocol
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 27 Nov 2019 13:58:06 GMT
Server: nginx
X-Amz-Err-Code: InvalidArgument
X-Amz-Err-Message: Unsupported Authorization Type
X-Amz-Req-Time-Micros: 18
X-Amz-Request-Id: 80000000000000150005ce27d18ca8bb

2019/11/27 16:58:06 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/11/27 16:58:06 ERROR : parse-to-cl.js: Failed to copy: s3 upload: 400 Bad Request: <?xml version="1.0" encoding="UTF-8"?>InvalidArgumentUnsupported Authorization Type
2019/11/27 16:58:06 ERROR : Attempt 1/1 failed with 1 errors and: s3 upload: 400 Bad Request: <?xml version="1.0" encoding="UTF-8"?>InvalidArgumentUnsupported Authorization Type
2019/11/27 16:58:06 Failed to copy: s3 upload: 400 Bad Request: <?xml version="1.0" encoding="UTF-8"?>InvalidArgumentUnsupported Authorization Type

ncw · November 27, 2019, 3:44pm

This looks like the root cause...

You appear to have set rclone up with v4 auth (the default). Does your version of Acronis support v4 auth?

Rclone can do v2 auth too so you could try this flag

  --s3-v2-auth    If true use v2 authentication.

If that doesn't work can you post your config file (XXX out sensitive info)

Maxim567890 · November 27, 2019, 4:04pm

[zones]
type = s3
env_auth = false
access_key_id = XXX
secret_access_key = XXX
endpoint = s3.my.domains
location_constraint =
acl = private
server_side_encryption =
storage_class =
s3-provider = Other

s3-v2-auth

With v4 auth I can auth and create buckets/directories etc.. But I can't upload. (PUT)
after switching to v2 auth acronis throws another error and I can't even ls/lsd with this option.
2019/11/27 19:02:00 DEBUG : HTTP/1.1 403 Forbidden
Connection: close
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token,X-Amz-User-Agent,X-Amz-Object-Map,Range,Content-MD5,if-match
Access-Control-Allow-Methods: GET, DELETE, POST, PUT, OPTIONS, HEAD
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Content-Length,ETag,X-Amz-Meta-Custom-Header,X-Amz-Notary-Provider,X-Amz-Cert,X-Amz-Notary-Hash,X-Amz-Notary-Protocol
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 27 Nov 2019 16:02:00 GMT
Server: nginx
X-Amz-Err-Code: AccessDenied
X-Amz-Err-Message: AWS authentication requires a valid Date or x-amz-date header
X-Amz-Req-Time-Micros: 19
X-Amz-Request-Id: 80000000000000150005ce298c9e0a81

Maxim567890 · November 27, 2019, 4:07pm

I've checked the cyberduck s3 communcation dump and it seems that the main difference with rclone, that it sends PUT parameters as request body but rclone sends them as URL params.

ncw · November 27, 2019, 4:22pm

Ah, if that is the case then using --s3-upload-cutoff 0 may fix the problem. This will upload all files as multipart uploads and these will use PUT requests with the parameters inline.

  --s3-upload-cutoff SizeSuffix   Cutoff for switching to chunked upload (default 200M)

system · November 30, 2019, 4:22pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.