I've tried with 3 different bucket-acl types:
default ""
public-read-write
public-read
I've tried the --s3-no-check-bucket flag, --s3-server-side-encryption aws:kms, and --s3-v2-auth (separately of course) and they all produce the same error more or less. v2-auth gives a 400 status code instead of 403.
People with similar problems all seemed to have them fixed by new releases but I'm on the newest stable past where their issues were resolved so I'm a bit stumped.
What is your rclone version (output from rclone version)
1.53.3
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Ubuntu 20.04
Which cloud storage system are you using? (eg Google Drive)
AWS S3
The command you were trying to run (eg rclone copy /tmp remote:tmp)
given, that we both agree that post is not related to your post,
i would edit that post and remove that item so other forum members, will take spend time reading it.
Yes. There was a failure with the VM, and so I moved the machine over to a new VM. Don't know why this would be related, but in case it is. FWIW my google drive remote on the same machine continues to function normally.
I confirmed I'm using the same remote as the config I sent. In case the VM failure caused some type of corruption I also made a new remote with the same settings. Both exhibit the same behavior.
Upgraded to the beta, same outcome:
2021/01/08 10:33:28 DEBUG : rclone: Version "v1.54.0-beta.5040.71edc75ca" starting with parameters ["rclone" "copy" "./test2.txt" "remote:/XXXXXX/input/XXXXX/" "-vv" "--dump" "responses" "--retries" "1"]
2021/01/08 10:33:28 DEBUG : Creating backend with remote "./test2.txt"
2021/01/08 10:33:28 DEBUG : Using config file from "/home/brad/.config/rclone/rclone.conf"
2021/01/08 10:33:28 DEBUG : fs cache: adding new entry for parent of "./test2.txt", "/home/XXX/XXXXX"
2021/01/08 10:33:28 DEBUG : Creating backend with remote "remote:/XXXX/input/XXXXX/"
2021/01/08 10:33:28 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/01/08 10:33:28 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/01/08 10:33:29 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/01/08 10:33:29 DEBUG : HTTP REQUEST (req 0xc00055c400)
2021/01/08 10:33:29 DEBUG : HEAD /XXXXX/input/XXXXXX HTTP/1.1
Host: s3.eu-central-1.amazonaws.com
User-Agent: rclone/v1.54.0-beta.5040.71edc75ca
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210107T213328Z
rclone copy -vv ./test2.txt saXXX:XXXXX/XXXX/XXXXXX/ --dump responses --retries 1
2021/01/09 11:16:28 DEBUG : rclone: Version "v1.54.0-beta.5040.71edc75ca" starting with parameters ["rclone" "copy" "-vv" "./test2.txt" "saXXX:XXXX/XXX/XXXXX/" "--dump" "responses" "--retries" "1"]
2021/01/09 11:16:28 DEBUG : Creating backend with remote "./test2.txt"
2021/01/09 11:16:28 DEBUG : Using config file from "/home/XXX/.config/rclone/rclone.conf"
2021/01/09 11:16:28 DEBUG : fs cache: adding new entry for parent of "./test2.txt", "/home/XXX/XXXXX"
2021/01/09 11:16:28 DEBUG : Creating backend with remote "saXXX:XXXXX/XXXX/XXXXX/"
2021/01/09 11:16:28 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/01/09 11:16:28 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/01/09 11:16:28 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/01/09 11:16:28 DEBUG : HTTP REQUEST (req 0xc0004b0400)
2021/01/09 11:16:28 DEBUG : HEAD /XXXXXX/XXXX/XXXXXXXX HTTP/1.1
Host: s3.eu-central-1.amazonaws.com
User-Agent: rclone/v1.54.0-beta.5040.71edc75ca
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210108T221628Z
I assume so since this was working before. Its not my bucket, so I can't show the permissions file, but I will ask for it.
I assume you mean like a lsf. This is the output, works fine.
rclone lsf -vv saXXX: --dump responses
2021/01/09 12:22:24 DEBUG : rclone: Version "v1.54.0-beta.5040.71edc75ca" starting with parameters ["rclone" "lsf" "-vv" "saXXXX:" "--dump" "responses"]
2021/01/09 12:22:24 DEBUG : Using config file from "/home/XXXX/.config/rclone/rclone.conf"
2021/01/09 12:22:24 DEBUG : Creating backend with remote "saXXXX:"
2021/01/09 12:22:24 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/01/09 12:22:24 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/01/09 12:22:24 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/01/09 12:22:24 DEBUG : HTTP REQUEST (req 0xc00049c500)
2021/01/09 12:22:24 DEBUG : GET / HTTP/1.1
Host: s3.eu-central-1.amazonaws.com
User-Agent: rclone/v1.54.0-beta.5040.71edc75ca
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210108T232224Z
Accept-Encoding: gzip
I don't think the aws s3 cp does any HEAD requests to see if the file already exists or check it arrived properly. You can turn on debug to show the HTTP requests I think.
Can you GET files using rclone? So copy them from the bucket?