Using AWS S3 with "Block public access (bucket settings)"

Help and Support
1

I have been using rclone for a month now to sync daily logs, and everything worked fine.

I recently enabled "Block public access (bucket settings)" on my S3 bucket Permissions, and that
broke rclone

/usr/bin/rclone -vv --config /opt/r2.conf --log-file /tmp/rclone2.log -P copy /tmp/hello.txt wj2:mytestbucket/

Any suggestions ?
Thanks

020/11/03 06:38:16 DEBUG : rclone: Version "v1.53.2" starting with parameters ["/usr/bin/rclone" "-vv" "--config" "/opt/r2.conf" "--log-file" "/tmp/rclone2.log" "-P" "copy" "/tmp/a4.txt" "wj2: mytestbucket/"]
2020/11/03 06:38:16 DEBUG : Creating backend with remote "/tmp/a4.txt"
2020/11/03 06:38:16 DEBUG : Using config file from "/opt/r2.conf"
2020/11/03 06:38:16 DEBUG : fs cache: adding new entry for parent of "/tmp/a4.txt", "/tmp"
2020/11/03 06:38:16 DEBUG : Creating backend with remote "wj2: mytestbucket/"
2020/11/03 06:38:16 DEBUG : fs cache: renaming cache item "wj2: mytestbucket/" to be canonical "wj2: mytestbucket"
2020/11/03 06:38:16 DEBUG : a4.txt: Need to transfer - File not found at Destination
2020/11/03 06:38:16 ERROR : a4.txt: Failed to copy: AccessDenied: Access Denied
status code: 403, request id: 64E2936CE9C7C7F4, host id: FEt2J0rT+Wc8hyO1Js/Is0+FAmkQN5HGd2RssvgBj6EyHcRBGzy2r2yJEdgv/bG0Ij7lxfWB814=
2020/11/03 06:38:16 ERROR : Attempt 1/3 failed with 1 errors and: AccessDenied: Access Denied
status code: 403, request id: 64E2936CE9C7C7F4, host id: FEt2J0rT+Wc8hyO1Js/Is0+FAmkQN5HGd2RssvgBj6EyHcRBGzy2r2yJEdgv/bG0Ij7lxfWB814=
2020/11/03 06:38:16 DEBUG : a4.txt: Need to transfer - File not found at Destination
2020/11/03 06:38:16 ERROR : a4.txt: Failed to copy: AccessDenied: Access Denied
status code: 403, request id: 06B0FEDDFBE15F54, host id: IuTYxhAWYTQdQxIdRBqqPbtT/S7k1lPZchpnf1ZFNaJlfq2So3vvmrF04/pLMAjnPhTMGeWxulc=
2020/11/03 06:38:16 ERROR : Attempt 2/3 failed with 1 errors and: AccessDenied: Access Denied
status code: 403, request id: 06B0FEDDFBE15F54, host id: IuTYxhAWYTQdQxIdRBqqPbtT/S7k1lPZchpnf1ZFNaJlfq2So3vvmrF04/pLMAjnPhTMGeWxulc=
2020/11/03 06:38:16 DEBUG : a4.txt: Need to transfer - File not found at Destination
2020/11/03 06:38:16 ERROR : a4.txt: Failed to copy: AccessDenied: Access Denied
status code: 403, request id: 5D515A3DFF38E377, host id: gS+trZiCrMuEPFxaq1G+26HiODPyAujOVPJUIuScp715omeqB+xCCbyXV8//1v1FolCZ05i6cm8=
2020/11/03 06:38:16 ERROR : Attempt 3/3 failed with 1 errors and: AccessDenied: Access Denied
status code: 403, request id: 5D515A3DFF38E377, host id: gS+trZiCrMuEPFxaq1G+26HiODPyAujOVPJUIuScp715omeqB+xCCbyXV8//1v1FolCZ05i6cm8=
2020/11/03 06:38:16 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 0.1s

2020/11/03 06:38:16 DEBUG : 4 go routines active
2020/11/03 06:38:16 Failed to copy: AccessDenied: Access Denied
status code: 403, request id: 5D515A3DFF38E377, host id: gS+trZiCrMuEPFxaq1G+26HiODPyAujOVPJUIuScp715omeqB+xCCbyXV8//1v1FolCZ05i6cm8=

2

Try adding --s3-no-check-bucket (https://rclone.org/s3/#s3-no-check-bucket) to your command and trying again.

3

there is a space character in the remote path "wj2: mytestbucket/"
rclone seems to accept it.

4

Thanks, but I just tried that and it did not work.
I have a hack now to disable 'block-public-access' but I do not want to do that
aws s3api put-public-access-block --bucket xyz --public-access-block-configuration "BlockPublicAcls=false,IgnorePublicAcls=false,BlockPublicPolicy=false,RestrictPublicBuckets=false"

2020/11/05 03:18:07 DEBUG : rclone: Version "v1.53.2" starting with parameters ["/usr/bin/rclone" "-vv" "--s3-no-check-bucket" "--config" "/opt/rclone.conf" "--log-file" "/tmp/rclone2.log" "--progress" "copy" "/tmp/A" "test1:xyz/"]
2020/11/05 03:18:07 DEBUG : Creating backend with remote "/tmp/A"
2020/11/05 03:18:07 DEBUG : Using config file from "/opt/rclone.conf"
2020/11/05 03:18:07 DEBUG : Creating backend with remote "test1:xyz/"
2020/11/05 03:18:07 DEBUG : fs cache: renaming cache item "test1:xyz/" to be canonical "test1:xyz"
2020/11/05 03:18:07 DEBUG : S3 bucket xyz: Waiting for checks to finish
2020/11/05 03:18:07 DEBUG : S3 bucket xyz: Waiting for transfers to finish
2020/11/05 03:18:07 ERROR : B/C/D/b.txt: Failed to copy: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
AccessDeniedAccess DeniedA140FB26D5959549dUeco3YAwWTiUFEOQwMKQgvXD0NtPFQleFLHOu0EY/yM93m76YpsDYVRSCkveJTbTehT/0OFQVc=
2020/11/05 03:18:07 ERROR : B/C/D/a.txt: Failed to copy: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
AccessDeniedAccess DeniedEEE5536D080DE740Gd+3qDMZyEJloW+VK66QGeUwymrdmG8Nt8ZZpBlRYwrWcJVI0O1A2o5hlBR9fAdcSCHFnHWcTAM=
2020/11/05 03:18:07 INFO : There was nothing to transfer
2020/11/05 03:18:07 ERROR : Attempt 1/3 failed with 2 errors and: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
AccessDeniedAccess DeniedEEE5536D080DE740Gd+3qDMZyEJloW+VK66QGeUwymrdmG8Nt8ZZpBlRYwrWcJVI0O1A2o5hlBR9fAdcSCHFnHWcTAM=
2020/11/05 03:18:08 DEBUG : S3 bucket xyz: Waiting for checks to finish
2020/11/05 03:18:08 DEBUG : S3 bucket xyz: Waiting for transfers to finish
2020/11/05 03:18:08 ERROR : B/C/D/a.txt: Failed to copy: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
AccessDeniedAccess Denied16CF5A9C8C79F7E7yAPsG2ECZbuIBhmy5zZXX9Xld03CMo4snf9AcZmHvH2irbxT8UFeN/QTCAdiopRstmKOT89vtTQ=
2020/11/05 03:18:08 ERROR : B/C/D/b.txt: Failed to copy: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
AccessDeniedAccess Denied7783C1351416F545+i5vSnCnYmt76e7A2M60WCjeewVM3m/88tR4MSOf82D4LVlkiSkLfKhYpmw1Uj3wF4Ne7J66emo=
2020/11/05 03:18:08 INFO : There was nothing to transfer
2020/11/05 03:18:08 ERROR : Attempt 2/3 failed with 2 errors and: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
AccessDeniedAccess Denied7783C1351416F545+i5vSnCnYmt76e7A2M60WCjeewVM3m/88tR4MSOf82D4LVlkiSkLfKhYpmw1Uj3wF4Ne7J66emo=
2020/11/05 03:18:08 DEBUG : S3 bucket xyz: Waiting for checks to finish
2020/11/05 03:18:08 DEBUG : S3 bucket xyz: Waiting for transfers to finish
2020/11/05 03:18:08 ERROR : B/C/D/b.txt: Failed to copy: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
AccessDeniedAccess DeniedAA1707CACEBC522ExkF8b1SP7voX9fZLiX2g2I0b2KH/TRtvK9ApwtEBsyeGPjXn9qg5t9Nj34f/69BkUwIqma93eXc=
2020/11/05 03:18:08 ERROR : B/C/D/a.txt: Failed to copy: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
AccessDeniedAccess DeniedBFE2CA7C4F925F887vkbSNQSMpv4JMQvHisI/Xrdne3cUmMxTuKWy+0veTlHb9TmG99ePQrZb8CxNKFbp0PqcWzUcBk=
2020/11/05 03:18:08 INFO : There was nothing to transfer
2020/11/05 03:18:08 ERROR : Attempt 3/3 failed with 2 errors and: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
AccessDeniedAccess DeniedBFE2CA7C4F925F887vkbSNQSMpv4JMQvHisI/Xrdne3cUmMxTuKWy+0veTlHb9TmG99ePQrZb8CxNKFbp0PqcWzUcBk=
2020/11/05 03:18:08 INFO :
Transferred: 36 / 36 Bytes, 100%, 529 Bytes/s, ETA 0s
Errors: 2 (retrying may help)
Elapsed time: 0.6s

2020/11/05 03:18:08 DEBUG : 4 go routines active
2020/11/05 03:18:08 Failed to copy with 2 errors: last error was: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
AccessDeniedAccess DeniedBFE2CA7C4F925F887vkbSNQSMpv4JMQvHisI/Xrdne3cUmMxTuKWy+0veTlHb9TmG99ePQrZb8CxNKFbp0PqcWzUcBk=

5

I had this same error. I think it went away when I stopped trying to copy a specific file and instead used a source directory. Ex the failure happened when source was /mnt/path/file.png, but worked when it was /mnt/path.

For what its worth my S3 bucket is locked down completely, no public access at all and I can now write to it.

6

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.