createBucket errors throws in aws s3 during rclone copy

Help and Support
1

Getting createBucket error in aws s3 while doing rclone copy?

rclone version that I am using is v1.51.0

I have a script that runs daily and uploads some files on daily basis into new daily folders. Bucket folder structure will be like:

{bucket_name}/20220921/{file_name}_1
{bucket_name}/20220922/{file_name}_1
{bucket_name}/20220922/{file_name}_2

When I am trying to do rclone copy, somehow files are getting copied properly with correct folder structure but I am getting "createBucket" event triggered from AWS s3 saying someone is trying to create a bucket. Now I am not sure if I am not giving proper permission to the bucket or is there something else that I am missing here.

I have given following permission to the bucket that I am uploading:

  • "s3:PutObject",
  • "s3:GetObject"
  • "s3:ListObjects*"

P.S would it help if I give "s3:PutObjectAcl" permission to s3 bucket?

2

Rclone tries to create the bucket.

The logic for this has changed so you might find the current release v1.59.2 does something a little different.

There is also

  --s3-no-check-bucket     If set, don't attempt to check the bucket exists or create it

Which you can set in the config too.

3

I can't use --s3-no-check-bucket because I want to create daily folder and then upload the file. If I pass this flag, it is not creating the daily folder.

4

based on the your policy snippet, should be able to create a subfolder.
maybe there is an issue with some other part of the full policy?

rclone copy file.ext remote:zorktest/20220921 -vv --s3-no-check-bucket 
DEBUG : rclone: Version "v1.58.1" starting with parameters ["C:\\data\\rclone\\rclone.exe" "copy" "file.ext" "remote:zorktest/20220921" "-vv" "--s3-no-check-bucket"]
DEBUG : Creating backend with remote "remote:zorktest/20220921"
DEBUG : remote: detected overridden config - adding "{Dn7qA}" suffix to name
DEBUG : fs cache: renaming cache item "remote:zorktest/20220921" to be canonical "remote{Dn7qA}:zorktest/20220921"
DEBUG : file.ext: Need to transfer - File not found at Destination
DEBUG : file.ext: md5 = c7f5af9b93f5aa17934c84ad53fd2cea OK
INFO  : file.ext: Copied (new)

rclone tree remote:zorktest
/
└── 20220921
    └── file.ext
5

Folders will get auto created when files get put in them. s3 has no concept of folder!

6

Is --s3-no-check-bucket available in rclone v1.51.0?

7

1.51 is a few years old as you want to upgrade.

8

any idea how it can be done in v1.51.0? It is difficult to for me to update to newer version quickly.

9

Nope - As that version is years old so trying to troubleshoot that is not a great use as if we find something, most likely it was fixed in a later version.

The install is a single binary to download or use the install script and is done in a few seconds.

Install (rclone.org)

What part is difficult to upgrade?

10

Other team members also uses same version and it will take some time to get verified by everyone.

Any other solution that I can try. Maybe changing some permission of s3 bucket? In one of the bucket I have given s3:PutObjectAcl permission and it is working as expected and not throwing any 'createBucket' warning.

11

Why not just download and test yourself and if it works, you can then share with your team members to get verified.

Not much else to do other than upgrade unfortunately.

12

Upgrading the rclone version worked like a charm. Thanks a lot everyone

13

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.