Rclone won't write to GCP storage bucket without read rights (--no-check-dest is set)

Help and Support
1

What is the problem you are having with rclone?

I want to write data to a gcp storage bucket from remote machines. These machines authenticate via a google service account which has permission to write into the bucket, but may not read from it.
To avoid reading from the bucket, I do set the --no-check-dest flag.

Problem: rclone still seems to try to read the bucket, because both copy and move fail with:

ERROR : xxx-StaticNodeLog: Failed to copy: failed to get bucket: googleapi: Error 403: xxx-bucket-writer-xx@project-xxx.iam.gserviceaccount.com does not have storage.objects.list access to bucket-xxx., forbidden

Also fails when replacing 'move' with 'copy'

What is your rclone version (output from rclone version)

1.51.0 (debian package)

Which OS you are using and how many bits (eg Windows 7, 64 bit)

ubuntu 18, 64 bits

Which cloud storage system are you using? (eg Google Drive)

GCP storage

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone move /var/tmp/gcp_test_a my_remote:my_bucket --no-check-dest

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

2020/03/20 15:41:13 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rclone" "move" "/var/tmp/gcp_test_a" "my_name-my_name-sec-lvl0:bucket-my_name-my_bucket_name" "--no-check-dest" "-vv"]
2020/03/20 15:41:13 DEBUG : Using config file from "/home/my_user/.config/rclone/rclone.conf"
2020/03/20 15:41:13 INFO  : GCS bucket bucket-my_name-my_bucket_name: Waiting for checks to finish
2020/03/20 15:41:13 INFO  : GCS bucket bucket-my_name-my_bucket_name: Waiting for transfers to finish
2020/03/20 15:41:13 ERROR : my_file: Failed to copy: failed to get bucket: googleapi: Error 403: sa-my_name-bucket-writer-@project-my_name.iam.gserviceaccount.com does not have storage.objects.list access to bucket-my_name-my_bucket_name., forbidden
2020/03/20 15:41:13 ERROR : my_file: Not deleting source as copy failed: failed to get bucket: googleapi: Error 403: sa-my_name-bucket-writer-@project-my_name.iam.gserviceaccount.com does not have storage.objects.list access to bucket-my_name-my_bucket_name., forbidden
2020/03/20 15:41:13 ERROR : Attempt 1/3 failed with 1 errors and: failed to get bucket: googleapi: Error 403: sa-my_name-bucket-writer-@project-my_name.iam.gserviceaccount.com does not have storage.objects.list access to bucket-my_name-my_bucket_name., forbidden
2020/03/20 15:41:13 INFO  : GCS bucket bucket-my_name-my_bucket_name: Waiting for checks to finish
2020/03/20 15:41:13 INFO  : GCS bucket bucket-my_name-my_bucket_name: Waiting for transfers to finish
2020/03/20 15:41:13 ERROR : my_file: Failed to copy: failed to get bucket: googleapi: Error 403: sa-my_name-bucket-writer-@project-my_name.iam.gserviceaccount.com does not have storage.objects.list access to bucket-my_name-my_bucket_name., forbidden
2020/03/20 15:41:13 ERROR : my_file: Not deleting source as copy failed: failed to get bucket: googleapi: Error 403: sa-my_name-bucket-writer-@project-my_name.iam.gserviceaccount.com does not have storage.objects.list access to bucket-my_name-my_bucket_name., forbidden
2020/03/20 15:41:13 ERROR : Attempt 2/3 failed with 1 errors and: failed to get bucket: googleapi: Error 403: sa-my_name-bucket-writer-@project-my_name.iam.gserviceaccount.com does not have storage.objects.list access to bucket-my_name-my_bucket_name., forbidden
2020/03/20 15:41:13 INFO  : GCS bucket bucket-my_name-my_bucket_name: Waiting for checks to finish
2020/03/20 15:41:13 INFO  : GCS bucket bucket-my_name-my_bucket_name: Waiting for transfers to finish
2020/03/20 15:41:13 ERROR : my_file: Failed to copy: failed to get bucket: googleapi: Error 403: sa-my_name-bucket-writer-@project-my_name.iam.gserviceaccount.com does not have storage.objects.list access to bucket-my_name-my_bucket_name., forbidden
2020/03/20 15:41:13 ERROR : my_file: Not deleting source as copy failed: failed to get bucket: googleapi: Error 403: sa-my_name-bucket-writer-@project-my_name.iam.gserviceaccount.com does not have storage.objects.list access to bucket-my_name-my_bucket_name., forbidden
2020/03/20 15:41:13 ERROR : Attempt 3/3 failed with 1 errors and: failed to get bucket: googleapi: Error 403: sa-my_name-bucket-writer-@project-my_name.iam.gserviceaccount.com does not have storage.objects.list access to bucket-my_name-my_bucket_name., forbidden
2020/03/20 15:41:13 Failed to move: failed to get bucket: googleapi: Error 403: sa-my_name-bucket-writer-@project-my_name.iam.gserviceaccount.com does not have storage.objects.list access to bucket-my_name-my_bucket_name., forbidden
2

What rclone does is list the bucket to make sure it exists.

So adding storage.objects.list to the permissions would make it work.

Is there any other way rclone can check the bucket exists with the permissions you've granted it?

An alternative might be to add a flag --gcs-no-check-bucket to see if it exists.

The code is here

If you wanted to experiment you could just add return nil above it and see if that fixes the problem. If so a flag would be very easy to add.

3

Thanks a lot for your fast reply Nick, am going to try that and update the thread within the next workdays!

1 Like
4

Hey Nick,
we have decided that listing the buckets is ok afterall, so move and copy work normally.
I have therefore not tried your suggestion (it would also involve creating a golang workspace), even though it makes perfect sense that this would work.

Thanks a lot for your help!

5

No problems :slight_smile:

6

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.