Setup Rclone config for GCS using ENV variables

What is the problem you are having with rclone?

I would like to use rclone to sync from a local S3 storage to a Google Cloud Storage bucket. The local config seems to work and I need help configuring the target (GCS).

What is your rclone version (output from rclone version)

Latest Official provided rclone Docker image

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Latest Official provided rclone Docker image will be running on Kubernetes. ATM it's running on my local machine (Ubuntu 18.04).

Which cloud storage system are you using? (eg Google Drive)

Local: MinIO S3
Remote: Google Cloud Storage (GCS)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

sudo docker run --env-file ./rclone.env rclone/rclone:latest sync SOURCES3:rclone-test-onprem TARGETS3:rclone-test-duryto

The rclone config contents with secrets removed.

RCLONE_CONFIG_SOURCES3_TYPE=s3
RCLONE_CONFIG_SOURCES3_ACCESS_KEY_ID=removed
RCLONE_CONFIG_SOURCES3_SECRET_ACCESS_KEY=removed
RCLONE_CONFIG_SOURCES3_ENDPOINT=removed

RCLONE_CONFIG_TARGETS3_TYPE=gcs
RCLONE_CONFIG_TARGETS3_CLIENT_ID=removed
RCLONE_CONFIG_TARGETS3_CLIENT_SECRET=removed
RCLONE_CONFIG_TARGETS3_PROJECT_NUMBER=removed

A log from the command with the -vv flag

2020/11/16 09:24:01 DEBUG : rclone: Version "v1.53.2" starting with parameters ["rclone" "sync" "-vv" "SOURCES3:rclone-test-onprem" "TARGETS3:rclone-test-duryto"]
2020/11/16 09:24:01 DEBUG : Creating backend with remote "SOURCES3:rclone-test-onprem"
2020/11/16 09:24:01 NOTICE: Config file "/config/rclone/rclone.conf" not found - using defaults
2020/11/16 09:24:01 DEBUG : Creating backend with remote "TARGETS3:rclone-test-duryto"
2020/11/16 09:24:01 Failed to create file system for "TARGETS3:rclone-test-duryto": failed to configure Google Cloud Storage: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.

I guess I'm using the ENV variables for configuring the GCS remote the wrong way. I don't want to use the JSON file for the Service Account, so I got an ACCESS/SECRET key combination to access the Storage bucket.

Note that these are for the oauth not for authentication.

I didn't know that was possible with GCS.

rclone doesn't currently support it.

Note that you can put the entire JSON credentials in --gcs-service-account-credentials '{json blob here}'

In fact GCS lets you create ACCESS/SECRET combinations for situations when you can't or won't use the JSON credentials.

Nevertheless, thank you for your help - maybe using this the above credentials could be taken in consideration for a future rclone release :slight_smile:

I didn't know that...

It probably wouldn't be too hard if you wanted to have a go?

Either way can you please make a new issue on github about this - thanks!

I've opened the issue on GitHub. I don't know if I'll have time this week, but I'll read into the code of rclone as soon as I have time.

1 Like

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.