Rclone not loading Google service account credentials


#1

Before I submit this on Github, I wanted to see if anyone else was having this issue.

After updating rclone from “v1.43.1” to “v1.45”, the service account credential JSON file is no longer read.

I am given the following error, which I understand to be the cryptic and uninformative output of Go’s JSON parser:

failed configuring Google Cloud Storage Service Account: error processing credentials: invalid character 'i' looking for beginning of value


#2

:frowning:

Can you try 1.44 and the latest beta too?

Yes that looks like a JSON error. I wonder if the internals of the gcs module changed as that bit of rclone code hasn’t changed.


#3

I can confirm that the same issue is present in the latest beta. I will try 1.44.


#4

I just tried this and it works ok for me with the latest beta.

What does your service account file look like? Mine looks like this

{
  "type": "service_account",
  "project_id": "XXX",
  "private_key_id": "XXX",
  "private_key": "-----BEGIN PRIVATE KEY-----XXXEND PRIVATE KEY-----\n",
  "client_email": "XXX",
  "client_id": "XXX",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://accounts.google.com/o/oauth2/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/rclone-test%40rclone-org.iam.gserviceaccount.com"
}

#5

I have not yet had a chance to try 1.44; however, if it is valid JSON it should not have a JSON error.


#6

That is the structure of the credential file, down to (almost) a tee - my token_uri is different, though this is likely because I am using Cloud Identity rather than the usual google sign-in. Nonetheless, the credential file is unaltered and works with 1.43.

I also tested with 1.44 and can confirm that it does not work with 1.44


#7

The error message might give you a clue as to what the JSON parser thinks is wrong with the file

This is what has changed in rclone in the google cloud storage backend from 1.43 to 1.44

$ git log v1.43..v1.44 backend/googlecloudstorage/
commit 6b8b9d19f399ec281548faec7154ad3fe61b2ee8
Author: Fabian Möller <fabianm88@gmail.com>
Date:   Tue Sep 4 12:28:45 2018 +0200

    googlecloudstorage: fix service_account_file been ignored - Fixes #2523
$ git show 6b8b9d19f399ec281548faec7154ad3fe61b2ee8
commit 6b8b9d19f399ec281548faec7154ad3fe61b2ee8
Author: Fabian Möller <fabianm88@gmail.com>
Date:   Tue Sep 4 12:28:45 2018 +0200

    googlecloudstorage: fix service_account_file been ignored - Fixes #2523

diff --git a/backend/googlecloudstorage/googlecloudstorage.go b/backend/googlecloudstorage/googlecloudstorage.go
index 1d74bccaf..2f47d9156 100644
--- a/backend/googlecloudstorage/googlecloudstorage.go
+++ b/backend/googlecloudstorage/googlecloudstorage.go
@@ -345,7 +345,7 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
        }
 
        // try loading service account credentials from env variable, then from a file
-       if opt.ServiceAccountCredentials != "" && opt.ServiceAccountFile != "" {
+       if opt.ServiceAccountCredentials == "" && opt.ServiceAccountFile != "" {
                loadedCreds, err := ioutil.ReadFile(os.ExpandEnv(opt.ServiceAccountFile))
                if err != nil {
                        return nil, errors.Wrap(err, "error opening service account credentials file")

Which means that in rclone 1.43 rclone isn’t loading the credentials from a file at all. If I try 1.43 I get this

$ rclone-v1.43.1 lsd gcs-iam:
2018/12/05 10:05:38 Failed to create file system for "gcs-iam:": failed to configure Google Cloud Storage: empty token found - please run rclone config again

So I conjecture that you are loading the token either from the config file using this.

–gcs-service-account-credentials

Service Account Credentials JSON blob
Leave blank normally.
Needed only if you want use SA instead of interactive login.

  • Config: service_account_credentials
  • Env Var: RCLONE_GCS_SERVICE_ACCOUNT_CREDENTIALS
  • Type: string
  • Default: “”

Can you show your config please (scrub any tokens from it) and describe how you are configuring rclone if it isn’t in the config file - thanks!


#8

Actually, I am using service_account_file in the configuration file


#9

I think I’ve figured it out.

service_account_credentials was set to ignore for some reason.

I believe that may have been due to rclone complaining about it not being present. Not sure.


#10

That makes sense - you were working around the bug in 1.43 that was fixed in the commit above.