Before I submit this on Github, I wanted to see if anyone else was having this issue.
After updating rclone from “v1.43.1” to “v1.45”, the service account credential JSON file is no longer read.
I am given the following error, which I understand to be the cryptic and uninformative output of Go’s JSON parser:
failed configuring Google Cloud Storage Service Account: error processing credentials: invalid character 'i' looking for beginning of value
Can you try 1.44 and the latest beta too?
Yes that looks like a JSON error. I wonder if the internals of the gcs module changed as that bit of rclone code hasn’t changed.
I can confirm that the same issue is present in the latest beta. I will try 1.44.
I just tried this and it works ok for me with the latest beta.
What does your service account file look like? Mine looks like this
{
"type": "service_account",
"project_id": "XXX",
"private_key_id": "XXX",
"private_key": "-----BEGIN PRIVATE KEY-----XXXEND PRIVATE KEY-----\n",
"client_email": "XXX",
"client_id": "XXX",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://accounts.google.com/o/oauth2/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/rclone-test%40rclone-org.iam.gserviceaccount.com"
}
I have not yet had a chance to try 1.44; however, if it is valid JSON it should not have a JSON error.
That is the structure of the credential file, down to (almost) a tee - my token_uri is different, though this is likely because I am using Cloud Identity rather than the usual google sign-in. Nonetheless, the credential file is unaltered and works with 1.43.
I also tested with 1.44 and can confirm that it does not work with 1.44
The error message might give you a clue as to what the JSON parser thinks is wrong with the file
This is what has changed in rclone in the google cloud storage backend from 1.43 to 1.44
$ git log v1.43..v1.44 backend/googlecloudstorage/
commit 6b8b9d19f399ec281548faec7154ad3fe61b2ee8
Author: Fabian Möller <fabianm88@gmail.com>
Date: Tue Sep 4 12:28:45 2018 +0200
googlecloudstorage: fix service_account_file been ignored - Fixes #2523
$ git show 6b8b9d19f399ec281548faec7154ad3fe61b2ee8
commit 6b8b9d19f399ec281548faec7154ad3fe61b2ee8
Author: Fabian Möller <fabianm88@gmail.com>
Date: Tue Sep 4 12:28:45 2018 +0200
googlecloudstorage: fix service_account_file been ignored - Fixes #2523
diff --git a/backend/googlecloudstorage/googlecloudstorage.go b/backend/googlecloudstorage/googlecloudstorage.go
index 1d74bccaf..2f47d9156 100644
--- a/backend/googlecloudstorage/googlecloudstorage.go
+++ b/backend/googlecloudstorage/googlecloudstorage.go
@@ -345,7 +345,7 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
}
// try loading service account credentials from env variable, then from a file
- if opt.ServiceAccountCredentials != "" && opt.ServiceAccountFile != "" {
+ if opt.ServiceAccountCredentials == "" && opt.ServiceAccountFile != "" {
loadedCreds, err := ioutil.ReadFile(os.ExpandEnv(opt.ServiceAccountFile))
if err != nil {
return nil, errors.Wrap(err, "error opening service account credentials file")
Which means that in rclone 1.43 rclone isn’t loading the credentials from a file at all. If I try 1.43 I get this
$ rclone-v1.43.1 lsd gcs-iam:
2018/12/05 10:05:38 Failed to create file system for "gcs-iam:": failed to configure Google Cloud Storage: empty token found - please run rclone config again
So I conjecture that you are loading the token either from the config file using this.
Service Account Credentials JSON blob
Leave blank normally.
Needed only if you want use SA instead of interactive login.
Can you show your config please (scrub any tokens from it) and describe how you are configuring rclone if it isn’t in the config file - thanks!
Actually, I am using service_account_file in the configuration file
I think I’ve figured it out.
service_account_credentials was set to ignore for some reason.
I believe that may have been due to rclone complaining about it not being present. Not sure.
That makes sense - you were working around the bug in 1.43 that was fixed in the commit above.
I'm getting the same issue. I am using rclone config to create the configuration but just can't seem to get it to work with SA JSON file. I am running it from a GCP VM so there is no ability to call out to a web browser for authentication with an account so I have to get the SA JSON part working.
arnold@rclonelinux1:~/rclone-v1.48.0-linux-amd64$ rclone listremotes
rcloneone:
arnold@rclonelinux1:~/rclone-v1.48.0-linux-amd64$ rclone ls rcloneone:
2019/08/14 19:31:58 Failed to create file system for "rcloneone:": failed configuring Google Cloud Storage Service Account: error processing credentials: invalid character 'r' looking for beginning of value
arnold@rclonelinux1:~/rclone-v1.48.0-linux-amd64$ rclone ls rcloneone:rclonestore
2019/08/14 19:32:25 Failed to create file system for "rcloneone:rclonestore": failed configuring Google Cloud Storage Service Account: error processing credentials: invalid character 'r' looking for beginning of value
arnold@rclonelinux1:~/rclone-v1.48.0-linux-amd64$
rcloneone]
type = google cloud storage
project_number = 482564684504
service_account_file = /home/arnold/rclone-v1.48.0-linux-amd64/rcloneone.json
service_account_credentials = rclonesa@rcloneone-249818.iam.gserviceaccount.com
object_acl = bucketOwnerFullControl
bucket_acl = private
location = us-central1
storage_class = REGIONAL
ERROR MSG
arnold@rclonelinux1:~/rclone-v1.48.0-linux-amd64$ rclone ls rclone2:
2019/08/14 20:48:46 Failed to create file system for "rclone2:": failed configuring Google Cloud Storage Service Account: error processing credentials: invalid character 'r' looking for beginning of value
I just can't seem to get past the above issue
Probably best to make a new post and use the bug/question template rather than bumping up an old post.
I think you might be hitting this issue too so check it out and if that doesn't solve it, a new post is in order 
You need either service_account_file as a pointer to the file, or service_account_credentials which should be the contents of that file, not both.