Trying to accomplish backup into Google Drive

I have 36403 files I am trying to get backed up into my Google Drive. I have followed the steps at (under “Making your own client_id”) to make my own client ID and client secret. However, I never seem to get more than about 30% of the way through the backup set before rclone starts giving me 401 errors:

Response: {
  "error" : "invalid_client",
  "error_description" : "The OAuth client was not found."

If I login to the Google API console, I can still see the client in there. If I do rclone config and ask it to generate a new token - WITHOUT saying yes to editing the client ID and client secret - it will generate a token URL (this is a headless box) that gives me an HTTP 400 error.

If I edit the client ID and client secret, and simply copy/paste the previous values as displayed by rclone, the token URL will work. I can then issue rclone -v copy /mnt/zpool1/business crypt:/business, and it will run fine for a few hours, and then repeats the whole process from the beginning.

I can totally understand that I’m probably hitting some Google Rate Limit, but in my case I have the ownership of the Google Business account being used, so I could create more users/API keys, if rclone were able to round-robin between them, thus preventing me hitting the API limit. However, I can’t find any indication that this is a possible option; also, it doesn’t really explain to me why I have to issue an rclone config and fake rclone into thinking I am editing the client ID and client secret in order to get a new, valid token. Can anyone help me understand what I’m doing wrong there? Also, is there any plan or mechanism in place to leverage multiple client IDs to get past the API rate limits?

Can you check your config file at the point when it gives 401 errors and see if it looks sensible, ie still has the client_id and client_secret it?

Yes, the .rclone.conf looks the same during/after 401 errors as it did immediately before I started getting them.

It might be because the token refresh isn’t working properly.

One thing to check is, is the time correct on the server? And does the expiry time look correct on the token (look in the config file).

Next time you get the 401s can you try editing the expiry time so it is back in the past (subtract one from the year is easiest) then retry.

Can you do a run with -vv and paste the log where I can see it?

chrony is configured on the box and seems to be working correctly (if the clock is off, it’s by less than a full second); however, when looking through the logs with -vv, I cannot see anything that even LOOKS like an attempt to refresh the token. I have generated a new token and am running it with -vv now to share with you.


And when it expires try this too

I changed "expiry":"2017-08-02T21:16:21.169188519-06:00" to "expiry":"2015-08-02T21:16:21.169188519-06:00", and it immediately returns a 401. Sent you a private message with a link to the logs (using -vv --dump-headers --log-file).

that is strange, it just says the client_id is invalid. Can you check the config file at this point after rclone has stopped to see if the client_id and client_secret look OK in it?

The client_id and secret in .rclone.conf never appear to be changed - in fact, the method I have for “fixing” the problem is to run rclone config and copy/paste the values it shows. When prompted to choose whether or not I need to edit the client_id and client_secret, if I say “No”, the token refresh link that is generated gives a 400 error in Google. However, if I edit the values and just REPASTE the exact value, it generates a valid token URL and everything starts working again (until the next timeout).

When watching .rclone.conf with inotifywait, I see only the following reported repeatedly:

.rclone.conf OPEN
.rclone.conf ACCESS

I am not seeing any kind of WRITE event which would indicate it is trying to update the token.

Can you see what changes before and after in the config file when you do the re-authenticate?

Any chance you could do that again but with --dump-bodies -vv? This will definitely have confidential stuff in though! Maybe you could examine it to see if the client_id is correct in the POST.

Using --dump-bodies showed there was an encoded null byte at the end of the client_id line; I deleted the entire .rclone.conf and regenerated it, and it has been running for about 18 hours now with no problem. No idea where the null byte came from or how it kept coming back through copy/paste, but thanks for sticking with me and showing me how to solve it. :slight_smile:

Maybe rclone should do a trim on the values so a cut / paste doesn’t introduce spaces or anything else. @ncw Worth making an issue to trim the values?

Glad we figured it in the end! Where did that null byte come from? How exactly did you cut and paste the id to get a null byte on the end?

That is probably a good idea!