Jottacloud: how well is it working with rclone?

Hi @ivandeex, and sorry for the delay in getting back to you -- lots of stuff happening in 'RL', only today I was able to get back to this.

I too have experienced Jottacloud token expiration, in the context of having the same remotes configured in more than one machine simultaneously. The symptom being that trying to access the JC remotes in a machine where they weren't used recently, results in errors like:

[REDACTED] couldn't get customer info: Get "https://api.jottacloud.com/account/v1/customer": couldn't fetch token - maybe it has expired? - refresh with "rclone config reconnect [REDACTED]:": oauth2: cannot fetch token: 400 Bad Request
Response: {"error":"invalid_grant","error_description":"Stale token"}

What I've been doing to re-establish access is to copy the relevant parts of ~/.rclone.conf from the machine which last accessed it (and where it is working), to the ones which were getting the above error. Right now I do it by manually grepping the config file in that machine for the appropriate lines, and sending the result to a location where it can be downloaded by the other machines and then incorporated into their config file in place of the ones with the expired tokens.

In this context, your --config-to addition is appreciated but not really practical, the reason being that the "machine who most recently accessed JC" has a public address on the internet (it runs on a server of mine), but the others don't (being behind NAT, etc). something that would work much better would be the other way around, eg a --config-from option where these "other" rclones would then contact the "server" rclone and 'pull' the configuration, instead of having it 'pushed' to them.

Thanks for your efforts and reporting, much appreciated!

Cheers,
--Durval.

With push mode we can ensure that the pushing node is updating the config most recently. With pull mode we cannot give such guarantee without timestamping config values which is not possible without breaking the config format. Due to that I don't think I will ever develop --config-from.

1 Like

Hi @ivandeex, and thanks for your continued responses.

I think I understand your explanation re: push vs pull (ie, --config-to vs --config-from) , but the way I'm experiencing the issue, this is not what happens. To wit: my 'central' machine is always the one that has accessed the JC remote last (the reason being it runs processes that go through the remote periodically). The other machines only need to pull the updated tokens when they experience the above error (which is about once a day in my usage), and when I merge the updated config lines in their ~/.rclone.conf, the remote starts working perfectly. I've done that a number of times, and it's 100% repeatable.

Thanks for being frank re: implementing --config-from, I understand and of course respect your stance on this. It's no big trouble, I think I will implement it locally as part of a script that will always run before rclone proper and then pull/merge the necessary lines.

Thanks again,
-- Durval.

Pulling can't be implemented in general since it needs a way to arbitrage between tokens pulled on demand from external rclone instance, tokens requested from the cloud and tokens recorded in rclone config. However it's possible if we know authentication logic of a concrete backend and its token structure, in particular how to obtain token expiry timers. The jotta-cloud backend supports a number of authentication flows so it's possible that arbitrage can be implemented for a subset only. I think you can submit a feature request on github and let @buengese analyze possible options or whether it might break the Jotta EULA.

At any rate, using Jotta client on multiple hosts must not result in operations forever stalled due to token refresh problems. From user's point of view current behavior is a bug. Backend should abort such operation after configurable timeout or provide workarounds similar to what you do manually. Again, this should be anayzed by @buengese, the backend author. I suggest you submit a github issue.

1 Like

There is no one that has my issue with 0byte files being created with Jotta servers?
To make this happen i create a ton of files and folders (like 100k) then copy them to the mount using file explorer, a 0byte file will be created "ahead of time" then it asks to overwrite it (that always fails).
Sometimes this does not occur until i copy the 100k files a few times.

@durval
I thought a little more about your use case and I think I have an idea how pulling might be implemented in general. This needs two flags/settings: --config-from rc://host:port pointing to another rclone instance with --rc enabled like above.

But additionally we need --config-transfer-filter section/value,... which defaults to nothing. User will set it knowing what they do so they guarantee that requested value is valid at all times.

When a value refresh is requested by backend (usually by oauth task), rclone will unconditionally do the pull request, retry it --low-level-retry times, return failure if any, on success return value to backend and update it in local config. No arbitrage, no attempts to check token expiry field, no nothing.

I can implement this logic in one of my betas. If it works for you or another volunteer I can submit it as PR for review by @ncw. However, for docker swarm (my use case) it's useless as there is no primary volume accessor, they are ever rotating.

Again, current jotta backend behavior (stall forever in token refresh) must be considered a bug and reviewed by @buengese. Please submit a request on github with due details.

1 Like

Hi @Krakkan, definitely not seeing this here. But then I'm not copying hundreds of thousands of files, just a few hundred so far. And in my use case (again, so far) the rclone mount is used mostly for reads, for copying files into the JC remote I've been mostly using goode-olde rclone copy.

I suggest you try and make a repeatable test case, and/or capture a full debug log from the rclone mount process, and then submit it/them as an issue on rclone's GitHub.

Hi @ivandeex, your plan for --config-from sounds just about perfect -- in fact it would automate and generalize what I already do here manually (and was about to turn into a script).

I think it could even go further than the jottacloud remote, and could be used to set a central machine as a configuration 'repository' (for potentially any kind of remote), from which other machines would then pull the needed configuration whenever they got an authentication error like in your plan, or perhaps even unconditionally.

Please count on me as a volunteer for testing your Beta when it's ready!

Thanks again,
-- Durval

1 Like

Not sure what you mean by "stall forever", here it shows an error and dies, eg:

rclone about JCREMOTE:
2021/08/06 04:54:46 Failed to create file system for "JCREMOTE:": couldn't get customer info: Get "https://api.jottacloud.com/
ac count/v1/customer": couldn't fetch token - maybe it has expired? - refresh with "rclone config reconnect JCREMOTE:": oauth2: cannot fetch token: 400 Bad Request 
Response: {"error":"invalid_grant","error_description":"Stale token"}

Ah I see now.
Your error is 400 Bad Request (Stale token)
My error is 429 Too Many Requests.
According to the code rclone will retry 429, here's the "stalled forever" feeling.

@buengese
I receive 429 even if I regenerate token at Jottacloud and run rclone config reconnect jotta: with generated string. What can I do to fix it?

1 Like

Hummmrm.... 429 Too Many Requests definitely doesn't sound like an expired token or any other authentication issue.

I can't be sure at it hasn't happened to me (at least yet) but perhaps Jottacloud is doing some kind of throttling/banning on the account/IPAddress you are using for this, perhaps as a result of too many retries on the 400 Bad Request (Stale Token) error?

Not sure how responsive their support is, but maybe contacting them could shed more light on that?

Anyway, please keep us posted, very curious about how that turns out.

I should clarify, it is Jottacloud whitelabel Elgiganten Cloud i'm using. I have no token problems and my speeds seem way higher then regular Jottacloud, i max my 600/200. Only issue i run into is the 0 byte files once in awhile. An alternative to try? :slight_smile: Sure would like someone to help me getting rid of that bug

Hi @Krakkan, the person for this would be @buengese, the backend author. But, as I advised you already, it will be very difficult (for him or anyone) to help you unless you can provide a repeatable test case and a debug log.

One further suggestion: do these 0-byte files also appear if you use plain rclone copy instead of File Explorer to copy that same ton of files/dirs to the remote?

And as an aside, I've checked and Elgiganten is a Danish service, right? Can you please share with us their prices and conditions? At least speed-wise they sound very interesting.

keep us posted

here you are

August 6

Hi
I am testing Jottacloud on my VPS. After a week of normal operation I cannot refresh Jotta token. Please help.

$ jotta-cli login

ERROR Could not login: auth.login.load-wellknown-configuration: invalid character '<' looking for beginning of value

Hi, Ivan!
I'll have one of our developers look into this and come back to you with some more information

Hi, Ivan!
Would you be able to try this again with a new token, and see if it happens again?

August 10

Tried. Logged into jottacloud.com, web/secure, generated token.
Logged into my vps over ssh:
$ jotta-cli login

ERROR Could not login: auth.login.load-wellknown-configuration: invalid character '<' looking for beginning of value

Doesn't work for me yet.

August 13

I repeated the test today
Clicked "Generate" on https://www.jottacloud.com/web/secure
Copy-pasted token to "jotta-cli login"
It failed again
Please help.

Here is jotta-cli output:

$ jotta-cli login
ERROR Could not login: auth.login.load-wellknown-configuration: invalid character '<' looking for beginning of value

😕

August 14

Hi!

Could you try the latest version of the CLI (https://docs.jottacloud.com/en/articles/1461561-release-notes-for-jottacloud-cli)?

You can find it here: https://github.com/jotta/jotta-cli-issues

Have a great weekend!

I am running the latest jotta cli, the above problems happend with it:
$ jotta-cli version
-------------------------------------------
jottad executable : /usr/bin/jottad
jottad appdata : /var/lib/jottad
jottad logfile : /var/lib/jottad/jottabackup.log
jottad version : 0.11.44593
jotta-cli version : 0.11.44593
-------------------------------------------

August 17

Hi!
One of our developers is looking into this again.

I'm sorry for the inconvenience!

August 23

Hi!
One of our developers looked deeper into this, and it seems like you should be able to log in again if you retry with a new login token.
I'm sorry for the inconvenience!

Have a nice evening

August 24

:-(((

$ jotta-cli login
ERROR Could not login: auth.login.load-wellknown-configuration: invalid character '<' looking for beginning of value

$ jotta-cli version
-------------------------------------------
jottad executable : /usr/bin/jottad
jottad appdata : /var/lib/jottad
jottad logfile : /var/lib/jottad/jottabackup.log
jottad version : 0.11.44593
jotta-cli version : 0.11.44593
-------------------------------------------

August 31

Hi, Ivan!

I'm sorry for the late reply.

Would you be able to try loading it via curl or something else from the same box as which jottad is running on?

Have a great day!

September 4

Hi, Ivan!

I'm sorry for not providing you with the link right away. Here it is: https://id.jottacloud.com/auth/realms/jottacloud/.well-known/openid-configuration

Have a great weekend!

September 8

I accessed it with curl as you advised

$ curl https://id.jottacloud.com/auth/realms/jottacloud/.well-known/openid-configuration
<!doctype html><title>error</title>
<body>error</body>

$ curl -L https://id.jottacloud.com/auth/realms/jottacloud/.well-known/openid-configuration
<!doctype html><title>error</title>
<body>error</body>

$ curl -v -L https://id.jottacloud.com/auth/realms/jottacloud/.well-known/openid-configuration
* Trying 2a0a:cc80:0:c1b2::26...
* Connected to id.jottacloud.com (2a0a:cc80:0:c1b2::26) port 443 (#0)
* successfully set certificate verify locations:
* Server certificate:
* subject: CN=*.jottacloud.com
* start date: Nov 20 00:00:00 2020 GMT
* expire date: Dec 21 23:59:59 2021 GMT
* subjectAltName: host "id.jottacloud.com" matched cert's "*.jottacloud.com"
* issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo RSA Domain Validation Secure Server CA
* SSL certificate verify ok
> GET /auth/realms/jottacloud/.well-known/openid-configuration HTTP/1.1
> Host: id.jottacloud.com
> User-Agent: curl/7.58.0
> Accept: */*
< HTTP/1.1 429 Too Many Requests
< Date: Wed, 08 Sep 2021 07:21:37 GMT
< Content-Length: 54
< Content-Type: text/html; charset=utf-8
< X-RESP: 9
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< Content-Security-Policy: script-src * 'unsafe-inline'
<!doctype html><title>error</title>
<body>error</body>
$ sudo systemctl stop jottad
$ sudo systemctl status jottad

Your host returns "429 too many requests"
I stopped jottad, don't know who else could perform a lot of requests to your host from my VPS. You can find IP of my VPS in the log on your side when the "too manyn requests" happened.

You can find IPs of my VPS below so you can check logs on your side when the "too many requests" event happened.

A wish for the future: it would be nice if jottad or jotta-cli could parse error replies and tell user the real error code "too many requests" from server instead of just "invalid character '<' looking..."

hmm
curl allows to force http requests over ipv4
if I do so, your site works fine

$ curl -4 -v -L https://id.jottacloud.com/auth/realms/jottacloud/.well-known/openid-configuration ; echo
* Trying 185.179.129.26...
* Connected to id.jottacloud.com (185.179.129.26) port 443 (#0)
* Server certificate:
* subject: CN=*.jottacloud.com
* start date: Nov 20 00:00:00 2020 GMT
* expire date: Dec 21 23:59:59 2021 GMT
* subjectAltName: host "id.jottacloud.com" matched cert's "*.jottacloud.com"
* issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo RSA Domain Validation Secure Server CA
> GET /auth/realms/jottacloud/.well-known/openid-configuration HTTP/1.1
> Host: id.jottacloud.com
> User-Agent: curl/7.58.0
> Accept: */*
< HTTP/1.1 200 OK
< Connection: keep-alive
< x-id: 843028339949
< Cache-Control: no-cache, must-revalidate, no-transform, no-store
< Content-Type: application/json
< Content-Length: 2381
< Date: Wed, 08 Sep 2021 09:11:33 GMT
< X-RESP: 2
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< Content-Security-Policy: script-src * 'unsafe-inline'
<
{"issuer":"https://id.jottacloud.com/auth/realms/jottacloud","authorization_endpoint":"https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/auth","token_endpoint":"https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/token","token_introspection_endpoint":"https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/token/introspect","userinfo_endpoint":"https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/userinfo","end_session_endpoint":"https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/logout","jwks_uri":"https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/certs","check_session_iframe":"https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/login-status-iframe.html","grant_types_supported":["authorization_code","implicit","refresh_token","password","client_credentials"],"response_types_supported":["code","none","id_token","token","id_token token","code id_token","code token","code id_token token"],"subject_types_supported":["public","pairwise"],"id_token_signing_alg_values_supported":["ES384","RS384","HS256","HS512","ES256","RS256","HS384","ES512","RS512"],"userinfo_signing_alg_values_supported":["ES384","RS384","HS256","HS512","ES256","RS256","HS384","ES512","RS512","none"],"request_object_signing_alg_values_supported":["ES384","RS384","ES256","RS256","ES512","RS512","none"],"response_modes_supported":["query","fragment","form_post"],"registration_endpoint":"https://id.jottacloud.com/auth/realms/jottacloud/clients-registrations/openid-connect","token_endpoint_auth_methods_supported":["private_key_jwt","client_secret_basic","client_secret_post","client_secret_jwt"],"token_endpoint_auth_signing_alg_values_supported":["RS256"],"claims_supported":["sub","iss","auth_time","name","given_name","family_name","preferred_username","email"],"claim_types_supported":["normal"],"claims_parameter_supported":false,"scopes_supported":["openid","address","email","jotta-default","offline_access","phone","profile","roles","web-origins"],"request_parameter_supported":true,"request_uri_parameter_supported":true,"code_challenge_methods_supported":["plain","S256"],"tls_client_certificate_bound_access_tokens":true,"introspection_endpoint":"https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/token/introspect"}

I added a hack in the local /etc/hosts. It will work until you change your IPv4

$ sudo vim /etc/hosts
$ tail /etc/hosts
185.179.129.26 id.jottacloud.com
$ ping -c1 id.jottacloud.com
64 bytes from id.jottacloud.com (185.179.129.26): icmp_seq=1 ttl=242 time=32.5 ms
$ curl https://id.jottacloud.com/auth/realms/jottacloud/.well-known/openid-configuration
{"issuer":"https://id.jottacloud.com/auth/realms/jottacloud","authorization_endpoint":"https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/auth","token_endpoint":"https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/token","token_introspection_endpoint":"https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/token/introspect","userinfo_endpoint":"https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/userinfo","end_session_endpoint":"https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/logout","jwks_uri":"https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/certs","check_session_iframe":"https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/login-status-iframe.html","grant_types_supported":["authorization_code","implicit","refresh_token","password","client_credentials"],"response_types_supported":["code","none","id_token","token","id_token token","code id_token","code token","code id_token token"],"subject_types_supported":["public","pairwise"],"id_token_signing_alg_values_supported":["ES384","RS384","HS256","HS512","ES256","RS256","HS384","ES512","RS512"],"userinfo_signing_alg_values_supported":["ES384","RS384","HS256","HS512","ES256","RS256","HS384","ES512","RS512","none"],"request_object_signing_alg_values_supported":["ES384","RS384","ES256","RS256","ES512","RS512","none"],"response_modes_supported":["query","fragment","form_post"],"registration_endpoint":"https://id.jottacloud.com/auth/realms/jottacloud/clients-registrations/openid-connect","token_endpoint_auth_methods_supported":["private_key_jwt","client_secret_basic","client_secret_post","client_secret_jwt"],"token_endpoint_auth_signing_alg_values_supported":["RS256"],"claims_supported":["sub","iss","auth_time","name","given_name","family_name","preferred_username","email"],"claim_types_supported":["normal"],"claims_parameter_supported":false,"scopes_supported":["openid","address","email","jotta-default","offline_access","phone","profile","roles","web-origins"],"request_parameter_supported":true,"request_uri_parameter_supported":true,"code_challenge_methods_supported":["plain","S256"],"tls_client_certificate_bound_access_tokens":true,"introspection_endpoint":"https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/token/introspect"}

curl now works without hacky flags

As a next step I enabled jottad back generated another token at https://www.jottacloud.com/web/secure and tried to login again

$ sudo systemctl start jottad
$ jotta-cli login
Logging in. Please wait...
Enter a name to identify this device.
Logged in as xxx on Jottacloud
OK
$ jotta-cli status
--------------------------------------------
Account : xxx on Jottacloud
Usage : 12.76MiB / 5.00GiB
Device : xxx
--------------------------------------------
OK

Please ask your developers to investigate why login over IPv6 fails

September 16

Hi, Ivan!

It looks like our developers found the issue here. The ipv6 range was blocked from our end due to previous malicious traffic. It is not blocked anymore, but that explains why it didn't work. I have also been told that our developers are going to improve the error message shown in the client

@durval Sry for the late reply pricing is free first 3 months then 8.59 USD (converted) a month.