With push mode we can ensure that the pushing node is updating the config most recently. With pull mode we cannot give such guarantee without timestamping config values which is not possible without breaking the config format. Due to that I don't think I will ever develop --config-from.
1 Like
Hi @ivandeex, and thanks for your continued responses.
I think I understand your explanation re: push vs pull (ie, --config-to vs --config-from) , but the way I'm experiencing the issue, this is not what happens. To wit: my 'central' machine is always the one that has accessed the JC remote last (the reason being it runs processes that go through the remote periodically). The other machines only need to pull the updated tokens when they experience the above error (which is about once a day in my usage), and when I merge the updated config lines in their ~/.rclone.conf, the remote starts working perfectly. I've done that a number of times, and it's 100% repeatable.
Thanks for being frank re: implementing --config-from, I understand and of course respect your stance on this. It's no big trouble, I think I will implement it locally as part of a script that will always run before rclone proper and then pull/merge the necessary lines.
Thanks again,
-- Durval.
Pulling can't be implemented in general since it needs a way to arbitrage between tokens pulled on demand from external rclone instance, tokens requested from the cloud and tokens recorded in rclone config. However it's possible if we know authentication logic of a concrete backend and its token structure, in particular how to obtain token expiry timers. The jotta-cloud backend supports a number of authentication flows so it's possible that arbitrage can be implemented for a subset only. I think you can submit a feature request on github and let @buengese analyze possible options or whether it might break the Jotta EULA.
At any rate, using Jotta client on multiple hosts must not result in operations forever stalled due to token refresh problems. From user's point of view current behavior is a bug. Backend should abort such operation after configurable timeout or provide workarounds similar to what you do manually. Again, this should be anayzed by @buengese, the backend author. I suggest you submit a github issue.
1 Like
@durval
I thought a little more about your use case and I think I have an idea how pulling might be implemented in general. This needs two flags/settings: --config-from rc://host:port pointing to another rclone instance with --rc enabled like above.
But additionally we need --config-transfer-filter section/value,... which defaults to nothing. User will set it knowing what they do so they guarantee that requested value is valid at all times.
When a value refresh is requested by backend (usually by oauth task), rclone will unconditionally do the pull request, retry it --low-level-retry times, return failure if any, on success return value to backend and update it in local config. No arbitrage, no attempts to check token expiry field, no nothing.
I can implement this logic in one of my betas. If it works for you or another volunteer I can submit it as PR for review by @ncw. However, for docker swarm (my use case) it's useless as there is no primary volume accessor, they are ever rotating.
Again, current jotta backend behavior (stall forever in token refresh) must be considered a bug and reviewed by @buengese. Please submit a request on github with due details.
1 Like
Hi @Krakkan, definitely not seeing this here. But then I'm not copying hundreds of thousands of files, just a few hundred so far. And in my use case (again, so far) the rclone mount is used mostly for reads, for copying files into the JC remote I've been mostly using goode-olde rclone copy.
I suggest you try and make a repeatable test case, and/or capture a full debug log from the rclone mount process, and then submit it/them as an issue on rclone's GitHub.
Hi @ivandeex, your plan for --config-from sounds just about perfect -- in fact it would automate and generalize what I already do here manually (and was about to turn into a script).
I think it could even go further than the jottacloud remote, and could be used to set a central machine as a configuration 'repository' (for potentially any kind of remote), from which other machines would then pull the needed configuration whenever they got an authentication error like in your plan, or perhaps even unconditionally.
Please count on me as a volunteer for testing your Beta when it's ready!
Thanks again,
-- Durval
1 Like
Not sure what you mean by "stall forever", here it shows an error and dies, eg:
rclone about JCREMOTE:
2021/08/06 04:54:46 Failed to create file system for "JCREMOTE:": couldn't get customer info: Get "https://api.jottacloud.com/
ac count/v1/customer": couldn't fetch token - maybe it has expired? - refresh with "rclone config reconnect JCREMOTE:": oauth2: cannot fetch token: 400 Bad Request
Response: {"error":"invalid_grant","error_description":"Stale token"}
Ah I see now.
Your error is 400 Bad Request (Stale token)
My error is 429 Too Many Requests.
According to the code rclone will retry 429, here's the "stalled forever" feeling.
@buengese
I receive 429 even if I regenerate token at Jottacloud and run rclone config reconnect jotta: with generated string. What can I do to fix it?
1 Like
Hummmrm.... 429 Too Many Requests definitely doesn't sound like an expired token or any other authentication issue.
I can't be sure at it hasn't happened to me (at least yet) but perhaps Jottacloud is doing some kind of throttling/banning on the account/IPAddress you are using for this, perhaps as a result of too many retries on the 400 Bad Request (Stale Token) error?
Not sure how responsive their support is, but maybe contacting them could shed more light on that?
Anyway, please keep us posted, very curious about how that turns out.
Hi @Krakkan, the person for this would be @buengese, the backend author. But, as I advised you already, it will be very difficult (for him or anyone) to help you unless you can provide a repeatable test case and a debug log.
One further suggestion: do these 0-byte files also appear if you use plain rclone copy instead of File Explorer to copy that same ton of files/dirs to the remote?
And as an aside, I've checked and Elgiganten is a Danish service, right? Can you please share with us their prices and conditions? At least speed-wise they sound very interesting.
keep us posted
here you are
August 6
Hi
I am testing Jottacloud on my VPS. After a week of normal operation I cannot refresh Jotta token. Please help.
$ jotta-cli login
ERROR Could not login: auth.login.load-wellknown-configuration: invalid character '<' looking for beginning of value
Hi, Ivan!
I'll have one of our developers look into this and come back to you with some more information
Hi, Ivan!
Would you be able to try this again with a new token, and see if it happens again?
August 10
Tried. Logged into jottacloud.com, web/secure, generated token.
Logged into my vps over ssh:
$ jotta-cli login
ERROR Could not login: auth.login.load-wellknown-configuration: invalid character '<' looking for beginning of value
Doesn't work for me yet.
August 13
I repeated the test today
Clicked "Generate" on https://www.jottacloud.com/web/secure
Copy-pasted token to "jotta-cli login"
It failed again
Please help.
Here is jotta-cli output:
$ jotta-cli login
ERROR Could not login: auth.login.load-wellknown-configuration: invalid character '<' looking for beginning of value
😕
August 14
Hi!
Could you try the latest version of the CLI (https://docs.jottacloud.com/en/articles/1461561-release-notes-for-jottacloud-cli)?
You can find it here: https://github.com/jotta/jotta-cli-issues
Have a great weekend!
I am running the latest jotta cli, the above problems happend with it:
$ jotta-cli version
-------------------------------------------
jottad executable : /usr/bin/jottad
jottad appdata : /var/lib/jottad
jottad logfile : /var/lib/jottad/jottabackup.log
jottad version : 0.11.44593
jotta-cli version : 0.11.44593
-------------------------------------------
August 17
Hi!
One of our developers is looking into this again.
I'm sorry for the inconvenience!
August 23
Hi!
One of our developers looked deeper into this, and it seems like you should be able to log in again if you retry with a new login token.
I'm sorry for the inconvenience!
Have a nice evening
August 24
:-(((
$ jotta-cli login
ERROR Could not login: auth.login.load-wellknown-configuration: invalid character '<' looking for beginning of value
$ jotta-cli version
-------------------------------------------
jottad executable : /usr/bin/jottad
jottad appdata : /var/lib/jottad
jottad logfile : /var/lib/jottad/jottabackup.log
jottad version : 0.11.44593
jotta-cli version : 0.11.44593
-------------------------------------------
August 31
Hi, Ivan!
I'm sorry for the late reply.
Would you be able to try loading it via curl or something else from the same box as which jottad is running on?
Have a great day!
September 4
Hi, Ivan!
I'm sorry for not providing you with the link right away. Here it is: https://id.jottacloud.com/auth/realms/jottacloud/.well-known/openid-configuration
Have a great weekend!
September 8
I accessed it with curl as you advised
$ curl https://id.jottacloud.com/auth/realms/jottacloud/.well-known/openid-configuration
<!doctype html><title>error</title>
<body>error</body>
$ curl -L https://id.jottacloud.com/auth/realms/jottacloud/.well-known/openid-configuration
<!doctype html><title>error</title>
<body>error</body>
$ curl -v -L https://id.jottacloud.com/auth/realms/jottacloud/.well-known/openid-configuration
* Trying 2a0a:cc80:0:c1b2::26...
* Connected to id.jottacloud.com (2a0a:cc80:0:c1b2::26) port 443 (#0)
* successfully set certificate verify locations:
* Server certificate:
* subject: CN=*.jottacloud.com
* start date: Nov 20 00:00:00 2020 GMT
* expire date: Dec 21 23:59:59 2021 GMT
* subjectAltName: host "id.jottacloud.com" matched cert's "*.jottacloud.com"
* issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo RSA Domain Validation Secure Server CA
* SSL certificate verify ok
> GET /auth/realms/jottacloud/.well-known/openid-configuration HTTP/1.1
> Host: id.jottacloud.com
> User-Agent: curl/7.58.0
> Accept: */*
< HTTP/1.1 429 Too Many Requests
< Date: Wed, 08 Sep 2021 07:21:37 GMT
< Content-Length: 54
< Content-Type: text/html; charset=utf-8
< X-RESP: 9
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< Content-Security-Policy: script-src * 'unsafe-inline'
<!doctype html><title>error</title>
<body>error</body>
$ sudo systemctl stop jottad
$ sudo systemctl status jottad
Your host returns "429 too many requests"
I stopped jottad, don't know who else could perform a lot of requests to your host from my VPS. You can find IP of my VPS in the log on your side when the "too manyn requests" happened.
You can find IPs of my VPS below so you can check logs on your side when the "too many requests" event happened.
A wish for the future: it would be nice if jottad or jotta-cli could parse error replies and tell user the real error code "too many requests" from server instead of just "invalid character '<' looking..."
hmm
curl allows to force http requests over ipv4
if I do so, your site works fine
$ curl -4 -v -L https://id.jottacloud.com/auth/realms/jottacloud/.well-known/openid-configuration ; echo
* Trying 185.179.129.26...
* Connected to id.jottacloud.com (185.179.129.26) port 443 (#0)
* Server certificate:
* subject: CN=*.jottacloud.com
* start date: Nov 20 00:00:00 2020 GMT
* expire date: Dec 21 23:59:59 2021 GMT
* subjectAltName: host "id.jottacloud.com" matched cert's "*.jottacloud.com"
* issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo RSA Domain Validation Secure Server CA
> GET /auth/realms/jottacloud/.well-known/openid-configuration HTTP/1.1
> Host: id.jottacloud.com
> User-Agent: curl/7.58.0
> Accept: */*
< HTTP/1.1 200 OK
< Connection: keep-alive
< x-id: 843028339949
< Cache-Control: no-cache, must-revalidate, no-transform, no-store
< Content-Type: application/json
< Content-Length: 2381
< Date: Wed, 08 Sep 2021 09:11:33 GMT
< X-RESP: 2
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< Content-Security-Policy: script-src * 'unsafe-inline'
<
{"issuer":"https://id.jottacloud.com/auth/realms/jottacloud","authorization_endpoint":"https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/auth","token_endpoint":"https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/token","token_introspection_endpoint":"https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/token/introspect","userinfo_endpoint":"https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/userinfo","end_session_endpoint":"https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/logout","jwks_uri":"https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/certs","check_session_iframe":"https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/login-status-iframe.html","grant_types_supported":["authorization_code","implicit","refresh_token","password","client_credentials"],"response_types_supported":["code","none","id_token","token","id_token token","code id_token","code token","code id_token token"],"subject_types_supported":["public","pairwise"],"id_token_signing_alg_values_supported":["ES384","RS384","HS256","HS512","ES256","RS256","HS384","ES512","RS512"],"userinfo_signing_alg_values_supported":["ES384","RS384","HS256","HS512","ES256","RS256","HS384","ES512","RS512","none"],"request_object_signing_alg_values_supported":["ES384","RS384","ES256","RS256","ES512","RS512","none"],"response_modes_supported":["query","fragment","form_post"],"registration_endpoint":"https://id.jottacloud.com/auth/realms/jottacloud/clients-registrations/openid-connect","token_endpoint_auth_methods_supported":["private_key_jwt","client_secret_basic","client_secret_post","client_secret_jwt"],"token_endpoint_auth_signing_alg_values_supported":["RS256"],"claims_supported":["sub","iss","auth_time","name","given_name","family_name","preferred_username","email"],"claim_types_supported":["normal"],"claims_parameter_supported":false,"scopes_supported":["openid","address","email","jotta-default","offline_access","phone","profile","roles","web-origins"],"request_parameter_supported":true,"request_uri_parameter_supported":true,"code_challenge_methods_supported":["plain","S256"],"tls_client_certificate_bound_access_tokens":true,"introspection_endpoint":"https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/token/introspect"}
I added a hack in the local /etc/hosts. It will work until you change your IPv4
$ sudo vim /etc/hosts
$ tail /etc/hosts
185.179.129.26 id.jottacloud.com
$ ping -c1 id.jottacloud.com
64 bytes from id.jottacloud.com (185.179.129.26): icmp_seq=1 ttl=242 time=32.5 ms
$ curl https://id.jottacloud.com/auth/realms/jottacloud/.well-known/openid-configuration
{"issuer":"https://id.jottacloud.com/auth/realms/jottacloud","authorization_endpoint":"https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/auth","token_endpoint":"https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/token","token_introspection_endpoint":"https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/token/introspect","userinfo_endpoint":"https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/userinfo","end_session_endpoint":"https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/logout","jwks_uri":"https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/certs","check_session_iframe":"https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/login-status-iframe.html","grant_types_supported":["authorization_code","implicit","refresh_token","password","client_credentials"],"response_types_supported":["code","none","id_token","token","id_token token","code id_token","code token","code id_token token"],"subject_types_supported":["public","pairwise"],"id_token_signing_alg_values_supported":["ES384","RS384","HS256","HS512","ES256","RS256","HS384","ES512","RS512"],"userinfo_signing_alg_values_supported":["ES384","RS384","HS256","HS512","ES256","RS256","HS384","ES512","RS512","none"],"request_object_signing_alg_values_supported":["ES384","RS384","ES256","RS256","ES512","RS512","none"],"response_modes_supported":["query","fragment","form_post"],"registration_endpoint":"https://id.jottacloud.com/auth/realms/jottacloud/clients-registrations/openid-connect","token_endpoint_auth_methods_supported":["private_key_jwt","client_secret_basic","client_secret_post","client_secret_jwt"],"token_endpoint_auth_signing_alg_values_supported":["RS256"],"claims_supported":["sub","iss","auth_time","name","given_name","family_name","preferred_username","email"],"claim_types_supported":["normal"],"claims_parameter_supported":false,"scopes_supported":["openid","address","email","jotta-default","offline_access","phone","profile","roles","web-origins"],"request_parameter_supported":true,"request_uri_parameter_supported":true,"code_challenge_methods_supported":["plain","S256"],"tls_client_certificate_bound_access_tokens":true,"introspection_endpoint":"https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/token/introspect"}
curl now works without hacky flags
As a next step I enabled jottad back generated another token at https://www.jottacloud.com/web/secure and tried to login again
$ sudo systemctl start jottad
$ jotta-cli login
Logging in. Please wait...
Enter a name to identify this device.
Logged in as xxx on Jottacloud
OK
$ jotta-cli status
--------------------------------------------
Account : xxx on Jottacloud
Usage : 12.76MiB / 5.00GiB
Device : xxx
--------------------------------------------
OK
Please ask your developers to investigate why login over IPv6 fails
September 16
Hi, Ivan!
It looks like our developers found the issue here. The ipv6 range was blocked from our end due to previous malicious traffic. It is not blocked anymore, but that explains why it didn't work. I have also been told that our developers are going to improve the error message shown in the client
Hi
I have a paid account (since like 2012) and i'm happy with JC, the upload speed is not all good, but for my needs 14 TB it works.
I do only backup, no sync. My jotta is only for disaster, NAS is my primary backup, day to day work on synced OneDrive shared with my employees.
The PC client is so so, but I have started to use rclone, and that is a relief.
Speed test today:
I have 500 MB down speed, my computer is wired on 1 gbit ethernet.
File downloaded:
This file is 3,6 GB, i took 1 min 59 sec to download.
hi, which plan to do have?
This plan, round 8 EUR / month
I think this is really fear price.
Also important, actually the main brake deal for me; servers are in Norway, and Norwegian legislation is about the best you get regarding privacy in the world, maybe the best.
To this account i rclone my shares from my Synology NAS, runs 24/7.
Hi all,
I have the Personal account and i've been using the windows client for a while with no issues.. i have about 2.4TB via the client with them..
I then uploaded 1TB of data using Rclone using a standard mount and the data ended up in the "Archive" location in the web gui.. not a problem.. in fact it's ideal for me.. the problem(?) is that it's not reporting that i've actually used that space.. my usage via the web gui and "rclone about jotta:" is still showing ONLY the space used by the windows client.
Has anyone else seen this? Is this some kind of loop hole or am i still going to get limited at 5tb?
This is the default, yes, but it is configurable with device and mountpoint options when setting up the rclone remote (upload to backup devices not supported until #5926 is accepted).
I don't think so, unfortunately.. Cannot what you observe be explained by the built-in deduplication?
I highly doubt that it's managed to dedupe nearly 1TB of new data..
On one free (5GB) account I test with, I have only data in Archive, and then everything is accounted for both in web gui and in rclone about.
showing ONLY the space used by the windows client.
Is that Sync or Backup data?