Dropbox token keeps expiring

GRX92 · June 8, 2023, 8:54am

What is the problem you are having with rclone?

On 2 of my nodes, I keep getting errors that the used token is expired shortly after generating a token using rclone authorize.

Run the command 'rclone version' and share the full output of the command.

Node 1

rclone v1.60.1-DEV
- os/version: ubuntu 23.04 (64 bit)
- os/kernel: 6.2.0-1004-raspi (aarch64)
- os/type: linux
- os/arch: arm64
- go/version: go1.19.4
- go/linking: dynamic
- go/tags: none

Node 2

rclone v1.60.1-DEV
- os/version: ubuntu 23.04 (64 bit)
- os/kernel: 6.2.0-20-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.19.4
- go/linking: dynamic
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Dropbox

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

Note: I tried to combine some options that I found in different topics on this forum, if it's suboptimal, advice is absolutely welcome!

rclone mount dropbox-crypt: /mnt/cloud/dropbox \
    --allow-other \
    --allow-non-empty \
    --vfs-cache-mode writes \
    --config=/rclone/rclone.conf \
    --cache-tmp-upload-path=/tmp/rclone/upload \
    --cache-chunk-path=/tmp/rclone/chunks \
    --log-file=/rclone/logs/dropbox.log \
    --dropbox-batch-mode async \
    --dropbox-batch-size 1000 \
    --dropbox-batch-timeout 10s \
    --transfers 32 \
    --log-level INFO

The rclone config contents with secrets removed.

[dropbox]
type = dropbox
client_id = {{removed}}
client_secret = {{removed}}
token = {"access_token":"{{removed}}","token_type":"bearer","refresh_token":"{{removed}}","expiry":"2023-06-07T17:10:50.25358+02:00"}

[dropbox-crypt]
type = crypt
remote = dropbox:/{{removed}}/
password = {{removed}}
password2 = {{removed}}

A log from the command with the `-vv` flag

Error 1:

ERROR : IO error: Post "https://api.dropboxapi.com/2/files/list_folder": couldn't fetch token - maybe it has expired? - refresh with "rclone config reconnect dropbox{CaXPI}:": oauth2: cannot fetch token: 400 Bad Request
Response: {"error": "invalid_grant", "error_description": "refresh token is invalid or revoked"}

Error 2:

2023/06/06 18:51:17 ERROR : IO error: io: read/write on closed pipe
2023/06/06 18:51:18 ERROR : {{path}}: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2023/06/06 18:51:19 ERROR : {{path}}: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2023/06/06 18:51:19 ERROR : {{path}}: WriteFileHandle.Flush error: upload failed: too_many_requests/..
2023/06/06 18:51:19 ERROR : IO error: upload failed: too_many_requests/..

Not sure if those 2 are related, but I think that's really all information that I can provide.

A sidenote; I can't seem to reach more than +/- 15mb/s upload on Dropbox, is that related to my setup or the limitation of the Dropbox API?

kapitainsky · June 8, 2023, 9:10am

Update rclone to the latest version

Do not use random stuff from random sources - better to use defaults in most cases.

e.g. --dropbox-batch-mode async does not guarantee successful writes. Do you have any special reason to use it?

What you need is to decide about cache mode and try to limit number of transactions (as Dropbox has heavily limited API)
by setting --tpslimit 12 --tpslimit-burst 0 otherwise you see what you see:

2023/06/06 18:51:19 ERROR : IO error: upload failed: too_many_requests/..

--transfers 32 is way too high for Dropbox - I would use default 4

Almost reference mount options for dropbox can be found in the link below.
https://github.com/animosity22/homescripts/blob/master/systemd/rclone-tv.service

kapitainsky · June 8, 2023, 9:15am

And forgot to mention. Very important. Create your own APP ID

If you have two nodes - create separate APP IDs for each.

GRX92 · June 8, 2023, 9:27am

Thank you for your detailed response, and the advice that comes with it!

e.g. --dropbox-batch-mode async does not guarantee successful writes. Do you have any special reason to use it?

I don't have a specific reason to use it, the first issue that I had with Dropbox is that writes were incredibly slow.
Doing a touch /rclone/dropbox/test took about 1-2 seconds to finish, so I tried to apply as much caching as possible to make sure it wrote to disk first, then uploaded the file to Dropbox instead of directly writing it to Dropbox.

I'm still not really familiar with how caching works with Dropbox (or rclone in general), so I might need some optimization there.

Do not use random stuff from random sources - better to use defaults in most cases.

Well not really completely random, when I was looking to solve issue x, I found topics with answers that had

I used --option=value and that solved x for me

But I do get your point.

Almost reference mount options for dropbox can be found in the link below.

That's a good one to read into, thank you!

And forgot to mention. Very important. Create your own APP ID
If you have two nodes - create separate APP IDs for each.

That's valuable, I was used to Gdrive where I could re-use the client_id/secret and app token between all my nodes.
I'll think of a way to properly distribute this via Ansible so each node will get their own client_id/secret

kapitainsky · June 8, 2023, 9:31am

Dropbox limits are applied by APP ID so using the same in multiple places limits overall throughput. Another problem with Dropbox is that their API limits are not publicly available so it is all test and try approach. E.g. --tpslimit 12 --tpslimit-burst 0 is what seems work the best for many people.

GRX92 · June 8, 2023, 9:43am

Dropbox limits are applied by APP ID

As the topic title states, the issues that I have primarily is that token seems to be constantly expired.
Yesterday around 1PM I've updated the token on a single node using rclone authorize dropbox, by 4PM the log file was already full with:

ERROR : IO error: Post "https://api.dropboxapi.com/2/files/list_folder": couldn't fetch token - maybe it has expired? - refresh with "rclone config reconnect dropbox{CaXPI}:": oauth2: cannot fetch token: 400 Bad Request
Response: {"error": "invalid_grant", "error_description": "refresh token is invalid or revoked"}

Are the APP ID and token in any way related? Or is there something else I can look into?

kapitainsky · June 8, 2023, 9:57am

Are you using rclone config reconnect dropbox on your node? and clone authorize dropbox on other machine with web browser?

kapitainsky · June 8, 2023, 10:04am

Other thing is thast rclone.conf are portable / pastable so you can create it on any machine and just copy it or paste it over.
If you have working token on one node just copy/paste it to another - assuming you are using the same APP ID.

Now when you are re-authorizing in web browser are you seeing the rclone logo? If yes then it means you are using default APP ID.

When using your own client_id and client_secret you should see your name there instead of rclone

For me the best option is to create new dropbox config on my laptop and copy paste to headless server.

GRX92 · June 8, 2023, 10:27am

I've never checked if the oAuth page displays Rclone or my own app.

I have copied the rclone.conf to my local machine (including the client_id/secret),
but when I run rclone --config={{path to rclone}} authorize dropbox. the browser shows Rclone instead of my app name.

Is there anything I'm missing here?

I've set the return url to: http://localhost:53682/ as per documentation.

kapitainsky · June 8, 2023, 10:31am

I think the best approach is to start from scratch as your token seems to be bound to default APP ID and this is what re-authorization is using. Run rclone config -> create new dropbox remote with client_id and client_secret and this should authorize it correctly. No rclone logo on authorization web page will be sign that all is on the right track.

E.g. for me it looks like this:

GRX92 · June 8, 2023, 10:53am

A start from scratch did the trick indeed, it's now using my created app.
I also took your advice on start command config, I've used the majority of the example you gave.

I'm going to start a new rsync job and see in a few hours if this solved the issue.
Thank you so far for your help, much appreciated!

kapitainsky · June 8, 2023, 10:54am

Cool. Share your results:) It is never ending learning and a lot of experiments so it always helps.

GRX92 · June 10, 2023, 11:11am

I've started an rsync job about 2 days ago and it's still running like a charm!

Still limited to the 15mb/s upload tho, but at least no issues with the token or hitting rate limits

Thank you for helping out!

kapitainsky · June 10, 2023, 11:17am

Great!

One more thing you can try is to add --bind <IPv4 Address> to your mount - it is worth to try if your ISP uses both IPv4 and IPv6. Often performance differs.

GRX92 · June 12, 2023, 10:18am

I've disabled IPv6 in my modem, so I'm only using an IPv4 address
But a good addition for this topic in case other users stumble on a similar issue!

FYI: I think setting transfers=32 to transfers=4 fixed the issue of expiring tokens, It looks like when I exceed the rate limit (too many times) the token automatically gets invalidated.

system · July 12, 2023, 10:19am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.