Proton Drive -- Too many recent API requests

Suspected Bug
1

What is the problem you are having with rclone?

When synchronizing with my Proton Drive remote, after around 3 minutes I get many "Too many recent API requests" warnings.

Run the command 'rclone version' and share the full output of the command.

rclone v1.65.1
- os/version: arch (64 bit)
- os/kernel: 6.7.0-zen3-1-zen (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.21.6
- go/linking: dynamic
- go/tags: none

Which cloud storage system are you using?

Proton Drive

The command you were trying to run

rclone copy /home/USER/SHARED_FOLDER ProtonDrive:rclone/ --checkers=128 --transfers=8 --checksum --progress --use-mmap --protondrive-replace-existing-draft=true --protondrive-enable-caching --track-renames --track-renames-strategy hash --tpslimit 0.0125 --tpslimit-burst 1 --fast-list --log-level=DEBUG --dry-run

The rclone config contents with secrets removed.

[ProtonDrive]
type = protondrive
username = USER
password = PASSWORD
client_uid = CLIENT_UID
client_access_token = CLIENT_TOKEN
client_refresh_token = CLIENT_REFRESH_TOKEN
client_salted_key_pass = CLIENT_SALTED_KEY_PASS

A log from the command with the -vv flag

My observations

Reducing "tpslimit" does not change the behaviour. I also checked with tcpdump and the number of calls per second (I suppose to the API) does not seem to change when "tpslimit" is changed.

It takes several hours to dry-run copy my 31585 files (spread across 8079 folders) with total size 5.5Go. This seems quite slow.

2

I think your should bring it to guys running Proton Drive and lobby for increasing their limits. Clearly as it is right now they do not like any "bulk" operations.

There is not much rclone can do I am afraid.

3

Thanks for your quick feedback.

Indeed, Proton Drive may not appreciate bulk operations.
But do I misinterpret the role of "tpslimit" then? Shouldn't a small value decrease the number of calls to the API (and thus prevent such issues)?

4

You are right for tpslimit and API calls but nobody knows what their logic and limits are.

Again it depends how their system works. Rclone can limit number of API calls - they still throttle. Maybe they do not like how many requests you did in the last hour? in the last 24h?

Maybe on their forums there are people who can answer these questions.

5

The current behavior is that I start having these messages after around 3 minutes of copy/sync (which can be mapped to a certain number of API calls). It does not seem to depend on requests in the last hour or day.
My goal was to find a sweet spot by tuning tpslimit but, as tcpdump confirms, values of 125 or 0.0125 seem to produce the same number of requests/transactions per second.

Note that if I increase the number of checkers from 128 to 256 (a factor of 2), then these 3 minutes become slightly more than 1.5 minutes (a factor of 1/2).
My guess is that tpslimit has no effect on Proton Drive remotes; this would be a bug.

6

And yes this is another problem. If you have such remote why you use insane values?

Default is 8 and 4

Try:
--checkers=2 --transfers=1 or --checkers=4 --transfers=2

7

This remote is pretty much experimental and API is not officially supported by Proton AFAIK. Me I would not use it for anything but testing.

Maybe some other Proton users here who could share their findings and settings?

8

I have the same issue with Proton Drive. It worked for a while, but then, i started having problems and my sync never finishes. even on small syncs, where only a few files were changed.

9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.