Proton Drive x rclone

Feature
1

Hi all, I'm an engineer at Proton and would like to provide some information to people using our service with rclone and how we currently keep rclone functional.

Changes to the upload flow

We do not intentionally introduce breaking changes that would disrupt third party clients.

In early 2024 we rolled out a new data‑integrity mechanism for file uploads. The feature is mandatory and is designed to protect users from many potential causes of data corruption. Rclone lacked support for this mechanism, which caused upload failures for its users. After detecting a spike in failed uploads, Proton made a temporary exemption that exempted rclone from this validation. We then provided guidance on implementing the new validation. Although we've seen that pull‑requests have been opened to add the extra validation, none have been merged or released to date.

The temporary exemption remains in place to continue supporting rclone functionality, as a gesture of goodwill to the rclone community.

Handling file uploads is a complicated multi-step process, where any step can get interrupted or fail. Proton Drive has the added complexity of being end-to-end encrypted, meaning the client has to encrypt the data and the server has no way to verify the integrity of the encrypted files. Client-side bugs in the encryption may lead to data being impossible to decrypt later. This is one of the reasons why Proton is developing an SDK, to make it as easy as possible for everyone to perform end-to-end encrypted uploads safely. The SDK is already being integrated into the official Proton Drive apps to ensure one uniform upload process across platforms.

Version‑header handling

rclone sends the HTTP header x‑pm‑appversion with the value macos‑drive@1.0.0‑alpha.1+rclone. We use this header to identify the app version and to upgrade clients with known issues or to support important new features such as Proton Docs. While the alpha version of the macOS Drive app that rclone masquerades as is officially unsupported, we allow it to ensure rclone continues to work. External apps should identify as external-drive-<project>@<version> (a pull request has been submitted to GitHub to inform the rclone maintainers about this https://github.com/rclone/rclone/pull/9189). External apps are not officially supported, so use at your own risk. Remember that with end to end encryption, there is very little data validation that can be done; we use the header to ensure that clients are properly safeguarding the integrity of user data. Improper use of this header prevents those safeguards from working properly, resulting in failed uploads.

Domain usage

rclone connects to Proton services via the outdated mail.proton.me/api. The official Proton Drive apps now use drive-api.proton.me. Drive usage of the mail.proton.me/api endpoint is deprecated but we've kept it around to avoid breaking rclone. As a gesture of goodwill we have submitted a pull request on GitHub to change the domain that rclone uses https://github.com/rclone/go-proton-api/pull/1.

Traffic and rate‑limit considerations

Proton does not block users for using rclone, but does enforce good behaviour usage of our API across the board for all clients, to protect availability of Proton for all customers. Official Drive apps use a lightweight event‑based model; In contrast, rclone scans every folder locally and issues a larger number of API calls to re-fetch metadata on every pass. This causes a large number of requests, especially when users have many files and folders. Official apps back off gradually, whereas rclone does not. Rclone causes a disproportionate amount of traffic to Proton's servers. All traffic is subject to the same rate‑limit policies; there are no special limits for rclone. When limits are reached, the server throttles the offending user to protect the availability of Proton for all users.

Human verification (CAPTCHA)

Our anti‑abuse system may occasionally require users to complete a CAPTCHA, again to protect the availability of Proton for all users. Because rclone generates a high volume of API requests it is more likely to trigger this verification, particularly when used over a VPN. Unlike the official Drive apps, rclone does not surface the CAPTCHA challenge to the user, which can leave the user appearing “stuck.”

Upcoming storage model

We are preparing a new storage model that will improve performance and security and be simpler to use. Files stored with this future model will not be readable by rclone until the client is updated. The Proton Drive SDK will include built‑in support for the new model, allowing third-party developers to handle the new format. In the shorter term, an upcoming change of algorithm in OpenPGP may affect rclone's ability to read files uploaded by different clients, or make changes to them.

Future developments

Proton understands that third party tools and applications are an important part of the open-source community and has continuously taken steps to ensure that legitimate third party use of the Proton Drive APIs is not unduly constrained. Proton must also ensure availability of those APIs for all users and continue to insist on correct handling of user data when interacting with those APIs. The upcoming SDK release later this year will provide all third party developers with a free, standard, performant, open-source way to interact with the Proton Drive APIs safely and correctly, supporting a robust ecosystem of third-party authors and projects. The best way to ensure that any given third party project will run smoothly will be to use the SDK for all interaction with the Proton Drive APIs. After the SDK is released and a suitable grace period (tbd) has elapsed, all old clients - not just the third-party ones - will be deprecated and their continued functionality will no longer be guaranteed. Third party clients will either need to adopt the (free, open-source) SDK or fully emulate its safety, correctness, and availability features to continue accessing the Proton Drive APIs.

We sincerely believe that the upcoming release of the Proton Drive SDK will offer every third party project - and most definitely rclone - the chance to be faster, simpler, and more reliable.

If you have questions or comments please check out our two blog posts:

And feel free to join the discussion on /r/ProtonDrive with the sdk-preview-discussion tag.

Thank you.

5 Likes
2

I see in your annoucement

The following are now available for preview:

  • JavaScript SDK
  • C# SDK
  • Bindings for Swift and Kotlin

Was there any plan for a Go SDK?

3

There's currently no plans for a Go SDK or bindings for Go. The focus is on supporting the languages used by the official Proton Drive clients which is where all the work is currently directed.

At this moment we cannot commit to supporting more languages in the official SDK repository but would recommend Go (or other) bindings to be maintained in a community project.

4

Captchas in a storage-backend. :rofl:
I am sure that regularly bothering the own users with accusations of being not a human will lead to very big economical success.

c
c397×500 68.5 KB

To add to this:
rclone should maybe formulate some guidelines on what storage backends should be supported.
With exemption of services that use such disruptive techniques.
We see with this thread that rclone has leverage over the storage providers. So the community should show the businesses what exactly is good practice and tolerable and what is not and if they deviate too much, no support.

5 Likes
5

I had been considering adding Proton to our list of storage providers, as it seemed to be a strong contender that met most of my specific requirements. However, I am pleased to have come across this information, as learning that Proton potentially intends to move away from rclone has spared me the time of having to migrate away from it at a later stage.

6

I was pleased to find via my logs that my rclone scripts began working again with Proton, and much faster than I recall from past use. I hope this continues and only improves over time. I use Gentoo and just upgraded today to 1.74.0.

7

All the integration tests are working with proton from v1.74.0 and we'll try to keep it like that :slight_smile:

3 Likes
8

That sounds good! But does it then make sense that Proton Drive is marked as T5 / deprecated here: Proton Drive ?

1 Like
9

I upgraded proton drive to T4 :slight_smile:

1 Like
10

I am unable to sync due to enabling the post quantum encryption. Is this something fixable by me? Thank you.

11

As announced earlier, Proton Drive is currently rolling out a change to the OpenPGP algorithm used for newly uploaded files, implemented inside of our Proton Drive SDK. This may impact rclone's ability to upload new versions to such a file.

The PGP messages that Drive uses for file contents consist of two packets, a public key encrypted session key (PKESK), and a symmetrically encrypted integrity protected data packet (SEIPD). The first is the node's content key packet. The second is the encrypted content. Files uploaded before this change use a v3 PKESK, and a v1 SEIPD. The new encryption algorithm requires a v6 PKESK and a v2 SEIPD. Since the session key is shared between revisions when files are updated, all revisions on a file must match the encryption scheme used by the first one. The symmetric encryption used for content is AES-GCM, which makes full use of hardware encryption on most devices, which is a massive improvement for power consumption, cpu utilization and overall speed on all clients and all platforms, including rclone!

This means ALL clients (including both Proton's official apps as well as all third-party applications, such as rclone) need to be updated to upload a new file revision. Clients built on the SDK will handle this automatically. Revisions that are submitted with the wrong version will be rejected. This means that clients that do not support the feature will not be able to update files that were uploaded once this change has come into effect.

Proton recognizes that the transition to the SDK plus the changes to the crypto model are a huge break with the architecture of the past - but one that is very necessary and, frankly, long overdue. We believe that third party development will be hugely simplified moving forward, either by integrating the SDK (highly recommended) or, if impractical, referring to the SDK's official, platform-agnostic implementations of changes like these. For future client-breaking changes like this, we will put in place a policy to announce them much earlier (order of several months) to give third-party developers the opportunity to either update their SDK integration or to emulate the necessary functionality. We have not yet decided on exactly where these announcements will be posted, but we'll let you know once we establish the location definitively.

This week's upcoming announcements will further clarify the situation and provide more background and details.

Additionally just as a heads-up, Proton is rolling out Post-Quantum Cryptography. When enabled, a new primary address-key will be a new PQC-key but this one cannot be used yet in Drive for cryptographic operations.

12

Thanks for the heads-up @dlaumen

Will the GitHub - ProtonMail/gopenpgp: A high-level OpenPGP library · GitHub be updated? If so then we can probably manage to update our code.

Or do you think we should start from scratch and port the c# SDK to Go?

Thanks

13

I had a look into this in a bit more detail. It looks like ProtonMail/openpgp v3 already has the support needed, we'd just have to start using it.

Have the protondrive clients started using the new format already? I think we can read v6 packets already so it is just the upload. Should we be using this as rclone's upload format immediately? Or does it need a feature flag?

14

@ncw GopenPGP is used by some of our official clients as well, it supports the new scheme indeed.

All the official Proton Drive clients are as of today uploading all new files with a v6 PKESK.

Since we retain support for some of our older clients, we do not prevent creating new files with a v3 PKESK yet nor do we plan to in the short term.

That being said, Proton Drive supports multiple versions of a file (revisions). If any client (official legacy or third party), attempts to upload a new revision with a v1 SEIPD to a file with a v6 PKESK, this will fail. Meaning if rclone supports creating multiple versions, this is something that would break if the version of the PKESK is not taken into account.

These details as well as a visual representation can be found on our blog now as well.

15

Hi @dlaumen

I gave this a test but I couldn't get it to work. According to my grepping on the SDK this feature is enabled with a server flag DriveCryptoEncryptBlocksWithPgpAead but I think this is disabled on my test account. Is the plan that this gets enabled for everyone soon?

Thanks

Nick

16

@ncw the flag only controls client-side behaviour and is enabled for everyone.

In GopenPGP v3 when decrypting the session key, it should track its version. Then when encrypting with this session key, it will use an seipdv2 (when needed) on a new revision.
This is the only part that could break. When someone uses an official client to upload a file, and then tries to upload a new version using another client without support for the new encryption.

To also create new files (nodes) with the new scheme (which is currently not needed as legacy nodes can still be created, and could be handled long term by a switch to the SDK), there's a configuration in GopenPGP.

  • should be set on the node key for files ONLY (not folders)
  • should be passed (via the profile) to the generate session key for the content key packet
  • it should NEVER be used (disabled regardless of key preferences) for the name, passphrase, and xattr encryption)

For revisions, the blocks must be encrypted with the CKP v3 -> block SEIPDv1 or CKP v6 -> block SEIPDv2. This happens automatically on GopenPGP v3 by decrypting the content key packet and then encrypting with it.