ACD oauth proxy broken!

Suspected Bug
1

I’m currently only getting “429 Too Many Requests message: Rate exceeded”. If using other applications I can access ACD like normal. I’m only getting this error using rclone with the oauth proxy. Is anyone else experiencing problems?
Thank you for your answers in advance!

2

Not helpful, but just confirming the same issue.

trying rclone lsd against my ACD gives me a rate exceeded error.

Hopefully it will correct itself in time. It often does, but we will have to see

3

Same here for me : "429 Too Many Requests message: Rate exceeded” :frowning:

4

So let’s hope that Amazon didn’t revoke the corresponding API keys.

5

Same issue for me

Right as I’m trying to finally move my data off ACD to Drive before my “unlimited” grace period ends :frowning:

6

Same here… many more TB to go though.

7

Same thing here… Im on the clock for the final 30 days with 912 GBs and 2.3 mil files.

More worried about the files then the size.

8

Same here…

9

Does anyone know if Nick himself is operating this proxy or if we should contact someone else?

10

Same issue over here !

11

Just came here to report that I cannot access ACD with rclone any more when it has been working OK for over 6 months. I think that it was working yesterday evening.

12

I can verify this problem.

I don’t know what is causing it, but it is affecting direct access to ACD and indirect access via the proxy so it isn’t the fault of the proxy.

It doesn’t seem to be limited to rclone as acd_cli seems to be affected too.

$ ./acd_cli.py sync
18-01-23 15:23:26.184 [CRITICAL] [acdcli.api.client] - Error getting endpoint data. Response: {"message":"Rate exceeded"}

429 is one of those temporary errors which kind of indicates the problem might go away…

It might be worth someone asking on the ACD forum.

3 Likes
13

thanks mate, I hope it get fixed soon.

I’ve checked their forums and seems none had mentioned the issue there yet.

14

I’ve made a request on the mentioned forum. Let’s hope, that they can help us. Nick, is there a way you could check if the API keys aren’t revoked? Thank you for your great work!!!

15

they answered -> https://forums.developer.amazon.com/questions/106405/rclone-and-acd-cli-not-working-anymore.html

The Answer:

Hi SebiTNT,

Please see my answer about 429 error here: https://forums.developer.amazon.com/questions/54132/getting-429-too-many-request-errors.html

and about rclone and acd_cli here: https://forums.developer.amazon.com/questions/71941/rclone-and-acdcli.html - thanks!

While the first thread says:

Answer by Amazonian Levon@Amazon · Jan 31, 2017 at 02:50 PM

Hi John,

Thanks for posting! Have you actively submitted your application for approval and gotten a direct response saying that > your app is at production level now? If not, then the app is still at developer level, which is designed to be used by a single digit number of users who are actively developing or testing an application. Because of that, if your app is at developer level and you have indeed many thousands of customers using it, then you will be getting error 429s.

The second only points to the old know thread saying they don’t have any integration with rclone anymore, and are not planning on reopening.

16

Looks like the keys were revoked based on Amazon’s reply to your post @SebiTNT :frowning:

Also, I tried a different client_id/secret which did work, so it definitely seems like the proxy that rclone was using has been blocked.

18

I’ve made a very very hacky “fix” for this. It involves using the ACD client itself and “stealing” its authentication headers via Fiddler (and FiddlerScript), and also a GUI automation program to simulate clicks on ACD.

It obviously has a smaller chance of getting disabled, but it doesn’t really work all that well - it obviously has the need for the PC to have it’s GUI occupied, and it does tend to break from time to time (so it needs supervision). But it’s still better than the alternative.

I’m in the process of downloading all 2.5 TB of data off from ACD anyway, so it doesn’t really have to be very robust.

The main thing is to customize Fiddler. The GUI automation is the easy part; I’m using Mouse Recorder - just because it was the first one I found (and I was too bored to make a program of my own).

Let’s start with Fiddler.

  1. Download / install Fiddler
  2. Go to Fiddler options and select the option to decrypt HTTPS
  3. Make ACD client use Fiddler as its proxy (from options)
  4. Set HTTP_PROXY and HTTPS_PROXY variables so that rclone will use Fiddler as a proxy (e.g. SET HTTP_PROXY=http://localhost:8888 and SET HTTP_PROXY=http://localhost:8888)
  5. The hard part now: Editing FiddlerScript.

Add the following to just under the class Handlers declaration:

public static RulesOption("Impersonate A&mazon Cloud")
BindPref("fiddlerscript.rules.AmazonCloud")
var m_ImpersonateAmazonCloud: boolean = false;
public static var m_AmazonAccessToken: String = "xxx";
public static var m_AmazonCloudDriveSource: String = "xxx";
public static var m_AmazonCloudDriveAppId: String = "xxx";

And the following code to the OnBeforeRequest function:

if (m_ImpersonateAmazonCloud && oSession.oRequest.headers.Exists('User-Agent') && 
    (oSession.oRequest.host.Contains('amazonaws.com') || oSession.oRequest.host.Contains('cloudfront.net'))) {
    // FiddlerObject.alert(oSession.oRequest.host);
    var userAgent = oSession.oRequest.headers['User-Agent'];
    if (userAgent == 'rclone/v1.39' && oSession.oRequest.host.Contains('amazonaws.com')) {
        oSession.oRequest.headers["User-Agent"] = "CloudDriveWin/5.0.11.ab4ed4be";
        oSession.oRequest.headers.Remove("Authorization");
        oSession.oRequest.headers["x-amzn-clouddrive-source"] = m_AmazonCloudDriveSource;
        oSession.oRequest.headers["x-amz-access-token"] = m_AmazonAccessToken
        oSession.oRequest.headers["x-amz-clouddrive-appid"] = m_AmazonCloudDriveAppId;
        oSession.oRequest.headers["x-amzn-RequestId"] = Guid.NewGuid().ToString();
        // FiddlerObject.log(oSession["x-ProcessInfo"]);
    } else if (userAgent.StartsWith("CloudDriveWin/")) {
        var amzAT = oSession.oRequest.headers["x-amz-access-token"];
        FiddlerObject.log("Amazon CloudDrive connected");
        if (amzAT != "" && amzAT != m_AmazonAccessToken) {
            m_AmazonAccessToken = amzAT;
            m_AmazonCloudDriveSource = oSession.oRequest.headers["x-amzn-clouddrive-source"];
            m_AmazonCloudDriveAppId = oSession.oRequest.headers["x-amz-clouddrive-appid"];
            FiddlerObject.log("NEW AMZ ACCESS TOKEN:" + m_AmazonAccessToken);
        }
    }
}

Here’s the “easy” part, the GUI automation:

  1. Open Amazon Cloud Drive and go to the downloads tab.
  2. Open Mouse Recorder (or whatever) and record a macro that:
    1. Activates the window (and waits a bit)
    2. Clicks on the “All” folder item (wait a bit)
    3. Clicks on any first level folder (and wait a lot more)

In my case this forces a refresh.

You need to run Amazon Cloud Drive and force it to refresh at least once before you run rclone. And you also need to run the GUI automation part at least once every 30 minutes (preferably more frequently - I run the macro above every 15 seconds, but YMMV)

I’m sure I’m missing something important, but that’s about most of my setup.

19

How can we get the proxy to work with Rclone again? I’m really having tons of TBs to download. Can someone talk to the one who had it running ?

20

i user “Air Explorer” to transfer my acd data to a other storage

The only problem, is that it only runs on Mac or Windows

2 Likes
21

What it looks like is that the keys have had their rating for production use removed, so they’ve been downgraded to developer keys. Or Amazon have change they way they do that sort of thing.

Thanks for confirming that.

1 Like