Start up Slow, then Too many requests

What is the problem you are having with rclone?

It seems slow, but perhaps my command is wrong

What is your rclone version (output from rclone version)

rclone: Version "v1.55.1"

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Mac OS Big Sur on M1

Which cloud storage system are you using? (eg Google Drive)

OneDrive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

./Applications/RClone/rclone rmdirs remote:stack -vv --dry-run

The rclone config contents with secrets removed.

[remote]
type = onedrive
region = global
token = {blabla,"expiry":"2021-07-01T22:12:57.010632+02:00"}
drive_id = something
drive_type = business

A log from the command with the -vv flag

./Applications/RClone/rclone rmdirs remote:stack -vv --dry-run
2021/07/01 21:13:51 DEBUG : Using config file from "/Users/marcelloh/.config/rclone/rclone.conf"
2021/07/01 21:13:51 DEBUG : rclone: Version "v1.55.1" starting with parameters ["./Applications/RClone/rclone" "rmdirs" "remote:stack" "-vv" "--dry-run"]
2021/07/01 21:13:51 DEBUG : Creating backend with remote "remote:stack"
2021/07/01 21:14:32 DEBUG : Too many requests. Trying again in 45 seconds.
2021/07/01 21:14:32 DEBUG : pacer: low level retry 1/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2021/07/01 21:14:32 DEBUG : pacer: Rate limited, increasing sleep to 45s
2021/07/01 21:14:32 DEBUG : pacer: Reducing sleep to 33.75s
2021/07/01 21:14:32 DEBUG : pacer: Reducing sleep to 25.3125s
2021/07/01 21:14:32 DEBUG : pacer: Reducing sleep to 18.984375s
2021/07/01 21:14:32 DEBUG : pacer: Reducing sleep to 14.23828125s
2021/07/01 21:14:33 DEBUG : Too many requests. Trying again in 35 seconds.
2021/07/01 21:14:33 DEBUG : pacer: low level retry 1/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2021/07/01 21:14:33 DEBUG : pacer: Rate limited, increasing sleep to 35s
2021/07/01 21:14:33 DEBUG : Too many requests. Trying again in 42 seconds.
2021/07/01 21:14:33 DEBUG : pacer: low level retry 1/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2021/07/01 21:14:33 DEBUG : pacer: Rate limited, increasing sleep to 42s
2021/07/01 21:14:33 DEBUG : pacer: Reducing sleep to 31.5s
2021/07/01 21:14:33 DEBUG : pacer: Reducing sleep to 23.625s

hello,

i do not see a client id/key your config file, make sure you have done this....
https://rclone.org/onedrive/#getting-your-own-client-id-and-key
"If you are having problems with them (E.g., seeing a lot of throttling), you can get your own Client ID and Key"

The thing is, when I do a sync:

./Applications/RClone/rclone sync --delete-during -v -P /Users/marcelloh/folder remote:stack

It doesn't have those messages.
(it has a different problem in the end: it never stops, so that's why I was trying just to remove the empty folders)

rclone sync can use a lot less api calls per second, as it takes time to transfer the files, the default is just four at a time and an api call per chunk.

you are using a client id, shared with a unknown count of fellow rcloners.
you can try using --tpslimit to slow down the number of api calls per second.
or try --fast-list to reduce the number of api calls.

perhaps @Ole has a suggestion?

Hi Marcelloh,

Your rmdirs command seems correct if you are trying to recursively remove all empty directories under remote:stack.

I do see some OneDrive throttling, but that could be OK if your have many folders/files and/or have had a lot of OneDrive activity in the past 24 hours. You therefore may get some good info from this post/thread on OneDrive throttling:

If the above post doesn't help, then I need some additional information to better understand your situation:

What are the characteristics of your data in remote:stack? How many folders? How many empty folders? How many files? Total size? Anything special?

What are the characteristics of the past 24 hours activity on your OneDrive? (both rclone and native clients)

Now I'm trying this:

./Applications/RClone/rclone rmdirs remote:stack --dry-run --transfers=2 --checkers=2 -vv 
2021/07/02 10:54:00 DEBUG : Using config file from "/Users/marcelloh/.config/rclone/rclone.conf"
2021/07/02 10:54:00 DEBUG : rclone: Version "v1.55.1" starting with parameters ["./Applications/RClone/rclone" "rmdirs" "remote:stack" "--dry-run" "--transfers=2" "--checkers=2" "-vv"]
2021/07/02 10:54:00 DEBUG : Creating backend with remote "remote:stack"
2021/07/02 10:54:13 DEBUG : Too many requests. Trying again in 311 seconds.
2021/07/02 10:54:13 DEBUG : pacer: low level retry 1/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2021/07/02 10:54:13 DEBUG : pacer: Rate limited, increasing sleep to 5m11s
2021/07/02 10:54:14 DEBUG : pacer: Reducing sleep to 3m53.25s
2021/07/02 10:54:14 DEBUG : pacer: Reducing sleep to 2m54.9375s

Idk how many empty folders, it's almost 300GB
My source:
296164 files
79241 folders
The last time I've accessed OneDrive was via Rclone yesterday.

I see, what did you learn by reading the thread I linked above?

That I can have extra params which I lowered to make sure, but without any success?

There is a lot more to be learned :grinning:
I suggest you read the entire thread again and pay attention to all details.

But that thread is about copying from OneDrive to GooglDrive, while I'm copying from localdrive to OneDrive.
What would be the slowest, most throttled command for that?

I now have this:

./Applications/RClone/rclone rmdirs remote:stack --dry-run --transfers=1 --checkers=1 -vv --progress --stats 10s 

Without success

Correct, but there is still a lot to be learned from that thread :grinning:
I suggest you read it and pay attention to all the details.

It must be a guessing game, well I'm more a person that likes to read solutions without having to solve a puzzle. Thanks for trying to help.

i offered several possible solutions, none of them helped?

keep in mind that each time you run your command with --dry-run, you are using a huge amount of api calls and nothing is getting removed.

I now try this one:

./Applications/RClone/rclone rmdirs remote:stack --transfers=1 --checkers=1 -vv --progress --stats 10s --tpslimit 1 --fast-list

I will let this run for a while, but for now, it's only showing me:
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time: 5m11.9s
(so 5 minutes doing nothing at all)

I probably will after I;m back from my holiday. Have more time then.

Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time:     20m41.9s

Back from holiday and made a client (Microsoft OneDrive)
I copied the client-id and the secret-id into an empty text document, edited the configuration and ran it again.
The first time it complains about an something being wrong (I think it was the ID).
Then when I do it again, it's just the same situation:

./Applications/RClone/rclone rmdirs remote:stack --transfers=1 --checkers=1 -vv --progress --stats 10s --tpslimit 1 --fast-list
2021/07/12 09:10:34 DEBUG : Using config file from "/Users/marcelloh/.config/rclone/rclone.conf"
2021/07/12 09:10:34 INFO  : Starting transaction limiter: max 1 transactions/s with burst 1
2021/07/12 09:10:34 DEBUG : rclone: Version "v1.55.1" starting with parameters ["./Applications/RClone/rclone" "rmdirs" "remote:stack" "--transfers=1" "--checkers=1" "-vv" "--progress" "--stats" "10s" "--tpslimit" "1" "--fast-list"]
2021/07/12 09:10:34 DEBUG : Creating backend with remote "remote:stack"
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time:      4m52.1s

So I edited it again, setting both to empty again.
Then run and stop.
Then edited it again, setting the 2 id's, to see that one time message:

If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth?state=_L0-d1TpFmiBi1qJM5D8Ng
Log in and authorize rclone for access
Waiting for code...
Got code
2021/07/12 09:46:15 Failed to configure token: failed to get token: oauth2: cannot fetch token: 401 Unauthorized
Response: {"error":"invalid_client","error_description":"AADSTS7000215: Invalid client secret is provided.\r\nTrace ID: 88bbda57-7836-45aa-84a4-ad7b70595200\r\nCorrelation ID: e0f4a7eb-c851-4f0c-88c3-bf1153786c87\r\nTimestamp: 2021-07-12 07:46:17Z","error_codes":[7000215],"timestamp":"2021-07-12 07:46:17Z","trace_id":"88bbda57-7836-45aa-84a4-ad7b70595200","correlation_id":"e0f4a7eb-c851-4f0c-88c3-bf1153786c87","error_uri":"https://login.microsoftonline.com/error?code=7000215"}

hi,

about the cliend id, make sure to use value, not Secret ID

In the instructions it shows "Copy and keep that secret for later use." and since the other one was about the "id" i'd figured this one was about the "id" as well.
I didn't copy that, so have to remove and retry.

No more errors when editing the config, but still:

2021/07/12 15:23:32 DEBUG : Creating backend with remote "remote:stack"
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time:       2m2.0s

ok, so now we are passed the config issue.

what is the problem now?

Idk, i I think it doesn't do what it supposed to do.

2021-07-12 16:23:00 DEBUG : remote: Saved new token in config file
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time:   1h57m12.0s