Frequent 429 errors from ClearlyDefined

[waynebeaton]

We're getting periodic (but with growing frequency) reports of failures to get a response from ClearlyDefined.

Many projects have integrated Dash License Tool calls into pull request workflows, for example. My (possibly naive) hypothesis is that each individual project is making a reasonable number of calls, but since these are running on shared infrastructure, during periods of high activity, the ClearlyDefined server is getting hammered in aggregate and is throttling. We're also likely competing for resources from other workflows in unrelated GitHub projects that use ClearlyDefined.

Some thoughts on how to deal with this...

1. Implement rate limiting. My concern with this is that doing this will increase build times while the workflow is just sitting waiting. Also, if my hypothesis is correct, there's no way of knowing if it would even work: it's possible that we could still get squeezed out by competing processes and wind up in an endless cycle of waiting.
2. Strongly encourage projects to configure their frequently executed workflows to treat this as a warning not a blocker.

What do you think, @HannesWell ?

[stbischof]

The limits are:

Endpoint	Method	Limit/Window
/definitions	POST	250 /min
/curations	POST	250 /min
/notices	        POST	250 /min

All other endpoints are max 2K requests per minute.

https://docs.clearlydefined.io/docs/get-involved/using-data

and they only are able to track the ip.

they also say
You can check the x-ratelimit-limit and x-ratelimit-remaining response headers to track your usage and reset window.

and i when test the api i see that the rate limit is higher

GET ( we do not use)

x-ratelimit-limit: 5000 
x-ratelimit-remaining: 4951  
x-ratelimit-reset: 1742738168 

POST

x-ratelimit-limit: 0
x-ratelimit-remaining: 0
x-ratelimit-reset: 1742739261 

I also opened an issue on:
https://github.com/clearlydefined/service/issues/
with number: 1296

To get an better understanding on how this should works

[HannesWell]

Many projects have integrated Dash License Tool calls into pull request workflows, for example. My (possibly naive) hypothesis is that each individual project is making a reasonable number of calls, but since these are running on shared infrastructure, during periods of high activity, the ClearlyDefined server is getting hammered in aggregate and is throttling. We're also likely competing for resources from other workflows in unrelated GitHub projects that use ClearlyDefined.

I assume that this is not a bad guess. But maybe Clearly defined can be asked to handle such 'big-players' like Github less strict. IIRC their IPs are well known and there is even a GH-API for their IPs.

Besides that I don't know a better solution than waiting longer based on an educated guess as Stefan suggests.

[stbischof]

An other option would be that dash becomes a server component and all ci jobs fire against this server which then handles the right limits.

This server could also lookup for new snapshots and releases of the eclipse foundation and check them.

[waynebeaton]

handle rate limit in httpClientService #450

IMHO, respecting rate limiting will buy us a little time, perhaps, to sort out a better solution. The problem is that multiple processes running on the same infrastructure are hammering on the API. Each process pausing for a bit while the others each take their turn and pause, and then everything waking up AT THE SAME TIME might prevent a small number of failures, but I'm thinking that it might actually amplify the problem.

I believe that we get a bigger win by just not using the tool as often. There is no reason to run the Eclipse Dash License Tool on every pull request. If project teams just run it weekly, then the likelihood of running into problems becomes much smaller.

An other option would be that dash becomes a server component and all ci jobs fire against this server which then handles the right limits.

Probably. But, I don't have resources to build -- and, more importantly, maintain and run -- a custom server-based application.

That the Eclipse Dash License Tool runs on the client was a key design feature. If we're going to deploy anything on the server, it'll be something like ORT (which it still on my list of things that we'd like to do).

This server could also lookup for new snapshots and releases of the eclipse foundation and check them.

Move it to the server AND make it actually go looking for work to do? Nope. This is out of scope for the Eclipse Dash License Tool. Perhaps another project could leverage Dash.

If we're going to do this, then we'd deploy ORT.

[stbischof]

Just for list all Options in my mind. Not saying that all are good, and in know that there are cons against that you described before.

• set an default proxy over that the comminication is done
• do not use dash to collect data and run a component that takes dependencies from GitHub inights dependencies endpoint and centralise the requests.

I am pritty Happy that dash runs on each pr..

[waynebeaton]

I am pritty Happy that dash runs on each pr..

Why? Even when you're not changing any dependencies?

I understand that there is some non-zero chance that our assessments could change the results, but every pr... really?

[stbischof]

Its just convinient, i would forget to start it ever x pr, or everythime when i should think on that.
An if i do it scheduled i would not see that it fails.
And so others could see (especially in the js/ts world) what stack of software this little change brings.

[boaks]

I believe that we get a bigger win by just not using the tool as often.

FMPOV, adding dependencies needs to be checked ahead. Running dash-license is then the verification of that check. In some cases also updates may cause new indirect dependencies, but that is more a question to use only well managed projects as dependency. So I would guess, as fast solution, a job, which runs "dash-license" once a week will do it very well. If someone has doubts about a new or updated dependency, it's also possible to run it locally.

[stbischof]

clearlydefined/service#1296 (comment)

After my issue about the caclulation (error) of the rate limits they seems so change something. looks very promising.

lets see, when they deploy from
https://dev-api.clearlydefined.io/
to
https://api.clearlydefined.io/

If someone also wants to test:
https://dev-api.clearlydefined.io/api-docs/#/definitions/post_definitions

[waynebeaton]

It occurs to me that having the bots authenticate against ClearlyDefined might solve the problem.

clearlydefined/service#1296 (comment)

[stbischof]

Sure,

but i am also sure about that the the x-ratelimit that was calculates was wrong on a factor of more that 20.
So would be interesting to see what happens if them deploy the changes to the production.

[waynebeaton]

AFAICT, ClearlyDefined does not appear to formally provide any support for providing a higher rate limits for queries when when you authenticate.

I decided to try it with my GitHub OAUTH anyway and had some disappointing results.

$ curl "https://api.clearlydefined.io/definitions/pypi/pypi/-/starlette/0.47.2" -H  "Authorization: token $GITHUB_OAUTH" -v

My first attempt resulted in an HTTP 401 (unauthorised) which suggests that they do look at the token.

Making the call a second time results in an HTTP 200 and payload result. This just seems weird.

I'm concerned that there may be no path forward.

[stbischof]

This is standard behavior because HTTP Basic Authentication is defined as a challenge-response mechanism.
The client first sends a request without an Authorization header → the server replies with 401 Unauthorized and a WWW-Authenticate: Basic header, indicating that authentication is required.
Only then does the client resend the request with credentials. This flow is specified in the HTTP standard (RFC 7617).