Reduce number of API calls for large filters

[Bisaloo]

openalexR will split large filters in multiple requests of 50 values:

openalexR/R/oa_fetch.R

Lines 114 to 125 in e116c3e

	# overcome OA limitation of combining 50 values (OR) for a filter at a time
	# https://docs.openalex.org/how-to-use-the-api/get-lists-of-entities/filter-entity-lists#addition-or
	# here, we assume there is only ONE "large" filter
	large_filter <- which(lengths(filter) > 50)
	if (length(large_filter) == 0) {
	list_id <- list(`1` = NULL)
	} else {
	list_id <- split(
	filter[[large_filter]],
	ceiling(seq_along(filter[[large_filter]]) / 50)
	)
	}

However, it seems that the limit of 50 was raised to 100 last year (or at least the docs were updated at that time):

ourresearch/openalex-docs@db68a28

Could openalexR be updated to reflect this so the number of API calls can be cut in half?

[rkrug]

I would actually go one step further, and to be able to use a customised limit. One example where this would make sense, is when I want to check which DOIs from a given list of let’s say 100 DOIs is in a search which is already quote long. I had a case where I had to reduce it to 25 DOIs as otherwise the dredges 4096 maximum API limit would =have been reached. Cheers Rainer

…

On 29 Oct 2025, at 13:44, Hugo Gruson ***@***.***> wrote: Bisaloo created an issue (ropensci/openalexR#349) <#349> openalexR will split large filters in multiple requests of 50 values: https://github.com/ropensci/openalexR/blob/e116c3ede88cc3de6ddbb0a9b2c7739e3af9ab0e/R/oa_fetch.R#L114-L125 However, it seems that the limit of 50 was raised to 100 last year (or at least the docs were updated at that time): ***@***.*** <ourresearch/openalex-docs@db68a28> Could openalexR be updated to reflect this so the number of API calls can be cut in half? — Reply to this email directly, view it on GitHub <#349>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADW6BEEQHBUWUFP72A7TMD32CZBLAVCNFSM6AAAAACKRQYXX2VHI2DSMVQWIX3LMV43ASLTON2WKOZTGU3DKOBQGQ3TSMA>. You are receiving this because you are subscribed to this thread.

[rkrug]

Just ignore - it is already the case.

…

On 29 Oct 2025, at 14:14, Rainer M Krug ***@***.***> wrote: I would actually go one step further, and to be able to use a customised limit. One example where this would make sense, is when I want to check which DOIs from a given list of let’s say 100 DOIs is in a search which is already quote long. I had a case where I had to reduce it to 25 DOIs as otherwise the dredges 4096 maximum API limit would =have been reached. Cheers Rainer > On 29 Oct 2025, at 13:44, Hugo Gruson ***@***.***> wrote: > > > Bisaloo > created an issue > (ropensci/openalexR#349) > <#349> > openalexR will split large filters in multiple requests of 50 values: > > https://github.com/ropensci/openalexR/blob/e116c3ede88cc3de6ddbb0a9b2c7739e3af9ab0e/R/oa_fetch.R#L114-L125 > > However, it seems that the limit of 50 was raised to 100 last year (or at least the docs were updated at that time): > > ***@***.*** <ourresearch/openalex-docs@db68a28> > Could openalexR be updated to reflect this so the number of API calls can be cut in half? > > — > Reply to this email directly, view it on GitHub <#349>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADW6BEEQHBUWUFP72A7TMD32CZBLAVCNFSM6AAAAACKRQYXX2VHI2DSMVQWIX3LMV43ASLTON2WKOZTGU3DKOBQGQ3TSMA>. > You are receiving this because you are subscribed to this thread. >

[trangdata]

Thank you @Bisaloo for raising this issue. Would love a PR from you!

[Bisaloo]

There is a complication: at some point, we get:

! HTTP status 400 Request Line is too large

It seems that this happens because the request itself really is too large. I can get it to happen with a variable number of DOI simply by including or not the "https://doi.org/" prefix.

With half the entries having the doi.org prefix, it happens around 90-95 entries. With all the entries having the doi.org prefix, it happens around 70-75 entries.

There are things openalexR could do to mitigate this issue, like stripping the doi.org prefix, but it doesn't solve the underlying issue that filters can fail at a arbitrary number of DOIs, depending on the length of each individual DOI.

So the current approach of splitting by 50 results in more requests but is less likely to trigger the error.

I'm not sure what is the best approach here.

[Bisaloo]

Or is it what you meant by the 4096 maximum API limit, @rkrug?

[rkrug]

Yup - that is what I meant. It is getting more extreme, when you combine a list of DOIS with a longer search. In which case I had to reduce to 10 DOIs to make it work. Is there any way of estimating the length of the request before it is send? Woulds make finding the "best" size much easier.
\