Skip to content

Conversation

aviau
Copy link
Member

@aviau aviau commented May 15, 2025

  • Add scroll_events which includes the logic to call an an events api with all uids returned from a feed api.
  • We will be able to use this exact method even when we add a new events api.
  • This would be our only method that includes ratelimiting, we have no choice but to implement it ourselves since we don't pass control back to the user (because of empty pages).
  • For now, I created a basic ratelimiter, but I kept it as a private argument. So this assumes that the users do not use this in two threads, and that they are happy with our default ratelimits.

I don't want to expose the limiter arg as public right now because I think in the future this can just be replaced by reading ratelimit headers from our API. So, for now, I focus on supporting this simple common use case. We can revisit when we get more feedback and after we have implemented ratelimit headers.

This allow us to replace this example by:

import datetime

from flareio import FlareApiClient


api_client = FlareApiClient.from_env()

from_timestamp: str = (
    datetime.datetime.now(tz=datetime.timezone.utc) - datetime.timedelta(hours=1)
).isoformat()

for event, next in api_client.scroll_events(
    method="POST",
    pages_url="/firework/v4/events/global/_search",
    events_url="/firework/v2/activities/",
    json={
        "size": 10,
        "order": "asc",
        "from": last_from,
        "filters": {
            "type": ["chat_message"],
            "estimated_created_at": {"gte": from_timestamp},
        },
        "query": {
            "type": "query_string",
            "query_string": "hello",
        },
    },
):
    print(f"Full event data: {event}")
    print(f"The next execution could resume using from={next}")

@aviau aviau force-pushed the aviau/scroll_events branch 11 times, most recently from 237d3c1 to c9162c5 Compare May 15, 2025 12:46
@aviau aviau force-pushed the aviau/scroll_events branch from c9162c5 to c4a6802 Compare May 15, 2025 12:49
@gpratte-flare
Copy link

gpratte-flare commented May 15, 2025

lgtm!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants