Skip to content

Error when fetching a large GitHub API resource that offers no pagination #206

@raahed

Description

@raahed

In relation to, and as the root cause of, issue #203.

Description

If findsources tries to fetch the tags of a GitHub repository with a very large number of tags (e.g., https://github.com/confluentinc/common with over 27,000 tags), the API times out or returns a Bad Gateway error.

The current implementation tries to fetch all tags starting with v (https://api.github.com/repos/confluentinc/common/git/refs/tags/v) in a generic way, which results in the same issue: the GitHub API times out (the GitHub gateway timeout is 10 seconds).

Possible Solution

In this case, the request for tags needs to be batched, e.g., by fetching each digit after the v one by one in separate requests.
Another option would be to skip repositories like this and treat them as if no tags exist.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions