Skip to content

upsertRepoCatalogs hits Convex 32k document read limit during import #3

@natea

Description

@natea

Bug

When importing starred repos, findRepoByFullName in convex/portfolio.ts falls back to a full table scan (.collect()) when the indexed lookup doesn't find an exact match. During first import, every repo is new so the fallback fires on every upsert. As repos accumulate in the table, each call reads more documents until it exceeds Convex's 32,000 document read limit.

Steps to reproduce

  1. Configure GitHub auth and Convex
  2. Click "Import starred repos" on a GitHub account with 500+ starred repos
  3. Observe Convex logs

Expected

Import completes successfully.

Actual

Import fails partway through with:

Uncaught Error: Too many documents read in a single function execution (limit: 32000).
    at async findRepoByFullName (../convex/portfolio.ts:76:0)
    at async handler (../convex/portfolio.ts:439:8)

Warnings escalate as the table grows:

[WARN] Many documents read in a single function execution (actual: 26225, limit: 32000)
[WARN] Many documents read in a single function execution (actual: 28725, limit: 32000)
[WARN] Many documents read in a single function execution (actual: 31225, limit: 32000)

Root cause

findRepoByFullName has two paths:

  1. Indexed lookup via by_fullName — fast, O(1)
  2. Case-insensitive fallback via .collect() — scans entire table

The fallback fires for every new repo (no existing match), turning import into O(n²) reads.

Fix

Remove the case-insensitive fallback. The indexed exact-match lookup is sufficient since GitHub returns consistently-cased fullName values.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions