Pace SEEK's detail hydration and stop storing body-less postings - #2018
Merged
Conversation
The SEEK adapter's first production run ingested 2,982 postings and left 2,593 of them (87%) with no description. Its GraphQL detail endpoint meters by a per-IP request budget. Unpaced, one crawl fired 3,267 detail POSTs in 95 seconds across 43 boards and was answered 429 on essentially all of them, while a single identical request from the same host returned 200 after two minutes idle. An immediate refusal that clears on idle is a window, not saturation, so this adds a rate limiter rather than the in-flight cap trudvsem and emagine use: eight fast requests a second from one worker is still eight a second. The pacer had no JSON-POST wrapper, so this adds one; only the detail path is wrapped, since the search listing is ~150 requests a run and has never been refused. The rate limit alone would not have been permanent damage. What made it so is that a posting whose detail fails was ingested body-less, and seen reports only row existence -- so every later crawl marks it SeenRefresh and it never hydrates again. pacer.go's emagine note already calls this loss permanent. seek now drops such a posting instead, leaving it new so the next crawl retries it. That reverses what hh and the other hydrating adapters do, deliberately and only here: their rule assumes a rare failure, and against a rate-limited endpoint the failure is the common case. Deferring a posting by one crawl is recoverable; storing it body-less is not. The rate is conservative, as every other constant in that file is: the backfill no longer fits one ingest window and accretes over a few runs, which the drop-and-retry rule makes safe.
|
Warning Review limit reached
Next review available in: 34 minutes Limit details: You’ve used all 2 included reviews currently available under your plan. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (11)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
strelov1
added a commit
that referenced
this pull request
Aug 16, 2026
#2018 landed while this branch was open and documented, in three places, that a posting ingested list-only "never re-fetches its detail" and that the loss "is permanent". With the retry window that is no longer true: such a row is re-offered for hydration until it is 14 days old. The distinction still matters and is now stated as the bound it actually is — seek drops a posting rather than storing it body-less precisely BECAUSE a backlog of thousands does not clear inside a bounded window, while a dropped posting stays new with no deadline. That is a stronger argument for what seek does than "the loss is permanent" was, and it is the true one. Refs freehire#1866
strelov1
added a commit
that referenced
this pull request
Aug 16, 2026
* Stop losing and showing job descriptions that never arrived 15,816 live jobs (0.48% of the open catalogue) carry an empty description. Investigating freehire#1866 found three separate causes and one non-cause. smartrecruiters (1,066 rows) excluded companyDescription as boilerplate. For these postings that section is the ONLY text the tenant wrote, so the exclusion stored nothing at all. It is now a preference, not a ban: the company section is used when the three role sections are empty. The hydrating sources (hh, justjoin, getro, nofluffjobs, echojobs, teamtailor — ~3.3k rows) fetch detail only for postings the catalogue lacks. Being stored is what marks a posting seen, so a posting whose one detail fetch failed was never retried and kept an empty body for life. A row without a description is now withheld from the seen-set until it is pipeline.HydrationRetryWindow (14 days) old, so the next crawl re-attempts the fetch. The window is bounded because the other reason a body is missing is that the source published none, and retrying those forever would spend a detail request per crawl on each one. The rest are not our bug: probing Oracle's and Paycom's detail APIs live shows every description field empty at the source — the employer published a requisition with no body. Those postings stay in the catalogue (they are real, and the lifecycle owns their closure) but leave the search index, alongside the already-excluded category-unresolved ones: a vacancy page with a title and nothing under it is not a listing anyone can act on. search.DescriptionMissing tests the VISIBLE text, because a source with an empty rich-text field serves markup with no words in it and the ingest sanitizer legitimately keeps those tags. The exclusion is self-healing — a row re-enters the index the moment a crawl fills its description. Refs freehire#1866 * Give the operator a way to repair the body-less backlog The hydration retry window is measured from created_at, so the rows that accumulated before it existed have already aged past it: no ordinary crawl will ever re-fetch their descriptions. HYDRATION_RETRY_DAYS widens the window for one deliberate run, re-offering every body-less row of a provider to its adapter. Parsed in main before the DB is opened, and an unparseable or non-positive value fails the run rather than falling back to the default — this is set by hand for a one-off repair, where quietly ingesting with the default would look exactly like a repair that found nothing to repair. Not a new worker: the pipeline already knows how to hydrate a posting, and the only thing standing between it and these rows is that being stored marks a posting seen. Widening the window is the whole repair. Refs freehire#1866 * Correct the permanence claim a bounded retry window invalidates #2018 landed while this branch was open and documented, in three places, that a posting ingested list-only "never re-fetches its detail" and that the loss "is permanent". With the retry window that is no longer true: such a row is re-offered for hydration until it is 14 days old. The distinction still matters and is now stated as the bound it actually is — seek drops a posting rather than storing it body-less precisely BECAUSE a backlog of thousands does not clear inside a bounded window, while a dropped posting stays new with no deadline. That is a stronger argument for what seek does than "the loss is permanent" was, and it is the true one. Refs freehire#1866
7 tasks
strelov1
added a commit
that referenced
this pull request
Aug 16, 2026
Both landed and were verified in production: the adapter (#2013) and the pacing fix (#2018). seek-source becomes a main spec, carrying the corrected detail-hydration requirement — paced, and deferring a posting it could not hydrate rather than storing it body-less. The archiver compares MODIFIED scenarios by header and reads a rename as a dropped scenario, so 'Failed detail never drops a posting' keeps its original header and states the reversed behaviour in its body.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
The SEEK adapter (#2013) shipped this evening. Its first production run ingested 2,982 postings and left 2,593 of them (87%) with no description. This fixes both faults behind that, found by running it rather than by reading it.
Fault 1 — the detail endpoint meters by a per-IP request budget. Unpaced, one crawl fired 3,267 GraphQL POSTs in 95 seconds across 43 boards, each detail pool bursting to
defaultDetailWorkers, and was answered 429 on essentially all of them. A single identical request from the same host returned 200 after two minutes idle.An immediate refusal that clears on idle is a window, not saturation — so this adds a rate limiter, not the in-flight cap
trudvsemandemagineuse. Eight fast requests a second from one worker is still eight a second.pacer.gohad wrappers forHTMLGetterandJSONGetterbut not for a JSON POST, so this adds one. Only the detail path is wrapped; the search listing is ~150 requests a run and has never been refused.Fault 2 — that refusal became permanent damage. A posting whose detail failed was ingested body-less, and
seenreports only row existence — so every later crawl marks itSeenRefreshand it never hydrates again.pacer.go's own emagine comment already calls this loss permanent.seek now drops such a posting rather than storing it, leaving it new so the next crawl retries it. This reverses what
hhand the other eight hydrating adapters do, deliberately and only here. Their rule is right where its premise holds — a detail failure is rare, so keeping the posting beats losing it. SEEK breaks the premise: refusals arrive in bursts of thousands. Deferring a posting by one crawl is recoverable; storing it body-less is not. That asymmetry is the whole argument, and it is written at the code.I got this wrong when the adapter shipped: CodeRabbit flagged the retry hole on #2013 and I answered that it was rare in practice on the strength of 31/31 successful local details. Production disagreed within 95 seconds.
Rate
2 req/s (500ms, burst 2) — ~17x below the rate that was refused, and conservative for the reason the file's other constants are: the true budget is unknown, under-shooting only lengthens a run, over-shooting re-enters the penalty. Consequence: the ~7.8k first backfill no longer fits one ingest window and accretes over roughly two to three runs, which is safe only because of fault 2's fix. Steady state is a few hundred new postings a day and finishes in minutes.
Recovery
The 2,982 body-less prod rows are pruned (archived to
pruned_jobs, not lost) so the paced crawl re-ingests them complete. The hourly timer goes on only after a verified run — tracked as tasks 4.1–4.3 in the change.Checklist
go build ./...,go vet ./..., andgofmt -l .(prints nothing) pass.go test ./...passes (andgo vet -tags=integration ./...— no DB/handler code touched).design-system/changes: n/a.web/changes: n/a.