Pace the Teamtailor crawl to stop producing the 403s - #1992
Merged
Conversation
A run fires about 37k requests — ~4k listing pages plus one detail page per each of 33375 postings — and it fired them in ten minutes: roughly 62 req/s at one career-site vendor. Teamtailor turned away nearly half the fleet, 1208 of 1987 boards. The refusal-retry proxy (#1989) recovered only a quarter of that, because a reputation 403 follows the volume onto whatever address carries it: the proxy is one shared IP and ~900 refused requests per run saturate it too. Pacing removes the burst instead of moving it, so the two compose — the pacer stops producing refusals, the proxy recovers the ones still produced. The interval comes from the run budget rather than a guess at Teamtailor's window: 37k requests must finish inside the unit's TimeoutStartSec of 3000s, which puts the floor near 12 req/s. 20 req/s sits above it with room, landing a full run around 31 minutes against the 10 it takes now. A test pins the arithmetic so neither number can move alone — too fast keeps the 403s, too slow gets the run killed mid-crawl, and a killed run is indistinguishable from a platform outage in board_health. Paced at both wirings, as careerspage is: the registry one covers local, dev and any run without a proxy; the proxied one covers prod.
|
Warning Review limit reached
Next review available in: 38 minutes Limit details: You’ve used all 2 included reviews currently available under your plan. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
strelov1
added a commit
that referenced
this pull request
Aug 16, 2026
The crawl fetched every posting's detail page every hour. Measured on prod: ~40k live postings on the board file, about ONE an hour genuinely new — so a run spent ~36.7k detail requests to discover ~100 postings, and fired them in ten minutes. That volume is what Teamtailor's edge turned away, 1208 of 1987 boards 403'd, and what pacing (#1992) could only spread out rather than remove. HydratingSource is the seam for exactly this and thirteen adapters already implement it, workday among them — a per-tenant ATS like this one. FetchNew enumerates the board as before but hydrates only ids the catalogue lacks; a seen posting is emitted as a liveness refresh carrying identity alone, which the pipeline routes to a touch instead of a write. That distinction is load-bearing: a content-less re-upsert would re-derive the facets from an empty description and wipe them. Fetch keeps hydrating everything. It is the fallback the pipeline uses when it cannot supply a seen set, and the case the pacer's interval is still sized for. A steady-state run should now cost ~4k requests instead of ~41k — nearly all of them the listing pages that enumerate the board.
King70870
pushed a commit
to King70870/freehire-1
that referenced
this pull request
Aug 16, 2026
A run fires about 37k requests — ~4k listing pages plus one detail page per each of 33375 postings — and it fired them in ten minutes: roughly 62 req/s at one career-site vendor. Teamtailor turned away nearly half the fleet, 1208 of 1987 boards. The refusal-retry proxy (strelov1#1989) recovered only a quarter of that, because a reputation 403 follows the volume onto whatever address carries it: the proxy is one shared IP and ~900 refused requests per run saturate it too. Pacing removes the burst instead of moving it, so the two compose — the pacer stops producing refusals, the proxy recovers the ones still produced. The interval comes from the run budget rather than a guess at Teamtailor's window: 37k requests must finish inside the unit's TimeoutStartSec of 3000s, which puts the floor near 12 req/s. 20 req/s sits above it with room, landing a full run around 31 minutes against the 10 it takes now. A test pins the arithmetic so neither number can move alone — too fast keeps the 403s, too slow gets the run killed mid-crawl, and a killed run is indistinguishable from a platform outage in board_health. Paced at both wirings, as careerspage is: the registry one covers local, dev and any run without a proxy; the proxied one covers prod.
King70870
pushed a commit
to King70870/freehire-1
that referenced
this pull request
Aug 16, 2026
…elov1#1994) The crawl fetched every posting's detail page every hour. Measured on prod: ~40k live postings on the board file, about ONE an hour genuinely new — so a run spent ~36.7k detail requests to discover ~100 postings, and fired them in ten minutes. That volume is what Teamtailor's edge turned away, 1208 of 1987 boards 403'd, and what pacing (strelov1#1992) could only spread out rather than remove. HydratingSource is the seam for exactly this and thirteen adapters already implement it, workday among them — a per-tenant ATS like this one. FetchNew enumerates the board as before but hydrates only ids the catalogue lacks; a seen posting is emitted as a liveness refresh carrying identity alone, which the pipeline routes to a touch instead of a write. That distinction is load-bearing: a content-less re-upsert would re-derive the facets from an empty description and wipe them. Fetch keeps hydrating everything. It is the fallback the pipeline uses when it cannot supply a seen set, and the case the pacer's interval is still sized for. A steady-state run should now cost ~4k requests instead of ~41k — nearly all of them the listing pages that enumerate the board.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The measurement
A Teamtailor run makes ~37,000 requests — ~4k listing pages plus one detail page for each
of 33,375 postings, because that is where the description lives — and the last run made them
in ten minutes. That is roughly 62 req/s aimed at a single career-site vendor.
Teamtailor turned away 1208 of 1987 boards. The refusals are ours, not a block: a "failing"
board answers 200 on demand from the same IP, and 1207 of the 1208 failures carried a timestamp
inside the crawl hour.
Why the proxy was not enough
#1989 sends a refused request out through the proxy, and that recovered only a quarter of
teamtailor's failures (1208 → 909) while doing most of the job for workable (1018 → 303).
The difference is what the two status codes mean. Workable's 429 is a per-IP counter and a
fresh address resets it. Teamtailor's 403 is its edge judging the address, so it follows the
volume onto whatever IP carries it — and our proxy is one shared address that ~900 refused
requests per run saturate on their own.
So the two changes compose rather than compete: the pacer stops producing refusals, the proxy
recovers the ones still produced.
Choosing the rate
From the run budget, not from a guess at Teamtailor's window:
TimeoutStartSec)A test pins that arithmetic, so neither the interval nor the volume assumption can move alone.
Too fast keeps the 403s; too slow gets the run killed mid-crawl — and a killed run is
indistinguishable from a platform outage in
board_health.This is the first deliberate rate this platform has had, so it should be tuned from observed
convergence: downward while boards are still refused, upward only while they are not.
Wiring
Paced at both call sites, the way careerspage already is — the registry one covers local, dev
and any run without a proxy, the proxied one covers prod. A test asserts both, since pacing only
one leaves either prod or dev still firing the burst.