Fetch a Teamtailor description only for a posting we do not have

[strelov1]

The waste

The Teamtailor crawl fetched every posting's detail page every hour, because that is where
the description lives. Measured on prod 2026-08-16:

live postings	~40,000
genuinely new in the last hour	1
new in the last 3 hours	341
detail requests per run	36,771

So a run spent ~36.7k requests to discover ~100 postings, and fired them in ten minutes — about
62 req/s at one career-site vendor. That volume is what Teamtailor's edge turned away (1208 of
1987 boards 403'd), and what #1989 and #1992 could only recover or spread out rather than remove.

The fix already had a seam

HydratingSource exists for exactly this and thirteen adapters already implement it —
workday among them, a per-tenant ATS like this one. The pipeline supplies a seen(externalID)
predicate; the adapter hydrates only what the catalogue lacks.

FetchNew enumerates the board exactly as before and then:

• unseen posting → hydrated from its detail page, as today;
• seen posting → emitted as a liveness refresh carrying identity alone, no request.

That second branch is load-bearing rather than an optimisation detail: the pipeline routes a
SeenRefresh to a liveness touch instead of a write, because a content-less re-upsert would
re-derive the facets from an empty description and wipe them.

Fetch is untouched and still hydrates everything — it is the fallback used when the pipeline
cannot supply a seen set, and the worst case the pacer's interval is still sized for.

Expected effect

A steady-state run should cost ~4k requests instead of ~41k, nearly all of them the listing
pages that enumerate the board. That also unwinds the trend this exposed: run duration had crept
31 → 42 minutes against the unit's 50-minute TimeoutStartSec, and it was creeping because the
earlier fixes worked — more boards answering means more postings found means more detail
fetches.

Tests

The request count is the deliverable, so it is asserted directly: three postings listed with two
already seen makes exactly three requests (two listing pages, one detail). Plus: a seen posting
is marked SeenRefresh and carries no content, an unseen one is hydrated, an all-seen board
touches no detail page at all, a failed listing still fails the board (an empty result would let
the unseen sweep close everything the board still has), and Fetch still hydrates everything.

Summary by CodeRabbit

• Performance

  • Improved incremental job updates by avoiding unnecessary detail requests for postings that have already been seen.
  • Maintained full detail loading during standard job board refreshes.
  • Added bounded concurrent loading for posting details.

• Reliability

  • Preserved listing-only behavior when all postings are already known.
  • Improved handling and propagation of listing failures.

• Tests

  • Added coverage for incremental updates, unseen postings, fully known boards, failures, and complete refreshes.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: c33ae318-f29e-4f04-800b-a33733d7fca6

📥 Commits

Reviewing files that changed from the base of the PR and between a1c2440 and 9dd1483.

📒 Files selected for processing (3)

• internal/sources/pacer.go
• internal/sources/teamtailor.go
• internal/sources/teamtailor_hydrate_test.go

Included review availability: Your plan includes up to 2 reviews per rolling hour; 1 remains after this review.

---

📝 Walkthrough

Walkthrough

Teamtailor now enumerates listing URLs separately from detail hydration. FetchNew hydrates only unseen postings and emits refresh records for seen postings. Fetch continues to hydrate every posting. Tests cover request counts, errors, and fallback behavior.

Changes

Teamtailor hydration flow

Layer / File(s)	Summary
Listing and fallback hydration internal/sources/teamtailor.go, internal/sources/pacer.go	Fetch enumerates posting URLs, hydrates each posting with bounded concurrency, and retains the existing fallback pacing.
Selective FetchNew hydration internal/sources/teamtailor.go	FetchNew skips details for seen native IDs, emits SeenRefresh records, skips postings without IDs, and hydrates unseen postings.
Hydration behavior validation internal/sources/teamtailor_hydrate_test.go	Tests verify selective detail requests, all-seen listings, listing failures, and full hydration through Fetch.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to 9dd14

The change limits detail-page fetching to postings the catalogue does not already contain while preserving liveness refreshes for known postings; no actionable merge-blocking risk remains after normal checks and review.

Sequence Diagram(s)

sequenceDiagram
  participant FetchNew
  participant jobURLs
  participant SeenSet
  participant DetailEndpoint
  FetchNew->>jobURLs: enumerate all posting URLs
  jobURLs-->>FetchNew: return URL list
  FetchNew->>SeenSet: check native posting ID
  alt Posting is seen
    FetchNew-->>FetchNew: emit SeenRefresh identity
  else Posting is unseen
    FetchNew->>DetailEndpoint: fetch posting details
  end

Loading

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main Teamtailor change: fetch descriptions only for postings that are not already in the catalogue.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch teamtailor-hydrate-new-only

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}