Skip to content

Add a consistency check for the SDK metrics reference - #4986

Merged
Duncanma merged 5 commits into
mainfrom
chore/check-metrics-reference
Jul 30, 2026
Merged

Add a consistency check for the SDK metrics reference#4986
Duncanma merged 5 commits into
mainfrom
chore/check-metrics-reference

Conversation

@Duncanma

@Duncanma Duncanma commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a CI check that validates the internal consistency of docs/references/sdk-metrics.mdx, so the class of drift fixed in #4985 cannot silently return.

The page states the same facts twice — once in the summary table, once in the per-metric section — and nothing kept them in sync. That is how the table came to link two metrics to each other's anchors and to disagree with five sections about which SDKs emit a metric.

Stacked on #4985. Based on fix/sdk-metrics-reference-accuracy rather than main, because the check asserts the reference page is clean and main still has the 14 problems that #4985 fixes. GitHub will retarget this to main when the parent merges.

What it checks

  • Every table anchor matches its metric name
  • Table rows and sections map one-to-one
  • Table and section agree on metric type and availability
  • Headings are backticked metric names, keeping anchors predictable
  • SDK names, metric types, "Emitted by" values, and tag names come from a known vocabulary
  • No metric description claims a unit, which would contradict the units admonition

The tag vocabulary is a single constant checked against both the shared key list at the top of the page and each metric's own Tags line, so a hyphen/underscore slip in either place fails rather than merely looking inconsistent.

Does it work?

Replayed against the pre-fix page from main, it reports 14 problems — every bug found by hand during the #4985 audit except the two that were wrong consistently in both the table and the section:

shared tag list: unknown tag `task-queue`
activity_task_error: availability disagrees (table "Go", section "Go,")
nexus_poll_no_task: availability disagrees (table "Core, Go, Java", section "Go, Java")
nexus_task_execution_failed: availability disagrees (table "Core, Go, Java", section "Go, Java")
nexus_task_execution_latency: availability disagrees (table "Core, Go, Java", section "Go, Java")
nexus_task_endtoend_latency: availability disagrees (table "Core, Go, Java", section "Go, Java")
num_pollers: availability disagrees (table "Core, Go", section "Core, Go, Java")
resource_slots_mem_usage: table links to #resource_slots_cpu_usage
resource_slots_cpu_usage: table links to #resource_slots_mem_usage
unregistered_activity_invocation: availability disagrees (table "Go", section "Go,")
activity_schedule_to_start_latency: description says "in seconds", which contradicts the units admonition
activity_task_error: unknown SDK "" in section availability
nexus_task_schedule_to_start_latency: description says "in seconds", which contradicts the units admonition
unregistered_activity_invocation: unknown SDK "" in section availability

Scope and limits

It reads no SDK source. It cannot tell you a documented metric name is wrong — it would not have caught local_activity_succeeded_endtoend_latency, nor sticky_cache_total_forced_eviction missing Core, because those were wrong consistently in both places. Catching those needs a scheduled diff against the SDK repos, which is a separate, more fragile piece of work worth doing next. The header comment states this so a green check is not mistaken for "verified against the SDKs."

No alphabetical-ordering rule. The table is mostly sorted, but the five Nexus rows are not, so the rule would fail today and force a reorder outside this PR's scope.

Implementation notes

Follows the existing bin/check-redirects-for-moved-pages.js pattern: exported functions, a main() that prints problems and exits non-zero, plus a colocated .test.js. Runnable locally as yarn check:metrics.

The workflow needs no yarn install since everything is plain Node, and it runs the checker's own unit tests before the checker itself.

Test plan

  • yarn check:metrics passes on the current page (48 metrics consistent)
  • 22 new tests pass, one per rule, each seeded with a bug that actually shipped
  • Full suite green, 39 to 61 tests
  • Replay against the pre-fix page reports the 14 problems above

┆Attachments: EDU-6840 Add a consistency check for the SDK metrics reference

Duncanma added 3 commits July 28, 2026 15:52
Rename local_activity_succeeded_endtoend_latency to
local_activity_succeed_endtoend_latency to match Core, Go, and Java.
Fix the swapped resource_slots_cpu_usage and resource_slots_mem_usage
anchors. Add Core to the four Nexus metric sections and to
sticky_cache_total_forced_eviction, add Java to num_pollers, and drop
the "Go and Java only" note from nexus_task, LocalActivityWorker, and
NexusWorker. Remove the "in seconds" claims that contradict the units
note, fix task-queue to task_queue, and drop two trailing commas.
Add break opportunities before underscores in table of contents entries,
tighten the leading within an entry, and hang continuation lines.
Validate that the summary table and per-metric sections agree on anchor,
type, and availability, that headings are backticked metric names, and
that SDK names, metric types, emitters, and tags come from a known
vocabulary. Runs on pull requests that touch the page.
@Duncanma
Duncanma requested a review from a team as a code owner July 28, 2026 23:14
@vercel

vercel Bot commented Jul 28, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
temporal-documentation Ready Ready Preview Jul 30, 2026 11:01pm

Request Review

@github-actions

github-actions Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

📖 Docs PR preview links

This PR does not change any pages in /docs. If you make updates, links to the modified pages will appear here.

@Duncanma
Duncanma marked this pull request as draft July 29, 2026 14:49
Base automatically changed from fix/sdk-metrics-reference-accuracy to main July 30, 2026 20:09
@Duncanma
Duncanma marked this pull request as ready for review July 30, 2026 20:17
@Duncanma
Duncanma enabled auto-merge (squash) July 30, 2026 20:17
@Duncanma
Duncanma merged commit 595ea0f into main Jul 30, 2026
14 checks passed
@Duncanma
Duncanma deleted the chore/check-metrics-reference branch July 30, 2026 23:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants