Improve render tests to catch console errors and UI error boundaries

[ushkinaz]

Summary

Automated tests in src/testRenderMods.ts and src/testRender.ts (and potentially other render tests) fail to detect when a component crashes and is caught by a <svelte:boundary>. Currently, a test will pass even if the page displays an "Error" heading, provided it doesn't contain specific strings like undefined. Additionally, Svelte's internal console.error logs for structural issues (like each_key_duplicate) do not trigger test failures.

Context

• Test Code: src/testRenderMods.ts
• Error Handling: src/Thing.svelte uses <svelte:boundary> which renders an .error block with "Error" text when a child component fails.
• Svelte Runtime: Svelte 5 logs many critical rendering issues directly to console.error.

Impact

• CI/CD Blindness: Regressions that cause certain items to fail rendering are not caught in CI if they trigger the error boundary or just log to the console.
• False Positives: Tests pass while the UI is broken for the user.

Severity

High. This affects the reliability of the entire automated test suite.

Steps to reproduce

1. Run pnpm vitest run src/__mod_tests__/mod.no_global_uniques.test.ts.
2. Observe that the test passes despite the output showing Svelte error: each_key_duplicate.
3. Manually verify that the component is rendered as an "Error" block via the Svelte boundary if you were to look at the DOM output.

Tech details

Two specific improvements are needed:

1. Assertion for Error Boundary: The renderCase in testRenderMods.ts should explicitly check that the rendered output does NOT contain the "Error" text or specific error boundary markers.
2. Console Error Interception: Configure the test environment (e.g., in src/test-setup.ts) to intercept console.error. Any call to console.error (especially those originating from Svelte's runtime) should fail the current test context.

Acceptance criteria

• pnpm vitest run src/__mod_tests__/mod.no_global_uniques.test.ts (and any other test hitting a rendering crash) fails explicitly.
• The failure message clearly indicates if it's due to a console.error or hitting an UI error boundary.

[coderabbitai]

🔗 Related PRs

#119 - chore(testing): isolate mod tests and refactor scripts taxonomy [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

✅ Successfully generated implementation plan.

💡 Use @coderabbitai <feedback> to replan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[coderabbitai]

Coding Plan

Summary

• Global console.error spy installed in src/test-setup.ts with afterEach validation hook — zero-invocation policy enforced across all tests
• Error boundary detection via .error CSS class selector in both src/testRender.ts and src/testRenderMods.ts — directly targets Thing.svelte fallback UI structure
• Failure messages explicitly differentiate between console error interception and UI boundary activation per acceptance criteria

Implementation Steps

Phase 1: Console Error Interception Module

Deploy global console.error surveillance to detect Svelte runtime failures and other subsystem error emissions. This mechanism must operate transparently across all test files without per-file configuration.

Task 1: Install Console Error Spy in Global Test Setup

Instrument console.error interception at test environment initialization.

• Modify src/test-setup.ts to establish vi.spyOn(console, 'error') at module scope
• Store spy reference for access by lifecycle hooks
• Spy must capture all arguments passed to console.error for diagnostic output

Task 2: Implement Per-Test Verification Hook

Configure automatic assertion of zero console.error invocations after each test execution.

• Add afterEach hook in src/test-setup.ts using Vitest's global hook registration
• Check spy's mock.calls array length; if non-zero, aggregate call contents into failure message
• Throw explicit error with format: console.error was called during test:\n{aggregated_messages}
• Execute mockClear() on spy after assertion (both pass and fail paths) to reset state for subsequent tests

🤖 Prompt for AI agents

Modify `src/test-setup.ts` to install a global `console.error` surveillance spy
and an `afterEach` validation hook. Goal: enforce zero-tolerance policy on
`console.error` emissions across all tests without per-file configuration.

- At module scope, call `vi.spyOn(console, 'error')` and store the returned spy
reference in a variable accessible by lifecycle hooks
- The spy must capture all arguments passed to `console.error` for diagnostic
output
- Register an `afterEach` hook using Vitest's global hook API:
  - Inspect `spy.mock.calls` array length
- If non-zero: aggregate all call contents into a single failure message and
throw an explicit error formatted as: `console.error was called during
test:\n{aggregated_messages}`
- Unconditionally call `mockClear()` on the spy after assertion — applies to
both pass and fail paths — to reset state for the next test

Phase 2: Error Boundary Detection Assertions

Implement explicit DOM inspection to detect when <svelte:boundary> fallback UI has been rendered, indicating a component crash was silently absorbed.

Task 1: Add Error Boundary Assertion to Mod Render Tests

Extend render verification in mod test infrastructure.

• Modify src/testRenderMods.ts render test body (within test.each callback)
• Insert assertion after loading-indicator null check, before text content validation
• Use container.querySelector('.error') to detect error boundary DOM structure
• Assert result equals null with message: Rendered output for ${modId}:${type}:${id} triggered error boundary.

Task 2: Add Error Boundary Assertion to Core Render Tests

Apply identical detection mechanism to sharded core tests.

• Modify src/testRender.ts render test body (within test.each callback)
• Insert assertion after loading-indicator null check, before text content validation
• Use container.querySelector('.error') to detect error boundary DOM structure
• Assert result equals null with message: Rendered output for ${type}:${id} triggered error boundary.

🤖 Prompt for AI agents

Instrument both `src/testRenderMods.ts` and `src/testRender.ts` with DOM-level
error boundary detection. Goal: force explicit test failure when
`<svelte:boundary>` fallback UI is rendered instead of silently absorbing the
crash.

For both files, within the `test.each` callback body:
- Insert a `container.querySelector('.error')` check **after** the
`loading-indicator` null check and **before** text content validation
- Assert the result equals `null`
- Use distinct failure messages per file:
- `src/testRenderMods.ts`: `Rendered output for ${modId}:${type}:${id} triggered
error boundary.`
- `src/testRender.ts`: `Rendered output for ${type}:${id} triggered error
boundary.`

The `.error` selector directly targets the `<div class="error">` fallback
structure rendered by `src/Thing.svelte` on child component failure.

Research

TEST INFRASTRUCTURE SCAN COMPLETE.

• src/testRender.ts: Core sharded render tests — executes parameterized render assertions via test.each
• src/testRenderMods.ts: Per-mod render tests — executes parameterized render assertions via test.each
• src/test-setup.ts: Global test setup — currently intercepts window.matchMedia and Element.prototype.animate only. CRITICAL GAPS DETECTED.
• src/Thing.svelte: Error boundary renders <div class="error"> containing <h1>{t("Error")}</h1> on child component failure
• Translation function t(): Returns source strings verbatim in test environments — CONFIRMED OPERATIONAL

Two structural blind spots identified:

1. UI error boundaries absorbed by <svelte:boundary> do not propagate to test failure state
2. Svelte runtime console.error emissions (e.g., each_key_duplicate) are suppressed without triggering assertions

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Modify `src/test-setup.ts` to install a global `console.error` surveillance spy
and an `afterEach` validation hook. Goal: enforce zero-tolerance policy on
`console.error` emissions across all tests without per-file configuration.

- At module scope, call `vi.spyOn(console, 'error')` and store the returned spy
reference in a variable accessible by lifecycle hooks
- The spy must capture all arguments passed to `console.error` for diagnostic
output
- Register an `afterEach` hook using Vitest's global hook API:
  - Inspect `spy.mock.calls` array length
- If non-zero: aggregate all call contents into a single failure message and
throw an explicit error formatted as: `console.error was called during
test:\n{aggregated_messages}`
- Unconditionally call `mockClear()` on the spy after assertion — applies to
both pass and fail paths — to reset state for the next test
===============================================================================

Task: 2

Instrument both `src/testRenderMods.ts` and `src/testRender.ts` with DOM-level
error boundary detection. Goal: force explicit test failure when
`<svelte:boundary>` fallback UI is rendered instead of silently absorbing the
crash.

For both files, within the `test.each` callback body:
- Insert a `container.querySelector('.error')` check **after** the
`loading-indicator` null check and **before** text content validation
- Assert the result equals `null`
- Use distinct failure messages per file:
- `src/testRenderMods.ts`: `Rendered output for ${modId}:${type}:${id} triggered
error boundary.`
- `src/testRender.ts`: `Rendered output for ${type}:${id} triggered error
boundary.`

The `.error` selector directly targets the `<div class="error">` fallback
structure rendered by `src/Thing.svelte` on child component failure.

💡 Iterate on the plan with: @coderabbitai <feedback>

Example Feedback
- `@coderabbitai` You can skip phase 3. Add a simple unit test case for phase 2.
- `@coderabbitai` For design choice 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord!