Skip to content

Unify E2E test helpers: centralized error handling, logging, and resource assertions #16032

@davidfowl

Description

@davidfowl

Summary

The CLI E2E tests have grown organically and would benefit from a unified helper layer that centralizes common patterns, error handling, and diagnostics.

Current state

  • CliE2EAutomatorHelpers.cs has AspireStartAsync/AspireStopAsync with good error handling (log dumping, dashboard validation)
  • Individual tests duplicate patterns for aspire init, aspire add, aspire restore, aspire run + Ctrl+C
  • Some tests use CreateDockerTestTerminal, others use CreateTestTerminal — different environments with different capabilities
  • Error diagnostics (log capture, JSON output inspection) are inline in AspireStartAsync but not available for other commands
  • Resource graph assertions are ad-hoc (e.g., checking file existence, grepping output)

Proposed improvements

1. Reusable resource graph assertions

  • Parse aspire describe --format json output into typed objects
  • Link the shared ResourceJson model (src/Shared/Model/Serialization/ResourceJson.cs) into the E2E test project
  • Build assertion helpers: AssertResourcesExistAsync, AssertResourceHasRelationshipAsync, AssertResourceStateAsync, etc.
  • Any E2E test that cares about resource materialization can use these

2. Centralized error handling and diagnostics

  • Extract the log-dumping logic from AspireStartAsync into a reusable helper that any command can use on failure
  • Standardize how tests capture and surface CLI logs, detach logs, and JSON output on failure
  • Make it easy to add diagnostic capture to any aspire command, not just start

3. Consistent terminal setup patterns

  • Document when to use CreateDockerTestTerminal vs CreateTestTerminal
  • Consider a unified factory that picks the right terminal based on what the test needs (Docker socket, Node.js, etc.)

4. Verbose logging control

  • Add a helper to enable/disable verbose CLI logging (--log-level) consistently across tests
  • Centralize log level configuration so debugging a failing E2E test is as simple as flipping a flag

Context

This came up during PR #15901 where we needed to assert un-awaited promise chains actually created resources at runtime. We added AssertResourcesExistAsync as a first step, but it uses node to parse JSON in the container rather than the typed ResourceJson model. A proper solution would make the shared model available to E2E tests.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions