Skip to content

bug: tests/run-full.sh cannot complete — directory guard globs the wrong cwd, postgres tier lacks REDIS_URL, skip audit allowlist rejects a clean run #2801

Description

@vybe

Summary

tests/run-full.sh (#2080, "the honest full run") has not been able to complete on dev since 2026-08-27. Three independent harness defects, found on the 2026-09-14 pre-release run:

  1. The directory-classification guard aborts every run. It iterates for _d in tests/*/ after the script has already done cd "$(dirname "$0")" (so the cwd is tests/). The glob matches nothing, bash leaves the literal, and the guard reports unclassified test directories: * and exits 1 before the api, standalone, and postgres tiers and the skip audit run. Introduced by a646de8 (feat(tests): tests/journeys/ tier + run-full.sh --tier journeys #2335, feat(ci): per-PR journey-smoke gate — first agent turn + start/stop against a prebuilt stack #2336). A corrected glob (*/) would then also flag __pycache__/ and reports/, which are not in NON_TIER_DIRS.
  2. The PostgreSQL tier cannot run alembic upgrade head. Revision 0041_secret_settings_encryption imports services.secret_settings inside upgrade(); that import loads services/__init__.py, which imports template_service and config.py, and config.py raises REDIS_URL must include credentials at import time. .github/workflows/pg-migrations.yml exports a dummy REDIS_URL; the script does not. Exporting any credentialed REDIS_URL lets the upgrade reach head 0061 cleanly. (A migration that needs the whole services package and a Redis URL to import is also a coupling worth removing at the source: the revision needs SECRET_SETTING_KEYS and plan_migration, which could live in a leaf module.)
  3. The skip audit allowlist does not match reality. With every tier actually run, 147 of 149 skips are "unallowlisted", including reasons the harness itself produces: no PostgreSQL reachable on the integration-tier PostgreSQL tests (the postgres tier re-runs only unit/), hard-coded Requires running backend with database skips in tests/unit/test_validation.py, documented-obsolete skips (bug: Execution shows Failed then Success after cleanup (race condition) #378, bug: 108 test files sit in tests/ root, where no CI job runs them #1895, refactor: decompose task_execution_service.py — uncovered top-3 code-health hotspot (dispatch hot path) #2314), shellcheck not installed, and Linux-only tests. The audit would fail on a perfect machine, so a real unexplained skip cannot stand out.

Context

Because of (1) the api tier (the ~1,700 live-backend tests under tests/) has been silently unexercised by the full-suite script for three weeks, which is the exact failure mode #2080 was written to prevent. When the tiers were run by hand on 2026-09-14 the api tier surfaced 27 failures, most of them stale tests that had drifted from code changed in August and September (tracked separately).

Acceptance Criteria

  • tests/run-full.sh runs every tier to completion on a clean checkout with a live dev stack and reaches the skip audit and summary.
  • The directory guard globs relative to TESTS_DIR, ignores dot/underscore directories, and reports is classified.
  • The postgres tier exports the environment the migrations need (at minimum REDIS_URL, matching pg-migrations.yml), or the 0041 import is moved to a leaf module so alembic upgrade head does not need it; ideally both.
  • The postgres tier also runs the integration-tier PostgreSQL tests (tests/integration/test_alembic_postgres.py, tests/integration/test_postgres_backend.py) and the TRINITY_TEST_PG_URL-gated canary tests, so their skips stop appearing.
  • tests/harness/audit_skips.py allowlist is reconciled with the skip reasons a fully-run suite emits, each with a written justification; a genuinely new skip reason still fails the run.
  • A guard (CI or a unit test) exercises the run-full directory classification so the glob cannot silently break again.

Technical Notes

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    complexity-mediumComplexity: medium (board points 5-8)priority-p2Importantstatus-readyGreenlit and ready for development (vetted; counterpart to status-incubating)theme-devexTheme: DevExtype-bugBug fix

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions