You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
tests/run-full.sh (#2080, "the honest full run") has not been able to complete on dev since 2026-08-27. Three independent harness defects, found on the 2026-09-14 pre-release run:
The PostgreSQL tier cannot run alembic upgrade head. Revision 0041_secret_settings_encryption imports services.secret_settings inside upgrade(); that import loads services/__init__.py, which imports template_service and config.py, and config.py raises REDIS_URL must include credentials at import time. .github/workflows/pg-migrations.yml exports a dummy REDIS_URL; the script does not. Exporting any credentialed REDIS_URL lets the upgrade reach head 0061 cleanly. (A migration that needs the whole services package and a Redis URL to import is also a coupling worth removing at the source: the revision needs SECRET_SETTING_KEYS and plan_migration, which could live in a leaf module.)
Because of (1) the api tier (the ~1,700 live-backend tests under tests/) has been silently unexercised by the full-suite script for three weeks, which is the exact failure mode #2080 was written to prevent. When the tiers were run by hand on 2026-09-14 the api tier surfaced 27 failures, most of them stale tests that had drifted from code changed in August and September (tracked separately).
Acceptance Criteria
tests/run-full.sh runs every tier to completion on a clean checkout with a live dev stack and reaches the skip audit and summary.
The directory guard globs relative to TESTS_DIR, ignores dot/underscore directories, and reports is classified.
The postgres tier exports the environment the migrations need (at minimum REDIS_URL, matching pg-migrations.yml), or the 0041 import is moved to a leaf module so alembic upgrade head does not need it; ideally both.
The postgres tier also runs the integration-tier PostgreSQL tests (tests/integration/test_alembic_postgres.py, tests/integration/test_postgres_backend.py) and the TRINITY_TEST_PG_URL-gated canary tests, so their skips stop appearing.
tests/harness/audit_skips.py allowlist is reconciled with the skip reasons a fully-run suite emits, each with a written justification; a genuinely new skip reason still fails the run.
A guard (CI or a unit test) exercises the run-full directory classification so the glob cannot silently break again.
Summary
tests/run-full.sh(#2080, "the honest full run") has not been able to complete ondevsince 2026-08-27. Three independent harness defects, found on the 2026-09-14 pre-release run:for _d in tests/*/after the script has already donecd "$(dirname "$0")"(so the cwd istests/). The glob matches nothing, bash leaves the literal, and the guard reportsunclassified test directories: *and exits 1 before theapi,standalone, andpostgrestiers and the skip audit run. Introduced by a646de8 (feat(tests):tests/journeys/tier +run-full.sh --tier journeys#2335, feat(ci): per-PR journey-smoke gate — first agent turn + start/stop against a prebuilt stack #2336). A corrected glob (*/) would then also flag__pycache__/andreports/, which are not inNON_TIER_DIRS.alembic upgrade head. Revision0041_secret_settings_encryptionimportsservices.secret_settingsinsideupgrade(); that import loadsservices/__init__.py, which importstemplate_serviceandconfig.py, andconfig.pyraisesREDIS_URL must include credentialsat import time..github/workflows/pg-migrations.ymlexports a dummyREDIS_URL; the script does not. Exporting any credentialedREDIS_URLlets the upgrade reach head 0061 cleanly. (A migration that needs the whole services package and a Redis URL to import is also a coupling worth removing at the source: the revision needsSECRET_SETTING_KEYSandplan_migration, which could live in a leaf module.)no PostgreSQL reachableon the integration-tier PostgreSQL tests (the postgres tier re-runs onlyunit/), hard-codedRequires running backend with databaseskips intests/unit/test_validation.py, documented-obsolete skips (bug: Execution shows Failed then Success after cleanup (race condition) #378, bug: 108 test files sit in tests/ root, where no CI job runs them #1895, refactor: decompose task_execution_service.py — uncovered top-3 code-health hotspot (dispatch hot path) #2314),shellcheck not installed, and Linux-only tests. The audit would fail on a perfect machine, so a real unexplained skip cannot stand out.Context
Because of (1) the api tier (the ~1,700 live-backend tests under
tests/) has been silently unexercised by the full-suite script for three weeks, which is the exact failure mode #2080 was written to prevent. When the tiers were run by hand on 2026-09-14 the api tier surfaced 27 failures, most of them stale tests that had drifted from code changed in August and September (tracked separately).Acceptance Criteria
tests/run-full.shruns every tier to completion on a clean checkout with a live dev stack and reaches the skip audit and summary.TESTS_DIR, ignores dot/underscore directories, andreportsis classified.REDIS_URL, matchingpg-migrations.yml), or the 0041 import is moved to a leaf module soalembic upgrade headdoes not need it; ideally both.tests/integration/test_alembic_postgres.py,tests/integration/test_postgres_backend.py) and theTRINITY_TEST_PG_URL-gated canary tests, so their skips stop appearing.tests/harness/audit_skips.pyallowlist is reconciled with the skip reasons a fully-run suite emits, each with a written justification; a genuinely new skip reason still fails the run.Technical Notes
tests/run-full.sh,tests/harness/audit_skips.py,src/backend/migrations/versions/0041_secret_settings_encryption.py,src/backend/services/__init__.py,.github/workflows/pg-migrations.yml.tests/journeys/tier +run-full.sh --tier journeys#2335 / feat(ci): per-PR journey-smoke gate — first agent turn + start/stop against a prebuilt stack #2336 (the guard), feat(db): adopt Alembic for PostgreSQL migrations (retire SQLite-only bespoke runner) #1183 (dual-track), bug: watchdog cannot close an execution stale >24.8 days — fabricated duration_ms overflows PostgreSQL INTEGER, and the sweep then fails every cycle forever #2434 (PG tier).