Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,12 +91,12 @@

- [ ] **Tier 2 container isolation (DEFERRED — trigger-gated per ADR 0001 section 5; implement only if a trust-model trigger fires, e.g. untrusted generation, multi-tenant host)**

- [ ] **Fix missing `configs/safety.yaml` and `config/` vs `configs/` path discrepancy**
- [x] **Fix missing `configs/safety.yaml` and `config/` vs `configs/` path discrepancy** _(done 2026-07-27)_
- Multiple safety-critical modules reference `configs/safety.yaml` (plural `configs/`) as the immutable safety configuration, but neither that file nor a `configs/` directory exists
- Only `config/` (singular) exists, containing `budget.yaml`, `logging.yaml`, etc. — no `safety.yaml`
- Referenced by: `evoseal/core/edit_scope_validator.py` (lines 27, 52, 80), `evoseal/core/safety_integration.py` (line 286), `evoseal/core/testrunner.py` (line 652), `evoseal/cli/commands/doctor.py` (line 181), and multiple safety tests
- Impact: `evoseal doctor` reports 'safety.yaml not found'; edit-scope allowlist references a nonexistent path; safety tests assert against a file that does not exist on disk
- Resolve by deciding whether the canonical path is `config/safety.yaml` or `configs/safety.yaml`, creating the file with appropriate defaults, and updating all references to match
- Created `configs/safety.yaml` with defaults for edit-scope enforcement, regression detection thresholds, checkpoint/rollback policy, and sandboxed test execution. Code already references `configs/safety.yaml` (plural) consistently — no path changes needed. Verified: `check_safety_yaml()` returns well-formed, `pytest tests/safety/ tests/unit/test_doctor_command.py` passes (54 tests), `evoseal doctor` check passes.

### Integration Testing

Expand Down Expand Up @@ -304,10 +304,10 @@
| Priority | Total | Done | Notes |
|----------|-------|------|-------|
| 🔴 P0 | 11 | 11 | Original 5 complete; all 6 critical bugs from 2026-07-22 whole-repo review fixed (PRs #74, #76-#79) |
| 🟠 P1 | 24 | 13 | Original safety/integration items done; +12 high-priority bugs from 2026-07-22 review; signal-handler init fix |
| 🟠 P1 | 24 | 14 | Original safety/integration items done; +12 high-priority bugs from 2026-07-22 review; signal-handler init fix; safety.yaml created |
| 🟡 P2 | 30 | 16 | Co-evolution loop gaps (7 items, 7 done) + existing P2 + 13 medium bugs from 2026-07-22 review + 4 latent collect->train bugs found closing the loop (1 fixed, 1 new HF-format gap logged) |
| 🟢 P3 | 24 | 11 | Makefile, pre-commit, Docker, ADRs, ADR refresh complete; +10 hygiene items from 2026-07-22 review |
| **Total** | **89** | **51** |
| **Total** | **89** | **52** |

> Update this table as you complete items. Recommended flow: P0 → P1 → P2 → P3.
>
Expand Down
83 changes: 83 additions & 0 deletions configs/safety.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
# EVOSEAL Safety Configuration (Immutable)
#
# This file defines safety thresholds and constraints for the evolution pipeline.
# It is enforced as IMMUTABLE by the edit-scope allowlist — the evolution loop
# must never modify this file. See threat model §3 and ADR 0001.
#
# NOTE: This directory is `configs/` (plural), NOT `config/` (singular).
# `config/` is a Python package containing mutable settings (budget.yaml,
# logging.yaml, settings.py). `configs/` holds standalone immutable YAML
# files referenced by path. Both are intentional — do not merge them.

# Edit-scope enforcement
edit_scope:
enabled: true
# Paths that the evolution loop is forbidden from modifying
forbidden_paths:
- .env
- Makefile
# Directories that are entirely off-limits
forbidden_dirs:
- .git
- .github/workflows
- .evoseal
- .pytest_cache
- __pycache__
- .venv
- venv
- configs
# Directories that ARE mutable by the evolution loop
allowed_dirs:
- evoseal/
- tests/
- examples/
- docs/
- scripts/
- benchmarks/
- config/
Comment on lines +13 to +37

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Align the edit-scope schema with its consumer.

SafetyIntegration reads edit_scope.allowed_patterns as a mapping, while this file defines allowed_dirs as a list; enabled is also not connected to the top-level enforcement flags used by SafetyIntegration. As written, these settings are inert and the validator falls back to hard-coded defaults instead of using this immutable configuration.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@configs/safety.yaml` around lines 11 - 35, The edit-scope configuration in
the safety YAML does not match the schema consumed by SafetyIntegration. Update
the edit_scope keys and value types to use allowed_patterns as a mapping, and
connect the enabled setting to the top-level enforcement flags expected by
SafetyIntegration, preserving the existing forbidden paths and directories.
Ensure the resulting immutable configuration is read instead of triggering
validator defaults.


# Regression detection thresholds
regression:
# Global default regression threshold (5%)
regression_threshold: 0.05
# Per-metric thresholds (positive = increase is regression, negative = decrease is regression)
metric_thresholds:
# Performance metrics (lower is better)
duration_sec:
regression: 0.10
critical: 0.25
memory_mb:
regression: 0.10
critical: 0.30
execution_time:
regression: 0.10
critical: 0.25
# Quality metrics (higher is better — negative thresholds)
success_rate:
regression: -0.05
critical: -0.10
pass_rate:
regression: -0.05
critical: -0.10
correctness:
regression: -0.01
critical: -0.05
Comment on lines +44 to +64

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Preserve thresholds for all monitored metrics.

RegressionDetector uses the supplied metric_thresholds map wholesale rather than merging it with defaults. This omits built-in thresholds for metrics such as accuracy and error_rate, both of which are in the detector’s default monitored set, so they lose their intended per-metric regression and critical limits. Include the complete map or merge configuration with the defaults.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@configs/safety.yaml` around lines 42 - 62, Expand the metric_thresholds
configuration used by RegressionDetector to include the built-in thresholds for
every default-monitored metric, especially accuracy and error_rate. Preserve the
existing thresholds for duration_sec, memory_mb, execution_time, success_rate,
pass_rate, and correctness, and ensure the supplied map is complete rather than
omitting default entries.


# Checkpoint and rollback policy
checkpoint:
# Maximum number of checkpoints to retain (oldest pruned first)
max_checkpoints: 10
# Automatically rollback on critical regression
auto_rollback_on_critical: true

# Sandboxed test execution
sandbox:
enabled: true
# Files mounted read-only during test execution
readonly_files:
- configs/safety.yaml
- .env
# Resource limits for test subprocesses
cpu_limit_secs: 120
memory_limit_mb: 512
fd_limit: 256
1 change: 1 addition & 0 deletions evoseal/core/edit_scope_validator.py
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ def __init__(self, allowed_patterns: dict[str, bool] | None = None) -> None:
"__pycache__",
".venv",
"venv",
"configs",
}
)

Expand Down
Loading