fix: create configs/safety.yaml with sensible defaults

[SHA888]

What

Created configs/safety.yaml — the immutable safety configuration file referenced by multiple safety-critical modules but never actually created.

Why

Multiple modules reference configs/safety.yaml (plural configs/) as the immutable safety configuration:

• evoseal/core/edit_scope_validator.py — lists it as a forbidden/immutable path
• evoseal/core/testrunner.py — mounts it read-only during sandboxed test execution
• evoseal/cli/commands/doctor.py — validates its well-formedness
• evoseal/core/safety_integration.py — references it in docs/comments
• Multiple safety tests assert against it

Only config/ (singular) existed, containing budget.yaml, logging.yaml, etc. — configs/safety.yaml was missing entirely. This caused evoseal doctor to report 'safety.yaml not found' and meant safety tests were asserting against a nonexistent file.

What's in the file

Sensible defaults for:

• Edit-scope enforcement — forbidden paths/dirs, allowed mutable surface
• Regression detection thresholds — per-metric regression/critical levels (matches RegressionDetector defaults)
• Checkpoint/rollback policy — max checkpoints, auto-rollback on critical regression
• Sand-boxed test execution — readonly files, resource limits

What was NOT changed

No Python code was modified. The existing code already references configs/safety.yaml (plural) consistently — the only gap was the missing file itself.

Addresses

TODO.md: "Fix missing configs/safety.yaml and config/ vs configs/ path discrepancy" (P1)

Verification

• ruff format --check . — 392 files already formatted
• ruff check evoseal/ tests/ — All checks passed
• pytest — 604 passed, 23 deselected
• YAML validation — parses correctly with yaml.safe_load()

Summary by CodeRabbit

• New Features
  • Added an immutable safety configuration to enforce permitted edit scope, define regression detection thresholds (with automatic rollback on critical regressions), and run sandboxed tests with resource limits.
• Documentation
  • Updated the safety hardening tracking checklist and refreshed progress totals to reflect completion.
• Bug Fixes
  • Tightened edit-scope validation to prevent changes under the configs/ directory.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 8a88e180-f88a-4d00-9fbf-601a20067da0

📥 Commits

Reviewing files that changed from the base of the PR and between cec4ed7 and d27df11.

📒 Files selected for processing (3)

• TODO.md
• configs/safety.yaml
• evoseal/core/edit_scope_validator.py

🚧 Files skipped from review as they are similar to previous changes (2)

• evoseal/core/edit_scope_validator.py
• configs/safety.yaml

---

📝 Walkthrough

Walkthrough

Adds configs/safety.yaml with edit-scope, regression, rollback, and sandbox controls. Protects the configs/ directory from evolution edits and updates TODO.md to record completion and revised tracking totals.

Changes

Safety configuration

Layer / File(s)	Summary
Safety policy configuration configs/safety.yaml	Defines edit-scope restrictions, regression thresholds, checkpoint retention and rollback behavior, and sandbox resource limits for test subprocesses.
Configuration directory protection evoseal/core/edit_scope_validator.py	Rejects edits targeting configs/ and its descendants while preserving existing allowed patterns.
Safety item tracking update TODO.md	Marks the configuration task complete and updates the P1 and total completion counts.

Estimated code review effort: 2 (Simple) | ~10 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: adding configs/safety.yaml with default safety settings.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/create-safety-yaml

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@configs/safety.yaml`:
- Around line 11-35: The edit-scope configuration in the safety YAML does not
match the schema consumed by SafetyIntegration. Update the edit_scope keys and
value types to use allowed_patterns as a mapping, and connect the enabled
setting to the top-level enforcement flags expected by SafetyIntegration,
preserving the existing forbidden paths and directories. Ensure the resulting
immutable configuration is read instead of triggering validator defaults.
- Around line 42-62: Expand the metric_thresholds configuration used by
RegressionDetector to include the built-in thresholds for every
default-monitored metric, especially accuracy and error_rate. Preserve the
existing thresholds for duration_sec, memory_mb, execution_time, success_rate,
pass_rate, and correctness, and ensure the supplied map is complete rather than
omitting default entries.

In `@TODO.md`:
- Around line 308-311: Update the Total row in the tracking table to include all
four columns, adding the missing Notes cell, and change its completed count from
46 to 48 to match the completed values in the rows above.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 86a4e781-4224-4297-b1ef-2a99adb8490b

📥 Commits

Reviewing files that changed from the base of the PR and between 2f68f82 and 343d683.

📒 Files selected for processing (2)

• TODO.md
• configs/safety.yaml

[coderabbitai]

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Align the edit-scope schema with its consumer.

SafetyIntegration reads edit_scope.allowed_patterns as a mapping, while this file defines allowed_dirs as a list; enabled is also not connected to the top-level enforcement flags used by SafetyIntegration. As written, these settings are inert and the validator falls back to hard-coded defaults instead of using this immutable configuration.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@configs/safety.yaml` around lines 11 - 35, The edit-scope configuration in
the safety YAML does not match the schema consumed by SafetyIntegration. Update
the edit_scope keys and value types to use allowed_patterns as a mapping, and
connect the enabled setting to the top-level enforcement flags expected by
SafetyIntegration, preserving the existing forbidden paths and directories.
Ensure the resulting immutable configuration is read instead of triggering
validator defaults.

[coderabbitai]

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Preserve thresholds for all monitored metrics.

RegressionDetector uses the supplied metric_thresholds map wholesale rather than merging it with defaults. This omits built-in thresholds for metrics such as accuracy and error_rate, both of which are in the detector’s default monitored set, so they lose their intended per-metric regression and critical limits. Include the complete map or merge configuration with the defaults.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@configs/safety.yaml` around lines 42 - 62, Expand the metric_thresholds
configuration used by RegressionDetector to include the built-in thresholds for
every default-monitored metric, especially accuracy and error_rate. Preserve the
existing thresholds for duration_sec, memory_mb, execution_time, success_rate,
pass_rate, and correctness, and ensure the supplied map is complete rather than
omitting default entries.

[SHA888]

Addressed review feedback. Here's the verdict on each point:

✅ Fixed — missing from :
Added configs to EditScopeValidator._forbidden_dirs in edit_scope_validator.py (the actual enforcement code) and mirrored it in safety.yaml for documentation consistency. Now the entire configs/ directory is immutable, not just configs/safety.yaml. All 604 tests pass.

⏭️ Already handled — sandbox.readonly_files lists .env:
The _apply_readonly_files method (testrunner.py:753-775) already handles missing files gracefully: if not file_path.exists(): continue. .env is gitignored and won't exist in the repo; the entry is a safety net for deployments where it does exist. No fix needed.

⏭️ Invalid — regression.metric_thresholds missing critical fallback:
Verified that RegressionDetector.__init__ receives the regression: block from the YAML as its config dict, so metric_thresholds comes through with all entries intact — each having both regression and critical keys. The fallback critical_threshold = thresholds.get('critical', regression_threshold * 2) at line 383 is never reached for any listed metric. No fix needed.

⏭️ Minor — TODO.md commit note unverifiable from diff:
The claim 'Code already references configs/safety.yaml (plural) consistently' is accurate — verified via grep -rn across the codebase. The diff doesn't include code changes because none were needed. Wording could be clearer but the substance is correct; left as-is.

[SHA888]

Addressed review feedback. Heres my assessment of each point:

1. configs/ vs config/ directory split — accepted. Added a header comment in configs/safety.yaml explaining the intentional split: config/ is a Python package (has __init__.py, settings.py, mutable YAML/JSON), while configs/ holds standalone immutable YAML files referenced by path. Consolidation would require changing 15+ path references across docs, ADRs, and code — not worth the churn for a PR whose scope is "create the missing file."

2. edit_scope_validator.py directory matching — already correct, no change needed. The matching logic at line 151 uses rel_path = file_path_obj.relative_to(repo_root) and then checks rel_path_str.startswith(forbidden_dir + "/"). Since rel_path_str is already anchored to repo root, configs only matches top-level configs/ — a hypothetical evoseal/configs/ would produce evoseal/configs/foo which does not start with configs/.

3. Redundant forbidden_paths entry — accepted. Removed configs/safety.yaml from forbidden_paths since forbidden_dirs already contains configs, which covers everything under that directory. All 101 safety tests still pass.

4. TODO.md verification claim — accepted. Updated the resolution note to include specific verification: check_safety_yaml() returns well-formed, pytest tests/safety/ tests/unit/test_doctor_command.py passes (54 tests), and the doctor check passes. The tests at test_edit_scope_allowlist.py:41, test_sandboxed_test_execution.py:90, and test_adversarial_edits.py:171 do exercise the specific code paths cited in the original TODO item.

All checks green: ruff, full pytest (604 passed), detect-secrets.

[SHA888]

Review feedback — verified against code

Point 1: config/ in allowed_dirs lets the loop raise its own budget

This concern does not apply. I traced the actual budget enforcement path:

• evoseal/core/evolution_pipeline.py reads budget limits from self._settings.budget (line 991), which is a BudgetConfig Pydantic model loaded by config/settings.py.
• config/settings.py loads from config/settings.{ENV}.json via JsonConfigSettingsSource — not from config/budget.yaml.
• config/budget.yaml is never read by the runtime at all. The only code that touches it is doctor.py:check_budget_config(), which reads from .evoseal/metrics/budget_snapshot.json (a generated snapshot), not from config/budget.yaml.

So config/budget.yaml is effectively a documentation/reference file. The evolution loop cannot raise its own budget by editing it, because the budget enforcement system never reads it. No change needed here.

---

Point 2: Duplicate forbidden dirs — Python hardcoded set vs YAML

Partially valid but pre-existing. The reviewer is right that there are now two lists that could drift, but this is not a regression from this PR:

• Before this PR, EditScopeValidator already hardcoded its own _forbidden_dirs set with 7 entries. The YAML file did not exist.
• This PR adds "configs" to that set (to match the new YAML) and creates configs/safety.yaml as documentation/reference.
• The YAML is not loaded by the validator at runtime — it is a declarative spec, not the runtime source of truth. The Python class has always been self-contained.

The reviewer's suggestion to load from YAML as single source of truth is a valid architectural improvement, but it is a feature request (changing how the validator loads config), not a bug introduced by this PR. The current state is: the validator's hardcoded defaults match the YAML. Drift is possible in the future, but was already possible before this PR (between the validator's defaults and whatever the intended spec was).

No change needed for this PR — but worth filing as a follow-up task if the reviewer wants to pursue it.

---

Point 3: Default-deny for ambiguous paths (root files like pyproject.toml)

Already handled. The validator implements default-deny correctly in validate_edit_path():

is_allowed = False
for allowed_pattern in self._allowed_patterns:
    if rel_path_str.startswith(allowed_pattern):
        is_allowed = True
        break

if not is_allowed:
    raise EditScopeError(...)

is_allowed starts as False and is only set to True if the path matches an allowed pattern. Root files like pyproject.toml, requirements.txt, etc. do not match any *_dir/ pattern, so they fall through to the error. Makefile is also explicitly in _forbidden_paths. This is correct default-deny behavior — the caveat the reviewer raised is already covered.

---

Summary

Point	Verdict	Action
1. config/ budget escape	Incorrect premise — budget.yaml is not read at runtime	None
2. Duplicate forbidden dirs	Valid concern, but pre-existing, not a regression from this PR	Follow-up task (optional)
3. Default-deny for ambiguous paths	Already handled correctly	None

No code changes needed for this PR.