Unreleased refinements: correlation grouping, DMARC gating, catalog, doctor exit codes#13
Merged
Merged
Conversation
Add correlation-group tags to the aws_hosting and email_gateway_present nodes so redundant co-firing indicators count once, matching the existing Microsoft 365 and Google Workspace grouping. A target exposing several AWS facets (or gateway vendors) no longer receives an over-confident posterior with a too-tight credible interval; the inference baseline is regenerated to the calibrated, wider values. Gate DMARC enforcement on the pct rollout tag: a monitoring-only pct=0 record is effective "none" and a partial rollout steps the policy down one level, so a partial deployment is no longer scored as full enforcement (RFC 9989). Derive spf_strict from the merged service set, where strict SPF is actually recorded, instead of an SPF-typed evidence record the pipeline never produced, so the signal can fire in production. Harden the Bayesian loader: reject boolean calibration values (bool is an int subclass, so calibration: true was silently read as 1.0) and range-check a priors override supplied directly to infer(), keeping the committed prior when the value is outside the open interval (0, 1).
Recognize the Marketo mktoapps.com landing-page backend and the Edgecast/Edgio zetacdn.net CDN domain, added to their existing vendor entries during a 2026-07 passive fingerprint-mining round. Anchor dot-less cname_target fragments (such as Marketo's mkto-) to a DNS label or hyphen boundary so a host that merely embeds the fragment mid-label (for example promkto-analytics.attacker.net) is no longer mis-attributed to the vendor.
recon doctor printed a failure summary but always exited 0, so a CI or monitoring script could not tell a broken environment from a healthy one. Exit 1 when any core check fails while an all-pass run or an optional-enrichment-only degradation (for example a crt.sh outage) stays 0. recon doctor --mcp likewise exits 1 when the MCP server cannot be validated (package missing, import failure, or no tools registered) and stays 0 when the setup is healthy, still emitting the copy-paste config. Document the doctor exit semantics in the operational contract and pin them with tests, including a corrected coverage test that had asserted the old always-0 behavior.
The discover command wrapper had no CLI-level coverage, so a regression in its exit-code routing or output routing could ship unnoticed. Pin the contract: a JSON candidate list on stdout, --output to a file, a malformed domain rejected with exit 2, no data with exit 3, and an unexpected error with exit 4.
Log the unreleased Bayesian correlation-group correction, DMARC coverage gating, spf_strict fix, fingerprint catalog additions, CNAME boundary hardening, and the recon doctor exit-code behavior.
There was a problem hiding this comment.
Pull request overview
This PR bundles several unreleased detection-model refinements (Bayesian evidence grouping, DMARC enforcement gating, SPF strict signal derivation, stricter loader validation) together with fingerprint-catalog expansions and improved operator observability (notably recon doctor / recon doctor --mcp exit-code behavior and new CLI contract tests).
Changes:
- Calibrates inference by extending correlation grouping to
aws_hostingandemail_gateway_present, plus tightens loader validation (boolean calibration rejection; range-checked priors overrides). - Refines email-security signals: DMARC enforcement now respects
pct=rollout coverage;spf_strictnow derives from the merged services set (where the marker is actually produced). - Expands and hardens the surface fingerprint catalog (Marketo
mktoapps.com, Edgecast/Edgiozetacdn.net, and safer dot-lesscname_targetboundary matching), and makes doctor/discover CLI behavior more scriptable via exit codes + contract tests.
Reviewed changes
Copilot reviewed 19 out of 19 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| validation/inference_baseline.json | Updates baseline posteriors/interval widths to reflect new grouping calibration. |
| tests/test_surface_attribution.py | Adds regression tests for new/adjusted cname_target classification behavior (mktoapps/zetacdn and boundary rules). |
| tests/test_security.py | Updates SPF strict signal expectations to match “service-marker-only” derivation. |
| tests/test_doctor.py | Adds explicit exit-code contract tests for recon doctor. |
| tests/test_doctor_mcp.py | Adds exit-code tests for recon doctor --mcp failure modes. |
| tests/test_discover_cli.py | Adds CLI contract tests for recon discover output routing and exit codes. |
| tests/test_cli_coverage.py | Updates existing CLI coverage expectation for doctor’s non-zero exit on core failures. |
| tests/test_bayesian_unit_math.py | Updates unit tests for new DMARC pct gating and SPF strict service-marker behavior. |
| tests/test_bayesian_inference.py | Adds shipped-network regression guards for correlation grouping on AWS/gateway nodes; adds priors-override range-check test. |
| src/recon_tool/sources/dns_tables.py | Tightens dot-less cname_target matching to require label/hyphen boundary to reduce spoofing/misattribution. |
| src/recon_tool/data/fingerprints/surface.yaml | Adds Marketo mktoapps.com and Edgecast/Edgio zetacdn.net cname_target rules. |
| src/recon_tool/data/bayesian_network.yaml | Adds correlation group metadata to AWS and gateway vendor evidence bindings. |
| src/recon_tool/cli_doctor.py | Changes doctor/doctor-mcp to exit non-zero on core failures; refactors render to report failure state. |
| src/recon_tool/bayesian.py | Implements DMARC effective-enforcement gating via pct; derives spf_strict from services. |
| src/recon_tool/bayesian_loader.py | Rejects boolean calibration values; range-checks priors overrides on the direct-argument path. |
| docs/roadmap.md | Updates roadmap notes to reflect shipped DMARC pct gating and catalog additions. |
| docs/operational-contract.md | Documents the doctor exit-code contract for scripting/CI gating. |
| docs/correlation.md | Updates correlation reference to v2.2.x status and documents grouped nodes as shipped. |
| CHANGELOG.md | Records the refinements/fixes (grouping, catalog additions, doctor exit codes, DMARC pct gating, priors-override validation, dot-less spoofing fix). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+189
to
+190
| if not tools_ok: | ||
| raise typer.Exit(code=EXIT_ERROR) |
Pin two previously untested error paths: recon <domain> --compare with a missing export file exits 2 before any network work, and the Bayesian calibration loader rejects a boolean value (bool is an int subclass, so it would otherwise be coerced to 1.0).
An unknown --profile or an invalid --explain-dag-format now exits 2 (validation) instead of 4: the lookup command's generic handler was catching the deliberate typer.Exit and reclassifying it to the internal error code, also printing a bare "Exit" line. recon batch --ndjson no longer prints the "N duplicate(s) removed" notice into the JSON stream, and recon batch --md no longer echoes the internal NUL error sentinel into stdout on a resolve failure (the error goes to stderr like the default panel). Found in a maintenance bug-hunt of the lower-coverage command modules; each fix is pinned by a regression test in test_cli_output_contracts.py.
A non-UTF-8 batch file or stdin now exits 2 with a clear decode error instead of an uncaught UnicodeDecodeError that read as an internal bug, and recon discover --output to a directory or unwritable path now exits 2 with a clear write error instead of reaching the crash handler. --md is now rejected with --chain/--compare/--exposure/--gaps, matching the existing --plain handling, instead of being silently dropped. Completes the medium-severity findings from the maintenance bug-hunt; each is pinned by a regression test.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bundles the unreleased detection-model refinements with two operator-observability improvements. Each area is a separate commit; the full local gate (lint, strict types, the 82 percent branch-coverage floor, and every guard script) is green at 89.52 percent coverage.
Detection model
Fingerprint catalog
Operator observability