UPDATES#1
Merged
Merged
Conversation
- timeline._filter_by_date: add parentheses so the conditional binds correctly and blank timestamps no longer pass the 'before' filter - extract_file: write exact icat bytes via a new capture_bytes path in safe_subprocess (lossy UTF-8 decode previously corrupted binary files and invalidated the extracted-file SHA256); refuse truncated artifacts rather than emit an inaccurate hash; fix unclosed file handle - analyze_memory: invoke the 'python3 -m volatility3' fallback as (binary, args) instead of a single bogus binary name under shell=False - add 6 regression tests (all 52 pass) https://claude.ai/code/session_01NK4kTPZokGafDyEdhacQ73
- safe_subprocess now stamps every execution with a unique, sortable execution_id (EXEC-<session>-<n>), recorded on both the tool response and the matching tool-execution.jsonl audit line - thread execution_id through all tool response envelopes (23 sites) - citation format now carries exec:<execution_id> so any finding traces to the exact tool call and its output_sha256 - document the logs/cost-ledger.jsonl schema (per-phase tokens + wall clock + tool-call counts) the orchestrator writes at phase boundaries, satisfying the 'logs with timestamps and token usage' requirement - add 3 traceability tests (57 pass) https://claude.ai/code/session_01NK4kTPZokGafDyEdhacQ73
- eval/run_eval.py: stdlib-only scorer that emits ACCURACY.md + accuracy.json (the rubric's required accuracy report). Recall over required ground-truth items; precision/hallucination over ASSERTED claims only (CONFIRMED verdict / tier 1-2 / HIGH) so Tier-3 inferences are reported but never penalized — honoring the 'confirmed vs inferred' distinction - ground_truth/nist-hacking-case.json: documented Mr. Evil facts (uncertain values flagged verify/required:false to avoid skewing the score) - ground_truth/memory-sample.json: fillable template for a public memory image - examples/ + tests/test_eval.py: fully-known fixture, 6 deterministic tests - eval/README.md: scoring model + dataset-acquisition docs - verified end-to-end: example scores P=66.67% R=75% F1=70.59% (60 tests pass) https://claude.ai/code/session_01NK4kTPZokGafDyEdhacQ73
Hallucination-management centerpiece. After self-correction, every finding
gets an independent verdict by re-deriving its claim from a FRESH tool call
(grounded, never by re-reasoning — avoids the documented self-critique
'coherence trap'):
- protocols/verification.md: chain-of-verification with a per-artifact-type
re-derivation table; UNVERIFIED asserted claims are downgraded + flagged,
never silently dropped; CONFIRMED requires a recorded verifier_exec_id
- finding-template.json: adds verification{verdict,verifier_tool,
verifier_exec_id,verifier_output_sha256} + provenance_exec_id; citations
now carry exec:<id>
- investigation-report.md: Verdict column + Verification Ledger section +
verifier lineage in the Audit Trail
- wired into orchestrator Phase 5 (full) and lean mode (CRITICAL/HIGH);
eval harness already scores by verdict so inferences are never penalized
https://claude.ai/code/session_01NK4kTPZokGafDyEdhacQ73
Implements the contest's named 'persistent learning loop' criterion: - protocols/persistent-loop.md: iterates on the same evidence until verifiable success criteria (grounded_ratio==1.0, zero open HIGH issues, kill-chain evidenced-or-marked) or stagnation; always terminates via --max-iterations; targeted course-correction toward UNVERIFIED claims / open issues / missing kill-chain phases (next_focus recorded each pass) - logs/progress.jsonl: per-iteration signals (grounded_ratio, unverified, open_high_issues, kill_chain_coverage, cost, and F1 when --truth given) - proves the delta: iteration-1 -> final F1 (or grounded_ratio) with full traces preserved (synthesis.vN.json + progress.jsonl + exec-id'd audit log) - regression guard: never ships a result worse than a prior iteration - reuses --iterate archiving, self-correction, verification, eval/run_eval.py - registered --loop mode in orchestrator + SKILL invocation https://claude.ai/code/session_01NK4kTPZokGafDyEdhacQ73
Detect timeline/memory/logs/persistence contradictions immediately after Tier-1 completes (reusing artifact-correlation contradiction patterns) and resolve by targeted tool re-invocation before Tier-2 consumes the artifacts; unresolved contradictions are passed forward as signals (both findings downgraded) rather than silently averaged. Records analysis/contradiction-pass.json. https://claude.ai/code/session_01NK4kTPZokGafDyEdhacQ73
New identity-plane analysis for intrusions that never touch a host disk — a distinctive differentiator vs host-only submissions: - mcp-server/tools/cloud.py: analyze_cloud_logs (read-only, in-process). Parses Entra ID sign-in/audit, Azure activity, and M365 UAL (JSON/NDJSON/ CSV, auto-merging UAL AuditData). Detects impossible travel, MFA fatigue, risky/legacy sign-ins, password spray/brute force, illicit OAuth consent, service-principal secret adds, privileged role grants, malicious inbox rules (BEC), and mass download — each mapped to ATT&CK for Cloud - parsers/common.py: record_in_process_execution() so non-subprocess tools also get an execution_id + audit line (same traceability guarantees) - registered analyze_cloud_logs in server.py (Tool + dispatch) - protocols/techniques/cloud-log-analysis.md + routing/evidence wiring (cloud_log type, Tier-1 dispatch, cloud selection rule) - eval/datasets + ground_truth synthetic sample (fully known) - tests/test_cloud.py: 9 tests incl. perfect-score eval integration (69 pass) https://claude.ai/code/session_01NK4kTPZokGafDyEdhacQ73
- .github/workflows/tests.yml: pytest on Python 3.10/3.11/3.12 + eval self-test - requirements.txt: pinned runtime/test deps (mcp bounded <2.0) - Makefile: test / eval / install / clean targets - pyproject.toml: fix invalid build-backend (was 'setuptools.backends._legacy:_Backend' -> 'setuptools.build_meta', so pip install -e . works) + register pytest testpaths/asyncio_mode - tests/test_server.py: real handle_tool_call dispatch + cloud end-to-end + evidence write-protection guardrail (importorskip mcp for resilience) - tests/test_hooks.py: executes pre-tool-use.sh and asserts evidence-write / destructive-command blocks vs benign allows - full suite: 81 pass with mcp installed (75 + server skip without it) https://claude.ai/code/session_01NK4kTPZokGafDyEdhacQ73
- README: verification verdicts, persistent loop, cloud breadth, eval harness, frontier-model (Opus 4.8) note; corrected test count (81); realigned judging matrix to the actual Stage-2 rubric (3 equal axes + hallucination/loop/guardrail/audit callouts); architectural-vs-prompt guardrails explicitly separated - architecture-diagram: add Verification + Persistent Loop layers, cloud.py module + cloud-log technique, all three recognized execution patterns (single/multi/loop); fix stale counts (73 binaries, 12 tools, 9 protocols); note exec_id makes quality-guardrail compliance independently checkable https://claude.ai/code/session_01NK4kTPZokGafDyEdhacQ73
Add verification verdicts, persistent learning loop, cloud (Entra/Azure/M365) breadth, and the quantified eval harness to differentiators + 'what it does'; 9 protocols, 81 tests + CI, Opus 4.8 model note; refresh roadmap (disk analysis already shipped) and Built With. https://claude.ai/code/session_01NK4kTPZokGafDyEdhacQ73
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.