Skip to content

SPECS.md

Latest commit

 

History

History
721 lines (606 loc) · 36.1 KB

SPECS.md

File metadata and controls

721 lines (606 loc) · 36.1 KB
Raw

Ultrafuzz Specification

This document is the Ultrafuzz product specification. It describes the behavior the TypeScript implementation is expected to provide, independent of the workflow engine, process supervisor, UI framework, or package layout used internally.

The words MUST, MUST NOT, SHOULD, SHOULD NOT, and MAY are normative.

Sources And Scope

This repository is the normative source for this spec. When older Ultrafuzz documents or implementations disagree with this specification, this specification wins unless this repository intentionally changes the product contract.

Ultrafuzz is an agentic campaign orchestrator for Solidity smart contract fuzzing. It initializes a project with editable campaign inputs, validates those inputs before launch, compiles a workflow from product state, links that workflow to durable run evidence, and leaves generated changes reviewable until the operator explicitly materializes them.

Ultrafuzz is not a proof that a protocol is bug-free, an automatic vulnerability submission system, an automatic production-code fixer, or a system that should silently recover from missing graph, prompt, config, reference, or artifact state.

Product-Owned Surfaces

A fresh project initialization MUST create or preserve one root product file:

ultrafuzz.toml

All other editable or generated Ultrafuzz product surfaces MUST live under .ultrafuzz/:

.ultrafuzz/
  topology.yml
  prompts/
  references.yml
  runs/
  workspaces/
  cache/

Initialization MUST preserve existing config, topology, prompts, and reference catalog files unless the operator explicitly requests replacement. Runtime execution MUST treat .ultrafuzz/topology.yml and .ultrafuzz/prompts/** as canonical inputs.

Generated workflow-engine files MAY exist outside .ultrafuzz/, but they are implementation plumbing and are not a stable user API.

CLI Surface

The CLI product surface consists of:

Command Required behavior
init Create root config plus .ultrafuzz/** product surfaces and workflow plumbing.
validate Validate config, topology, prompts, references, path guards, agent references, and trust posture without launching agents.
json validate Validate one RFC 8259 JSON document against one local strict Draft 2020-12 schema without mutating either file.
run Validate, render prompts, build run evidence, compile and launch a linked workflow.
references status Report whether pinned references are present in the local digest-checked cache.
references sync Explicitly fetch pinned references into the local cache.
references update Rewrite the project reference catalog to newer pinned commits when requested.
ps List Ultrafuzz runs and linked workflow status.
inspect <run-id> Show product evidence and linked workflow details for a run.
resume <run-id> Delegate resume for the linked workflow after product checks.
replay <run-id> Delegate replay for the linked workflow after product checks.
fork <run-id> Delegate fork for the linked workflow after product checks.
report <run-id> Show the agent-written final report artifacts.
materialize <run-id> Copy selected reviewed outputs into the target project after confirmation and path checks.
clean <run-id> Remove selected generated run paths after confirmation and path checks.

Commands that support automation MUST emit ultrafuzz.cli.result.v2 with ok, public diagnostics, and command-discriminated data. Each current command payload MUST be closed and schema-validated before serialization. Unknown invocation failures MUST use ok: false and data: null. Explicit operator workflow input and redacted third-party tool input/output are the only deliberately opaque nested JSON values. Version-1 envelopes and compatibility conversion are unsupported.

run MUST use mutually exclusive --input-json and --input-file flags; it MUST NOT reinterpret malformed inline JSON as a path or retain the historical --input fallback. Both forms use the strict JSON parser, and file input uses one bounded immutable regular-file snapshot.

ultrafuzz json validate MUST remain separate from project-wide ultrafuzz validate. Its canonical invocation is:

ultrafuzz json validate --schema <schema.json> --file <artifact.json>

Exit 0 means portable whole-document shape validation succeeded. Exit 1 means the producer-owned instance is missing, unreadable, invalid UTF-8/JSON, contains duplicate keys, or violates the schema. Exit 2 means schema, reference, invocation, resource, or tool setup failed. The CLI and host MUST use the same strict, offline, worker-bounded parser and Ajv configuration and MUST NOT coerce, default, remove, normalize, repair, or rewrite either file. Exit 0 does not replace named host semantic/context gates.

Configuration

ultrafuzz.toml is the operator-controlled runtime settings surface. It MUST define typed project/run settings, model profiles, permission posture, invariant defaults, and triage defaults. Strategy execution behavior MUST be selected in topology, not in TOML.

A compatible config MUST support:

  • schema_version
  • dynamic_strategies_enumerator
  • [project] repo
  • [run] output_dir, max_parallel_agents, max_dynamic_nodes, keep_workspaces, workspace_mode, default_timeout_seconds, workflow_deadline_seconds, and controller_lease_seconds
  • [models] default plus [models.<id>] agent, model, and timeout_seconds
  • [retry] same_agent_attempts plus an optional ordered agents list of model profile IDs
  • [permissions] trust_model, prompt_review_required, and materialize_outputs_as_unstaged
  • [invariants] property_priority_threshold, invariant_testing_smoke_timeout, and invariant_testing_fuzzer_timeout
  • [triage] quorum and panel_size

run.workspace_mode MUST be git-worktree. Other workspace modes are outside the product contract.

Unknown TOML keys MUST fail validation. Model profile IDs and agent references MUST use safe identifiers. Omitted model selection in topology MUST resolve to the configured default model profile only; model fan-out MUST be explicit in a node or group default.

Retry profile IDs MUST resolve through [models.<id>] without parsing the ID. Fallback MUST be disabled by default. Automatic retries MUST use the original prompt in a fresh session, MUST NOT classify error text, and MUST exhaust the bounded primary attempt count before trying ordered fallbacks. Node and group max_attempts overrides MUST take precedence over the project primary count.

Config resolution SHOULD apply built-in defaults, project TOML, supported environment overrides, and runtime overrides in deterministic order. The resolved config MUST be persisted for each run. Persisted config intended for display or replay SHOULD redact secret-looking values. Redaction placeholder launch guards MAY be implemented; when implemented they MUST fail before launch rather than running with literal placeholders.

Topology

Topology MUST live at .ultrafuzz/topology.yml. The topology version specified here is 2. Version 1 is unsupported.

Topology is the campaign graph source of truth. It defines logical node IDs, dependencies, groups, group defaults, node overrides, versioned output contracts, prompt bindings, reference bindings, loop behavior, timeout overrides, and explicit model-profile fan-out.

version: 2
defaults:
  strategy_loops: 1
groups:
  strategies:
    label: Strategies
    color: "#7c3aed"
    defaults:
      loops: 3
      model_profiles:
        - default
nodes:
  - id: __start__
    kind: meta
    role: start
    depends_on: []
  - id: boundary-tests
    kind: agentic
    prompt: strategies/boundary-tests.md
    group: strategies
    depends_on:
      - setup
    outputs:
      - path: findings.json
        contract: ultrafuzz/findings@2
        primary: true
  - id: __finish__
    kind: meta
    role: finish
    depends_on:
      - final-report

Top-level fields:

  • version MUST be 2.
  • defaults.strategy_loops is the global fallback loop count.
  • groups MAY define labels, colors, and defaults.
  • nodes MUST be an ordered list of logical nodes.

Group defaults MAY include loops, timeout_seconds, max_attempts, and model_profiles. Node fields override group defaults. The default scaffold SHOULD use three loops for normal strategy nodes through the strategies group and explicit loops: 1 for exception flows such as stateful invariant and differential groups.

Every node MUST define id and depends_on. Agentic nodes SHOULD define prompt and group, and every executable node MUST define outputs. Each output MUST name a resolvable, versioned contract, and exactly one output MUST be primary. Meta nodes MUST use kind: meta with role: start or role: finish. Reference nodes MUST use kind: reference with a catalog reference ID and required reference artifacts.

Validation MUST reject unsupported versions, missing nodes, duplicate IDs, unsafe IDs, invalid groups, invalid colors, malformed meta/reference nodes, unknown dependencies, duplicate dependencies, cycles, zero loops, expanded graph size over implementation limits, missing or duplicate output paths, unresolved contracts, missing or duplicate primary outputs, unknown topology fields, invalid prompt paths, missing prompt files when prompts are required, invalid model profile IDs, and prompt artifact references to unknown or non-ancestor producers. The runtime-owned artifact-manifest.json path MUST NOT be declared as a node output.

Loop expansion MUST be deterministic:

  • loops: 1 expands to concrete ID id.
  • loops: N expands to id-0 through id-(N-1).
  • loop_mode: parallel makes every attempt depend on expanded dependencies.
  • loop_mode: series chains attempt i after attempt i - 1.

Concrete IDs MUST NOT collide. Expanded graphs SHOULD preserve graph version, topology version, groups, logical ID, concrete ID, label, kind, dependencies, artifact directory, loop metadata, contracted outputs, primary output marker, timeout, reference revision, and model fan-out provenance.

Every planned JSON output MUST resolve through the checked-in schema registry. The planned and expanded graph representations MUST persist the schema filename, fragment-free schema ID, schema SHA-256, package schema-bundle SHA-256, and validator build identity. Missing or partial bindings MUST fail planning or host verification. Operators declare the versioned contract in topology; they MUST NOT supply these trust identities manually in YAML.

Prompts

Prompts MUST live under .ultrafuzz/prompts/** and SHOULD be Markdown or MDX treated as Markdown-compatible prompt text. Runtime execution MUST prefer project-owned prompt copies.

Prompt frontmatter MAY include only identity/display metadata:

---
id: boundary-tests
display_name: Boundary Tests
---

id is execution identity. display_name is a label and MUST NOT change execution identity by itself. Unknown frontmatter fields MUST fail validation. Prompt frontmatter MUST NOT own execution knobs such as loops, enabled state, model profiles, or timeouts.

Prompt rendering MUST happen before workflow launch and the rendered prompt MUST be stored as a node artifact. Unknown template variables MUST fail validation.

The prompt variable set includes:

  • repo_path
  • workspace_path
  • schema_path
  • artifact_path
  • artifact_dir
  • run_metadata_path
  • output_findings_path
  • output_patch_path
  • strategy
  • attempt_index
  • strategy_loop_index
  • strategy_loop_count
  • triage_quorum
  • triage_panel_size
  • dynamic_strategies_enumerator
  • invariant_property_priority_threshold
  • invariant_property_priority_filter
  • invariant_property_priorities
  • invariant_testing_smoke_timeout
  • invariant_testing_fuzzer_timeout
  • strategy_attempt_test_dir
  • artifact_path:<logical-node-id>
  • artifact_handoff:<logical-node-id>
  • ancestor_contract_artifact_authority:<contract>
  • ancestor_artifact_path_authority:<path>[,<path>...]

Runtime-generated prompts MAY additionally use scalar item.* variables and namespaced replacement keys supplied by the selected source item. Their values MAY recursively reference other item-scoped variables but MUST NOT override built-in runtime variables.

Artifact handoff variables MUST resolve only to ancestor nodes. Handoff producers MUST declare a primary contracted output. Exact artifact paths MUST resolve to declared producer outputs. The legacy collection helpers ancestor_artifacts, ancestor_artifacts_by_path, ancestor_generated_test_manifests, and ancestor_generated_test_manifest_authorities MUST fail prompt validation with an actionable compact-authority migration. Workflow-node prompts MUST NOT expand an ancestor collection into producer paths or authority rows.

ancestor_contract_artifact_authority:<contract> MUST accept only a registered artifact contract ID and MUST select matching outputs only from the current task's transitive artifact-ancestor closure. The renderer MUST emit only a bounded pointer to the task-local JSON document at <task-workspace>/.ultrafuzz/authorities/<attemptId>.json, the exact current attempt ID, and the requested contract. It MUST NOT expand matching producer paths, model-fanout attempts, or source-authority rows into the prompt. The renderer MUST still record the exact matching logical ancestor IDs and contract in prompt artifact references so topology validation and contextual semantic gates share the same selector.

Immediately before every model attempt, the runtime MUST reparse and validate the sealed execution snapshot's controls/tasks.json itself. That shared control file and its parent directory MUST NOT be admitted to the agent. The current task's dependencyArtifactDirs is the declared closure and optionalDependencyArtifactDirs is its exact optional subset. Matching producers outside that optional subset MUST remain admitted. A producer inside the optional subset MUST be admitted only when its exact runtime verification marker exists; a markerless optional producer MUST be excluded. Every admitted agentic producer MUST cross the complete verifier boundary before the authority is derived.

The derived ultrafuzz.prompt-artifact-authority.v1 document MUST contain only schema_version, run_id, attempt_id, the absolute relocated run root as artifact_path_base, the current prompt's canonical selectors, and matching producers. Each producer contains only its attempt ID, logical node ID, canonical artifacts/<attemptId> directory, and selected output path and contract declarations. Controller-host paths, workspaces, source identities, model metadata, unrelated tasks, and unselected outputs MUST be omitted. An empty producers array is the authoritative no-match representation. Every relative path MUST reject absolute prefixes and traversal before it is resolved beneath artifact_path_base.

The deterministic serialized authority MUST be rejected when it exceeds 32 MiB, before the runtime creates or writes the task-local sidecar.

The runtime MUST restore the deterministic authority bytes before a retry and compare the exact file bytes after every model call, including failed calls and schema-correction calls. Missing, malformed, replaced, linked, or modified authority files MUST fail verification.

ancestor_artifact_path_authority:<path>[,<path>...] MUST accept one or more distinct safe declared output paths and MUST apply the same transitive closure, runtime verification-marker admission, per-task JSON projection, bounded rendering, portable-path ordering, and prompt artifact-reference requirements as ancestor_contract_artifact_authority. It MUST select admitted producer output declarations only when their exact declared path is in the requested set. It MUST record the exact matching logical ancestor IDs, a deterministic SHA-256 selector ID, and the canonically ordered requested output paths in the run plan. The sealed task manifest and task-local authority MUST preserve that path group and MUST reject an ID that does not match its paths. Prompt prose MUST name only the fixed-size selector ID; it MUST NOT enumerate the group's paths, matching producer paths, model-fanout attempts, or source-authority rows.

Both compact authority selectors MUST select only planned nodes with kind: agentic. If a selector's exact contract or path filter matches any transitive kind: reference ancestor, the renderer MUST fail closed, including when the same selector also matches an agentic ancestor. The diagnostic MUST identify the reference node and direct fixed reference consumers to artifact_path:<logical-node-id> or artifact_handoff:<logical-node-id>. A selector with no matching ancestor MUST remain valid and MUST be represented by an empty logicalIds array in the run plan and an empty producers array at runtime.

For every agent-authored JSON output, the centrally rendered output contract MUST include both safely shell-quoted commands using the exact resolved paths and declared contract:

Validation command: `ultrafuzz json validate --schema '<absolute schema path>' --file '<absolute artifact path>'`
Contract validation command: `ultrafuzz artifact validate '<contract-id>' '<absolute artifact path>'`

For ultrafuzz/generated-tests@3, it MUST also render a task-context command whose --run-id, --logical-node-id, and --artifact-root values come from the sealed task authority:

Task-context validation command: `ultrafuzz artifact validate 'ultrafuzz/generated-tests@3' '<absolute artifact path>' --run-id '<run-id>' --logical-node-id '<logical-node-id>' --artifact-root '<absolute task artifact root>'`

The shared prompt MUST require the producer to write the canonical document, run every command after its final write and before returning, correct and rerun an exit-1 draft during that same session, rerun after any later change, never edit the supplied schema, treat exit 2 as a setup failure, and finish only after every command exits 0. It MUST say that ordinary contract validation covers document-local semantics, generated-test task-context validation also checks companion files plus the sealed run and logical producer, validation is non-mutating, and host semantic/context gates still run afterward. No validation receipt or message-schema extension is required.

A deterministic workflow verification task MUST validate every agent output before downstream tasks become eligible. Artifact manifests MUST record output contract identities and digests plus the exact prerequisite manifest digests consumed by the attempt. Runtime failure evidence MUST distinguish agent, provider, artifact-contract, and dependency-cascade failures.

The terminal structured report MUST satisfy ultrafuzz/report@3 before scoring or publication. Invalid terminal output MUST persist a typed non-publishable state without persisting raw output or diagnostics.

References

Reference nodes are a product concept. .ultrafuzz/references.yml pins external reference material to full commits and safe relative paths. Normal runs SHOULD use only the local digest-checked cache. Fetching or updating references MUST be an explicit operator action.

Reference materialization MUST write durable artifacts under the reference node artifact directory, including normalized Markdown and references/manifest.json when declared by topology. A kind: vulnerability-database reference MUST instead preserve the validated external database tree byte-for-byte, discover record paths only from its pinned catalog.json, and retain repository, commit, schema, aggregate, and per-file digests. Missing cache entries, digest mismatches, unsafe paths, unknown reference IDs, malformed database contracts, or missing required reference artifacts MUST fail before dependent agentic nodes run. Ultrafuzz MUST NOT execute code from the external reference while validating it.

Execution And Lifecycle

Runtime MUST validate product state, render prompts, create run evidence, compile workflow tasks, launch a linked workflow, and keep the linked workflow identity in run metadata. The workflow engine is an implementation choice.

Before model work, a schema-backed producer MUST receive a host-managed Ultrafuzz launcher ahead of target-controlled PATH entries. Local runs MUST bind the launcher to a content-addressed, read-only closure containing the planned CLI bytes and every transitive package, plus the validator build and schema-bundle digest in run-owned metadata. The launcher MUST verify that closure before dispatch, clear ambient Node loader/search injection, and reject every non-builtin module whose lexical or physical resolution escapes the closure. Ordinary artifact and schema data reads remain outside this module boundary. Modal MUST provide the equivalent root-owned, read-only entrypoint. Both environments MUST run a real known-valid fixture and verify the returned schema ID, schema digest, bundle digest, and validator build; command -v alone is insufficient. A missing, tampered, or stale launcher or closure is a setup failure for new model work. It MUST NOT turn historical seals or schema identities into resume authorization. A current-controller continuation MAY select current validator packages while retaining historical source and artifacts as provenance.

Before or at launch, each run MUST persist:

  • run.json
  • source-run.json when the run derives from another run
  • config.resolved.toml
  • config.redactions.json
  • graph.json
  • graph.fingerprint
  • state.json
  • events.jsonl
  • attempts.jsonl
  • plan.json
  • trusted-cli.json, a run-owned trusted launcher, and content-addressed trusted CLI closures for schema-backed producers
  • immutable rendered prompt snapshots under prompt-snapshots/
  • per-node artifacts under artifacts/
  • review artifacts under review/
  • event query indexes under events.index/
  • workspace metadata under workspaces/

Reporting is agentic and lives in final-report artifacts.

Run and node state MUST be explicit. Node statuses MUST include pending, ready, runnable, running, succeeded, failed, skipped, timed-out, reused-from-prior-run, and invalidated. Run statuses MUST include pending, running, paused, succeeded, failed, timed-out, and canceled.

Every nonterminal node MUST persist when its current wait began, a typed wait reason, and the next action that can make it eligible. Run state MUST persist the workflow deadline, last state transition, renewable controller-lease status, requested and effective concurrency, queue depth, active work, and queued, active, and idle durations. Lost-controller recovery MUST use an atomic takeover claim and MUST NOT repeat completed work. Workflow deadlines and recovery decisions MUST be testable with a fake clock.

Current run state MUST use schema version ultrafuzz.run-state.v5 and schema ID urn:ultrafuzz:schema:artifacts:run-state:5. Older persisted state and graph versions MAY fail strict inspection, replay, or rendering, but MUST NOT prevent ordinary same-ID Smithers continuation. The runtime MUST NOT coerce historical state into the current contract as a condition of resume.

Run provenance MUST contain the complete sealed workflow binding. Node provenance, when present, MUST match exactly one closed variant: execution, pinned reference, or dependency blocker. Execution provenance MUST use the canonical output_contracts field and complete agent/verifier task identities; generic JSON values, required_artifacts, partial identities, and workflow-state aliases MUST be rejected rather than normalized.

Completed node attempts MUST be appended immutably to attempts.jsonl with the exact Smithers identity (workflow_run_id, source_event_sequence). Strategy-attempt, control-generation, node, iteration, attempt, and reuse-source coordinates MUST remain validated evidence fields and MUST NOT become surrogate ledger identities. Entries MUST preserve lifecycle timestamps, typed outcomes, reuse source coordinates, and input and output manifest digests. Attempt counts and terminal summaries MUST derive from the ledger. Failure categories MUST remain separate from raw diagnostics, and ledger entries MUST NOT persist raw inputs, outputs, or configuration.

The CLI MUST derive per-node timing, token components, estimated cost, model, attempt/retry disposition, and completeness from existing run evidence without requiring a precomputed statistics artifact. The same projection MUST operate offline only from the registered current v3 report-bundle manifest containing current state.json, graph.json, graph.fingerprint, and run.json documents. Present attempt and usage ledgers MUST pass the same strict UTF-8, duplicate-key, schema, run-binding, identity, and history readers as local run evidence; malformed, duplicate, conflicting, or cross-run rows MUST fail the query. Local evidence MUST be accepted only after two byte-identical reads of the complete evidence set, with bounded retries for recognized mutation races; the snapshot timestamp MUST be taken immediately after the accepted second read. A capture timestamp MUST NOT precede any historical observation carried by the snapshot or be later than the statistics clock. A genuinely absent optional ledger MUST remain unavailable rather than being represented as zero. When usage evidence is absent from a bundle, unauthenticated metadata accounting MUST also remain unavailable; a local run that retains accounting but loses its usage ledger MUST fail closed. The CLI result MUST use the exact closed ultrafuzz.stats.v1 data contract inside ultrafuzz.cli.result.v2.

status, pause, replay, and fork operate on strict linked workflow evidence. Ordinary resume MUST delegate the persisted workflow and same run ID to Smithers with workflow-change acceptance, without requiring control seals, link journals, controller generations, current schema bindings, graph identity, or metadata projections. It MUST NOT automatically reset, replay, timetravel, fork, migrate, or rewrite completed task artifacts. --refresh-controller MUST render current controller source without making retained provenance an authorization protocol.

Artifacts, Findings, And Reports

Node artifacts are the durable message-passing and evidence system. Required artifacts MUST match topology declarations. Artifact paths MUST be safe, project-local relative paths under the node artifact directory.

Artifact manifests MUST use ultrafuzz.artifact-manifest.v3 and record run, node, and producer identity; relative file paths, sizes, SHA-256 digests, and provenance; output contract IDs and digests; exact prerequisite manifest digests; and the primary marker. Every JSON output entry MUST also record the complete schema_file, schema_id, schema_sha256, schema_bundle_sha256, and validator_build binding. Non-JSON entries MUST omit those fields. Runtime-owned manifests and verification markers MUST be generated canonically as ultrafuzz.artifact-manifest.v3 and ultrafuzz.artifact-verification.v2 documents and strictly validated against their registered schemas. Manifest provenance metadata, when present, MUST be exactly either pinned-reference materialization metadata or Smithers task identity metadata; arbitrary JSON metadata bags are forbidden. The host MUST NOT upgrade an old manifest or marker in place.

Each retained agent-authored JSON handoff MUST have exactly one current named contract, one complete Draft 2020-12 whole-document schema, and one canonical version spelling. ultrafuzz/json-object@1, ultrafuzz/json-array@1, old contract IDs, aliases, coercion readers, and compatibility fallbacks MUST NOT be available. Where Zod remains useful for typed access, it MUST accept the same shape as JSON Schema without preprocessing, transforms, defaults, coercion, property stripping, or normalization. Constraints that JSON Schema cannot express MUST remain explicit named semantic/context gates.

After the producer returns, every declared agent-owned artifact byte MUST stay unchanged through host validation, synchronization, reporting, dashboard reads, publication, and bundling. Missing or invalid output MUST be a terminal post-agent failure when it prevents safe artifact admission. Designated missing optional metadata MUST instead produce bounded, attributable warnings; supplied conflicts, ambiguous identity, and missing execution-critical data remain errors. Host-owned diagnostic presentations and explicitly named companions MAY expose authenticated verifier warnings, including warnings about the final report itself, without replacing or rewriting the producer artifacts. The runtime MUST NOT invoke a correction turn or full-node model retry, synthesize an empty artifact or Markdown from a final response, normalize or convert fields, reseal changed bytes, rebuild from dependencies, or fall back to a sibling or historical artifact. Exact byte copying and explicitly runtime-owned sidecars remain permitted.

Event logs MUST redact secret-looking payload values before persistence. JSON/JSONL plus query indexes are sufficient; SQLite events are not required.

Findings MUST use ultrafuzz/findings@2 and be arrays in findings.json. Each finding MUST include exact schema_version literal ultrafuzz.finding.v2, producer-authored id, title, canonical status, and a non-empty summary, plus any stage-specific required decisions and evidence. Preliminary severity_guess and lowercase confidence MAY be omitted with validation warnings; supplied metadata MUST be preserved through review stages. Missing IDs, summary, evidence, or decisions MUST NOT be synthesized. Schema conformance alone MUST NOT be presented as proof of substantive audit completeness.

Finding status MUST be one of:

  • candidate
  • needs-review
  • duplicate
  • false-positive
  • confirmed
  • fixed
  • wont-fix

Other lifecycle strings and earlier schema-version spellings MUST be rejected.

When present, triage_classification MUST be one of:

  • true-positive
  • false-positive
  • undetermined
  • incomplete-spec
  • harness-defect
  • repair-candidate
  • spec-gated
  • defensive-hardening

Findings SHOULD preserve source node, strategy, attempt index, model profile, model name, model index, loop index, affected files/functions, evidence, patch references, notes, and dedupe or family metadata when available. evidence entries MAY be non-empty string references or objects with optional kind, path, and additional metadata. When present, object kind and path values MUST be non-empty strings, and relative evidence paths MUST remain safe artifact-relative paths without embedded selectors. A single source span MAY use positive integer line and end_line metadata. Disjoint spans MUST use at least two ordered line_ranges objects with a required positive integer line and an optional end_line that does not precede it. Independent explanatory detail MUST remain separate from structural range metadata.

Generated-test production is opt-in per topology node. A findings producer MAY omit ultrafuzz/generated-tests@3 when tests or proofs of concept are optional supporting evidence rather than a required artifact, and MAY instead declare the contract as an optional, empty-allowed evidence channel satisfied by the schema-defined empty bundle when no proof of concept was produced. Only a node that declares that contract receives generated-test bundle instructions, and downstream consumers MUST use contract-derived manifest intake instead of assuming that every strategy or findings producer emits generated-tests.json. Review stages MUST define a validation path for findings that arrive with an empty manifest or from a producer that declares no manifest, instead of treating either case as blocked.

Default review flows SHOULD deduplicate findings, classify severity, aggregate generated tests, and write final report artifacts. ultrafuzz report MUST read agent-written final-report artifacts such as:

artifacts/final-report/report.md
artifacts/final-report/report.json

Materialization And Cleanup

Materialization MUST be explicit, selected, confirmed, and path-safe. The materialization surface is copy-only. Patch artifacts MAY exist as evidence, but patch application MUST be rejected until a safe patch applier is implemented.

Materialized files SHOULD be left as ordinary unstaged working-tree changes. Repository mutation constraints such as no commit, push, pull request, external submission, staging, or merge are prompt-level trust assumptions and SHOULD be expressed in prompts. Ultrafuzz does not treat those constraints as deterministic repository-mutation enforcement.

Cleanup MUST remove only selected generated .ultrafuzz/** paths after confirmation. It MUST reject unsafe paths, symlink escapes, missing selections, and product state files that are not valid cleanup targets.

Trust And Safety

Agents run under a trusted local execution model. Ultrafuzz does not maintain a command allowlist, network allowlist, or sandbox approval policy as product configuration. The durable product boundary is reviewable prompts before launch, explicit references sync, durable artifacts, explicit materialization, and normal repository review tools.

Product file operations MUST reject traversal, absolute-path injection, unsafe relative paths, and symlink escapes. Run evidence SHOULD redact secret-looking values before persistence.

Dashboard And API Expectations

Dashboard/API is a product requirement. The target UX SHOULD match the original Ultrafuzz direction: a local operator UI for graph inspection, run status, evidence, prompt editing, topology editing, config review, report review, materialization, and cleanup.

Dashboard/API servers MUST bind to loopback by default. Mutating APIs MUST use local request protections and a cryptographically random session token. Path and run-ID inputs MUST use the same safe-path and safe-ID validation as the CLI.

The dashboard SHOULD show logical topology nodes by default. Expanded attempts and model fan-out MAY be shown in technical details, but edits SHOULD map back to logical project files. Editors for prompts, topology, and config MUST validate changes before accepting them and MUST NOT leave partial writes after a failed validation.

Dashboard command jobs MUST map only to supported Ultrafuzz product operations. Destructive or target-repo-mutating jobs such as clean and materialize MUST require confirmation.

Evaluation And Benchmarking Guidance

Evaluation SHOULD track true positives, underspecified but actionable findings, false positives, precision, recall, F1, cost, wall-clock time, token usage, model profile, strategy, loop count, attempt count, and cumulative unique valid findings across repeated campaigns.

False positives in evaluation SHOULD be treated as harness defects when the generated test fails for reasons that do not reflect production behavior.

Repeated campaign behavior SHOULD be evaluated like pass@k: a single campaign can be useful, but repeated nondeterministic campaigns may reveal different issue sets. Implementations SHOULD preserve enough provenance to measure which strategies, attempts, and models found each issue.

Adjudication SHOULD use independent reviewers or model judges under a fixed rubric. A quorum policy SHOULD determine whether a finding enters benchmark counts or reports, and disagreements SHOULD remain inspectable.