Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -220,6 +220,15 @@ and unsupported claims. Catalog counts are historical observations—not proof
that every listed route is live—and quantified portfolio claims require a
reproducible artifact before they can be presented as verified results.

## Portfolio and consulting

The [portfolio proof matrix](docs/portfolio/PORTFOLIO_PROOF_MATRIX.md) maps the
project’s audience-specific stories to reproducible evidence and states the
limits of each claim. For teams operating multiple AI gateways, the [AI Gateway
Assurance Audit](docs/portfolio/AI_GATEWAY_ASSURANCE_AUDIT.md) describes a
scoped review of route identity, capability qualification, policy/failover,
and privacy-safe evidence.

---

## Project Structure
Expand Down
95 changes: 95 additions & 0 deletions docs/portfolio/AI_GATEWAY_ASSURANCE_AUDIT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
# AI Gateway Assurance Audit

## A practical review of what an AI route can do, may do, and can prove

Verdict is built for teams that have multiple gateways, providers, model
aliases, tools, and fallback paths but cannot confidently answer whether a
route is actually eligible for a task. The AI Gateway Assurance Audit turns
that uncertainty into a scoped engineering report.

This is an engineering review of a configured route set. It is not a promise
that a provider is available, that a model is intelligent, or that a system is
production-ready.

## What the audit covers

1. **Inventory and identity** — enumerate configured gateways, providers,
connections, endpoints, protocols, model revisions, aliases, and fallback
edges without collapsing distinct executable routes.
2. **Capability qualification** — review claimed metadata separately from
direct observations; run only consented, bounded protocol, schema, and tool
checks against approved test routes.
3. **Policy and failover behavior** — inspect hard capability predicates,
freshness, quota/cooldown handling, fallback legality, and whether advisory
ranking can reintroduce an excluded candidate.
4. **Evidence and privacy** — verify that decisions, execution outcomes, and
qualification reports are scoped, redacted, integrity-linked, and useful to
an operator without retaining prompts or response bodies.
5. **Remediation plan** — return prioritized gaps, an owner-ready backlog,
requalification triggers, and a release-evidence checklist.

## Inputs

- A route and provider inventory, or a read-only export of the configured
gateway catalog.
- The task capabilities and risk classes that must be hard requirements.
- Optional opt-in access to a test environment for bounded probes. Live calls
are never implied by catalog membership or HTTP success.
- Existing policy, CI, incident, and evidence artifacts that the customer
wants reviewed.

Credentials and raw prompts are not required for the offline contract review.
If live probes are authorized, credentials remain in the customer-controlled
environment and the public report contains only sanitized diagnostics and
digests.

## Deliverables

- A route identity and capability matrix showing `supported`, `unsupported`,
and `unknown` states with provenance and expiry.
- A redacted qualification report explaining why each requested capability is
admitted or rejected.
- A policy/failover findings register with severity, evidence reference,
reproduction step, remediation, and residual limitation.
- A release-readiness checklist that distinguishes local contract evidence,
observed provider evidence, and evidence that is still missing.
- A 30-minute handoff walkthrough for the engineering owner.

## Typical timeline

The offline review is designed as a focused one-week engineering engagement:

- **Day 1:** scope, threat model, route inventory, and success criteria.
- **Days 2–3:** contract, identity, policy, and evidence review.
- **Days 3–4:** optional consented qualification fixtures or bounded test-route
probes.
- **Day 5:** report, remediation backlog, and handoff.

The timeline is a planning target, not a service-level guarantee; provider
access, route count, and review scope determine the actual effort.

## Explicit exclusions

This audit does not certify provider uptime, model quality, security of a
provider’s infrastructure, regulatory compliance, production readiness,
adoption, performance leadership, or a specific business outcome. It does not
publish credentials, private endpoint details, raw prompts, response bodies,
or customer data. Any quantitative claim must have a dated, reproducible
artifact with a defined metric, baseline, environment, and raw-result boundary.

## Evidence boundary

The public Verdict repository demonstrates deterministic local contracts for
eligibility, passports, protocol qualification, structured output, tools,
receipts, and credential-free quickstart behavior. Those contracts are not a
substitute for customer-specific live evidence. The [public proof
matrix](../proof/EVIDENCE_INDEX.md) and [claims ledger](../proof/claims_ledger.v1.json)
show what is verified, observed, partial, or explicitly not approved.

## Contact and next step

Open a scoped request in the repository’s [GitHub issue
tracker](https://github.com/mrnicholasbcarter-code/verdict-core/issues) with
the route count, required capabilities, environment constraints, and desired
deadline. Do not include credentials, raw prompts, customer data, or private
endpoint URLs in the issue.
37 changes: 37 additions & 0 deletions docs/portfolio/PORTFOLIO_PROOF_MATRIX.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# Portfolio proof matrix

This is the recruiter, hiring-manager, and client-facing index for the
Verdict portfolio. It keeps the conversion story tied to evidence that can be
opened and reproduced.

| Audience | Proof asset | What it demonstrates | Boundary |
| --- | --- | --- | --- |
| Hiring manager | [Credential-free quickstart](../../README.md#run-the-credential-free-flagship-quickstart) | A deterministic local decision flow and explicit hard-gate exclusions | It is a fixture, not live provider execution |
| Platform engineer | [Capability passports](../CAPABILITY_PASSPORTS.md) and [qualification reports](../QUALIFICATION_REPORTS.md) | Exact route identity, provenance, expiry, redaction, and fail-closed admission | A report projects existing evidence; it does not create live evidence |
| Reliability/security reviewer | [Public evidence index](../proof/EVIDENCE_INDEX.md) | Traceability from claims to source, tests, schemas, and limitations | Release readiness remains partial until an exact tagged bundle exists |
| AI platform client | [AI Gateway Assurance Audit](AI_GATEWAY_ASSURANCE_AUDIT.md) | A concrete review scope, deliverables, inputs, exclusions, and handoff | Customer-specific provider findings require customer-authorized evidence |
| Systems/quant reviewer | [Kalshi case study](KALSHI_TRADING_BOTS_CASE_STUDY.md) | Architecture and risk-control narrative | Its quantitative figures are self-reported and not independently verified here |

## Approved positioning

> I build trustworthy decision systems under uncertainty—from AI execution
> assurance to capital and risk systems.

> Verdict tells you what an AI stack can actually do, what it is allowed to
> do, and what evidence proves what happened.

These lines describe the engineering thesis and the repository’s evidence
boundary. They do not claim adoption, production readiness, model quality, or
performance leadership.

## Before publishing a stronger claim

Run the proof validator and inspect the linked row in the claims ledger:

```bash
python scripts/verify_proof_matrix.py
```

Do not promote a claim from `unsupported`, `self_reported`, `aspiration`, or
`partial` until the ledger’s missing evidence is present, reproducible, dated,
and linked to the exact released artifact under discussion.
8 changes: 8 additions & 0 deletions docs/proof/EVIDENCE_INDEX.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,14 @@ what it deliberately refuses to claim.
| Adversarial checklist | [`ADVERSARIAL_REVIEW_CHECKLIST.md`](ADVERSARIAL_REVIEW_CHECKLIST.md) |
| Validator | `python scripts/verify_proof_matrix.py` |

## Conversion assets

- [Portfolio proof matrix](../portfolio/PORTFOLIO_PROOF_MATRIX.md) maps each
audience to an evidence-backed project story and preserves limitations.
- [AI Gateway Assurance Audit](../portfolio/AI_GATEWAY_ASSURANCE_AUDIT.md)
defines the scoped consulting offer, deliverables, exclusions, and safe
contact path.

## Verified local contracts

- Hard eligibility is applied before advisory ranking; excluded candidates
Expand Down