Skip to content

science(talent): bind intended policy regime to actual resolved execution provenance #403

Description

@seonghobae

Finding

Fresh review of Talent owner PR #293 at exact 47dca8e2a21fd3a89fc5f449773fdfbdb831c038 found a distinct false-authority gap after #401.

#401 makes the policy-regime identity owner-recomputable from a versioned canonical semantic projection and pins every material released external dependency by owner identity plus immutable version/digest. That closes the intended-procedure identity problem.

What remains undefined is whether a material stage can prove that the procedure actually executed resolved and used the same dependency set and material parameters that the governing policy regime declared.

ADR 0292 already records model/tool/version and evidence/policy versions in MaterialDecisionStageProvenance, but it does not require deterministic reconciliation between that observed execution provenance and the policy regime's canonical PolicySemanticProjection. Without that reconciliation, a manifest can carry a perfectly reproducible regime identity while the runtime actually used a stale cache, wrong model build, wrong Job/FJA/KSAO release, wrong assessment scoring contract, wrong required-evidence resolver, or another materially different dependency. The denominator and downstream result would then be scientifically bound to the declared procedure rather than the executed procedure.

This is separate from #401. #401 answers "what procedure does this regime mean?" This issue answers "did this particular stage occurrence actually run that procedure?"

Engineering / provenance evidence

Use the following as engineering evidence, not as HR validity authority:

  • SLSA v1.2 is the current approved specification. Its provenance model distinguishes declared parameters from resolvedDependencies, and Build Provenance describes resolved dependencies as the concrete artifacts actually fetched/resolved during execution. This is useful by analogy for distinguishing Orgmetra's intended semantic projection from observed runtime dependency resolution: https://slsa.dev/spec/v1.2/ and https://slsa.dev/spec/v1.2/build-provenance
  • NIST AI RMF Playbook MANAGE 3.1 recommends documenting and regularly monitoring third-party AI resources because external data/software/hardware/services can change system risk. That supports runtime dependency reconciliation when material Talent stages consume externally owned components: https://airc.nist.gov/airmf-resources/playbook/manage/

The repository-specific consequence is narrower: a high-impact Talent stage must bind its immutable stage provenance to owner-observed resolved material dependencies/parameters and prove they are congruent with the governing policy-semantic projection before its outcome can authorize downstream progression or high-impact finalization.

Owner boundary

talent_management owns only the execution-congruence evidence for its own material stage: governing policy-regime identity/projection, observed resolved dependency identities/versions/digests, material resolved parameters, execution/attempt identity, comparison outcome, and controlled mismatch reason.

It does not copy upstream Job/FJA/KSAO, assessment, model/tool, performance, Person/Employment/Assignment, protected demographic, or Workforce Validation source truth. Observed dependencies are released/versioned owner references plus exact immutable evidence.

workforce_validation remains authoritative for validity/fairness/adverse-impact, transportability, estimand/grouping, and scientific interpretation. Execution congruence proves which procedure ran; it does not prove that procedure is valid or fair.

Required contract refinement while ADR 0292 remains Proposed

Define an owner-side execution attestation / stage-resolution contract equivalent to the following. Exact UL may change, but these distinctions must survive:

  • every material stage occurrence binds the exact governing policy_regime_id, semantic-projection schema/version, and canonical material dependency set expected by that regime;
  • before material stage output can authorize advancement/exclusion/ranking/suppression/routing or high-impact finalization, the owner records the observed resolved identity/version/digest of every material external dependency actually consumed, plus every material runtime parameter not already fixed by the semantic projection;
  • observed execution evidence is generated or verified by the trusted Talent execution boundary, not accepted solely from caller-supplied metadata;
  • expected-versus-observed comparison is deterministic and versioned. Missing, extra, ambiguous, mutable/floating, stale, or digest-mismatched material dependencies fail closed or resolve to an explicit non-authorizing unavailable/error state;
  • stage provenance distinguishes execution_congruent, execution_incongruent, and controlled unavailable/not-verifiable states rather than silently recording only the intended regime identity;
  • retry/transport attempts may have separate attempt telemetry, but only the canonical material attempt that produced the terminal stage effect is bound to the stage outcome. A retry that resolves a different dependency set cannot silently inherit the first attempt's congruence verdict;
  • caches, gateways, model routers, feature/qualification resolvers, assessment scorers, and equivalent indirection layers must expose the final resolved immutable artifact/contract identity when they can materially affect stage behavior. A configured alias alone is insufficient;
  • if a dependency materially changes between policy projection creation and execution, the stage cannot claim the old regime merely because the configured policy bytes or alias are unchanged;
  • every released manifest/downstream result binds the exact stage execution-congruence evidence needed to distinguish intended-procedure provenance from actual-execution provenance;
  • historical evidence remains immutable. Discovering an execution mismatch after release creates correction/reconsideration and new result evidence where applicable; it never rewrites the original stage into a different procedure retroactively.

RED acceptance cases

Before ADR acceptance or production implementation, fail at least when:

  1. the policy regime expects model/tool/artifact digest A but the stage actually resolves B and still advances;
  2. a configured mutable alias such as latest, model group, cache key, or floating owner label is stored without the immutable artifact/version/digest actually resolved at execution;
  3. expected dependency A is missing at runtime and a fallback B is used without making the stage non-congruent and non-authorizing;
  4. runtime consumes an extra material dependency or material parameter absent from the governing semantic projection and still claims congruence;
  5. caller-supplied resolved_dependency_digest is accepted without owner-side observation/verification;
  6. stale cache content materially differs from the regime-pinned owner release while provenance records only the expected release;
  7. retry attempt 1 resolves A, retry attempt 2 resolves B, but terminal outcome inherits attempt 1's congruence evidence;
  8. a model/router/tool gateway records only the requested alias/provider/group while the final immutable model/tool artifact identity used is unavailable;
  9. required-evidence/FJA/KSAO/assessment resolver returns a different released contract version from the one pinned in the regime and the stage still authorizes progression;
  10. execution-congruence evidence is absent, ambiguous, unverifiable, or digest-mismatched but downstream manifest/result is released as though the declared procedure definitely ran;
  11. a discovered post-release runtime mismatch mutates the historical stage/result in place instead of append-only correction/reconsideration/new evidence;
  12. workforce_validation receives only the intended policy-regime identity and cannot distinguish declared procedure from the resolved procedure actually used for the stage.

Completion boundary

Documentation reconciliation alone is not completion. Keep this issue open through ADR/TRACEABILITY/doctoring reconciliation, executable domain/API/schema contracts, RED→GREEN tests for expected-versus-observed dependency reconciliation, trustworthy runtime resolution evidence, downstream workforce_validation consumption, and normal protected integration.

Do not close it merely because MaterialDecisionStageProvenance stores a model/tool/version field or because #401 stores a dependency-closed policy digest.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions