feat(plugins): add the scaled-evals plugin for large-scale agent evaluation

[arpitsardhana]

Summary

Adds plugins/nemo-scaled-evals, a control plane for running container-isolated agent
evaluations at scale. It serves a REST surface under /apis/scaled-evals/v1 that takes an
uploaded task pack, builds it into a container image, runs a harness against that image in an
ephemeral sandbox, and returns a score alongside artifacts, a provenance manifest, and an SBOM.
The change is additive: the plugin owns its own Postgres and object storage, declares no
platform service dependencies, and is not listed in enabled-plugins, so an existing
deployment behaves exactly as before until someone installs it explicitly.

Changes

• Plugin surface (src/nemo_scaled_evals_plugin/): a NemoService that mounts 11
  vendored routers under /apis/scaled-evals, applies migrations on startup with bounded
  retries so a database that is not yet accepting connections self-heals, stamps platform
  authorization rules onto every route, and redacts request bodies out of validation errors.
• Control plane (src/scaled_evals/api/): tasks, task revisions, benchmarks, benchmark
  runs, evaluations, credentials, configuration profiles, users, teams, agent bundles, and an
  admin surface. Includes the repository layer, Fernet-encrypted credential storage with key
  rotation, object-store access for S3-compatible and Google Cloud backends, and secret
  redaction helpers.
• Build worker (src/scaled_evals/api/build/): converts an uploaded task pack into an
  image through one of four backends — BuildKit, a managed cloud builder, an external image
  builder service, or a pre-built reference — then validates the resulting image identity and
  digest against a registry allowlist.
• Dispatch worker (src/scaled_evals/dispatch/): a Postgres-queued worker using
  FOR UPDATE SKIP LOCKED that claims evaluations, materializes caller-supplied credentials
  into the sandbox environment, launches and monitors the run, collects artifacts, and tears
  the sandbox down. Ships one built-in runtime backend for Kubernetes sandboxes; Gym-based
  runtimes register through an opt-in plugin hook.
• Harness patches (harbor-patches/): a small overlay applied to the harness inside its
  own virtual environment at image build time, so the runtime behavior this control plane
  depends on does not require changes upstream.
• Result publishing (src/scaled_evals/intake/): converts harness trajectories into ATIF
  and posts them to the platform's Intake service, creating one experiment per run.
• Schema (db/): the baseline schema plus ordered, re-appliable migrations. Applied
  automatically on startup, or by an external job when that is turned off.
• CLI (src/scaled_evals/cli/): a thin HTTP client over the same REST API for creating
  tasks, credentials, profiles, and evaluations, and for following logs, events, artifacts,
  and provenance.
• Deployment (deploy/): a Docker Compose stack for local use, and a Kustomize example
  for Kubernetes, each with an end-to-end smoke script.
• Examples and tests (examples/, tests/): one example task pack and 18 plugin tests.
• Repository wiring: adds the plugin to the workspace in the root pyproject.toml,
  refreshes uv.lock, and lists the plugin in plugins/README.md.

Workflows supported

• Bring your own task. Upload a task pack, finalize it, and the build worker produces and
  verifies a container image for it.
• Run an evaluation. Point an evaluation at a task revision and a runtime; the dispatch
  worker runs the harness in an isolated sandbox and records the score.
• Bring your own credentials. Provider keys are supplied per evaluation, stored encrypted,
  and injected only into that run's environment.
• Group runs into benchmarks. Benchmarks and benchmark runs fan a single request across
  many tasks and aggregate the results.
• Collect evidence. Every terminal run emits artifacts, a provenance manifest, and a
  CycloneDX SBOM, retrievable individually or as one archive.
• Publish results. Finished runs can be pushed to the platform's Intake service as ATIF
  trajectories for later analysis.
• Reuse configuration. Named configuration profiles hold harness, routing, and result
  publishing settings so an evaluation request stays small.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with documentation updates
• Documentation only
• Contributor tooling or automation
• CI, build, or test infrastructure

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Documentation updated for user-visible behavior
• Documentation not applicable — justification: the plugin is not in enabled-plugins
  and has no user-facing entry point in the published docs yet; setup, configuration, and
  both deployment paths are documented in plugins/nemo-scaled-evals/README.md and the
  per-directory READMEs that ship with the change.

Known limitations

These are deliberate for a first landing and are called out so review can focus on them.

1. Authorization is authenticated-but-permissionless. Routes are stamped so they mount,
   but no identity provider is wired, so every caller resolves to one principal and the
   deployment is effectively single-tenant. Mutating endpoints do not scope by owner, and the
   admin router is not gated while authentication is disabled. This must not be exposed to
   untrusted or multi-tenant traffic in its current form.
2. Thin test coverage. 18 tests cover the plugin seams — migrations, license headers,
   the harness overlay layout, validation redaction. The upstream project's much larger suite
   is not vendored yet, so the control plane's own logic is largely covered by the end-to-end
   smokes rather than by unit tests.
3. The default runtime cannot be built from PyPI alone. The sandbox library the built-in
   Kubernetes runtime uses is not published on PyPI, so building the image requires pointing
   HARBOR_EXTRA_INDEX_URL at an index that carries it. Without that value the build fails
   rather than silently producing a broken image.
4. The Kubernetes example is cloud-coupled. It assumes Google Kubernetes Engine, Cloud
   Build, Artifact Registry, Cloud Storage, and Workload Identity. The Compose path has no
   such dependency, but a portable Kubernetes deployment is still outstanding.
5. It owns its own storage rather than the platform's. The plugin reads
   SCALED_EVALS_DATABASE_URL and never the platform's database configuration, and it talks
   to object storage directly instead of through the Files service. Postgres is required — the
   claim queues rely on FOR UPDATE SKIP LOCKED, JSONB, advisory locks, and enum types, so
   the platform's SQLite default is not a usable target. When the database is unreachable the
   plugin degrades to a 503 on /v1/readyz instead of failing platform boot.
6. Cold-start ordering is unguarded on Kubernetes. The dispatch worker starts polling
   before the API has finished applying migrations, so on a brand-new database it restarts a
   couple of times before settling. Compose gates this with a health-conditioned dependency;
   the Kubernetes manifests rely on restart backoff instead.
7. Result transcripts are not content-gated. ATIF payloads include full conversation
   transcripts, and the configuration flag intended to gate that is not wired up, so prompts
   and responses reach Intake unredacted.
8. Provider base URLs are overridden for some providers. Dispatch injects a default
   upstream inference endpoint for certain credential providers, which overrides caller
   configuration. This needs to become opt-in before the path is generally useful.
9. Inert surfaces. The model-routing lease/publish router is intentionally not mounted, so
   the matching CLI command cannot reach an endpoint, and its cloud-builder client is the one
   code-scanning finding below that is unreachable rather than fixed. Deleting that surface is
   a follow-up rather than part of this landing. One Gym runtime variant also pins a source
   revision that is not resolvable from the public repository.
10. Request bodies are loose. Most request models do not forbid unknown fields, so a
    misspelled key is dropped silently rather than rejected.

Verification

• Pull request title follows the repository's Conventional Commit format
• Every commit includes an appropriate Signed-off-by: trailer
• uv run pre-commit run -a passes, or any blocked checks are identified below
• Targeted tests pass, or tests are marked not applicable above
• No secrets, API keys, or credentials are included

Targeted validation:

Repository gate

uv run pre-commit run -a

Run after rebasing this branch onto the current main. Every content hook passes —
ruff, ruff format, ty typechecks, config-reference, copyright headers, uv.lock drift,
the plugin import boundary, toolchain-version checks, and Flox locks. Two hooks fail for
local-toolchain reasons rather than on content, and both need confirmation from CI:

• Run uv lock with platform uv — the hook requires uv 0.9.14 to regenerate the lock and
  the local toolchain is uv 0.9.15. The separate Check for uv.lock drift hook passes, so
  uv.lock is consistent with pyproject.toml; this is a version guard, not drift.
• Helm Docs — helm-docs is not installed locally. This change touches no chart.

Two hooks did modify files, and both results are folded into the commits below:

• ruff format rewrote 72 files, because the vendored sources were wrapped for a narrower
  line length than this repository allows. Reflow only; the plugin tests pass on both sides
  of it.
• Fix copyright headers stamped 8 files once this branch picked up main's version of the
  hook, which covers file types the plugin had not stamped.

tools/lint/lint-all.sh was also run in full. Everything that executes passes; the
failures left are the same local-toolchain gaps — four lints shell out through Flox, which
is not installed here, plus the two pre-commit hooks above.

Tests, lint, and types

uv run --frozen --group scaled-evals pytest plugins/nemo-scaled-evals/tests -q
                                                       ->  17 passed, 1 skipped
uv run --frozen ruff check                             ->  All checks passed
uv run --frozen ruff format --check                    ->  3451 files already formatted
tools/lint/lint-python-types.sh                        ->  All checks passed
actionlint .github/workflows/ci.yaml                   ->  no findings
kubectl kustomize plugins/nemo-scaled-evals/deploy/k8s ->  renders 21 objects

This plugin is intentionally absent from enabled-plugins, so the default sync CI's lint and
test entry points use leaves it and psycopg uninstalled; that is why the type gate initially
reported 57 unresolved imports and pytest could not collect its tests. A non-default
scaled-evals dependency group now carries it, named only where coverage is wanted: the type
lint asks for it directly, and the plugin's tests get a dedicated job. Adding it to the shared
unit-test run is not viable — the platform then discovers the service and blocks on a Postgres
that is not there, which slowed that suite roughly sixfold and broke timing-sensitive daemon
tests — so the plugin's conftest.py skips its own directory when the group is absent.

The type gate needed real fixes, not suppression. Each repository class defines a list
method, which shadows the builtin for annotations evaluated in the same class body, so
list[...] on 32 signatures referred to that method rather than to the type;
from __future__ import annotations is why it never raised at runtime. Those now say
builtins.list[...], and a runtime check confirms every one resolves. The harness overlay
imports the sandbox library it replaces, which by design exists only inside the image it is
injected into, so it joins the existing exclusions for runtime-only imports. Two remaining
diagnostics were a settings singleton whose required fields come from the environment, and a
CLI summary that now tolerates a server answering with a non-list field instead of raising.

The plugin's own dependencies come in through --all-packages; a plain uv sync --frozen
leaves psycopg out and the tests then fail to import. The kustomize render is included
because the copyright hook added comment lines to two files that feed config maps — the
envs: generator drops comments, and the one mounted as a file is parsed by a reader that
already skips # lines.

The platform's unit-marked suite was also run (pytest -m unit) and is unaffected; this change
adds a new plugin directory and touches no existing package beyond the workspace list, the lock
file, and the plugin index. One unrelated service's tests fail to collect on this checkout for a
pre-existing reason and were excluded.

Static analysis

Code scanning raised two findings on this branch, and secret scanning raised ten. All are
addressed:

• Clear-text logging of sensitive information, in the registry auth refresh job. The value
  logged was the field name the docker config is stored under in the Secret, never a
  credential; the variable was simply called secret_key. It is now named for what it holds,
  and the log line reports the host and Secret rather than the field.
• Partial server-side request forgery, in the cloud builder client. The caller-supplied
  segment was already percent-encoded with safe='', and its only caller is the
  model-routing router that is deliberately not mounted, so the path is unreachable in a
  running deployment. Cloud Build ids are UUIDs, so the client now rejects anything else
  before it reaches a URL.
• Ten unverified Postgres connection strings, in two Kubernetes manifests and two test
  modules. None pointed at a reachable host, but the shape was real. The deployments now
  take the password from PGPASSWORD, which libpq reads directly, so no manifest carries a
  credential in a connection string; the test fixtures assemble their throwaway DSNs from
  parts. Re-running the scanner over this branch reports none of them.

Compose stack, end to end

Built the image from source in this tree and ran the bundled smoke, which creates a task,
uploads a pack, finalizes it, builds through BuildKit, and pushes to the stack's registry:

docker compose -f plugins/nemo-scaled-evals/deploy/compose/docker-compose.yml build api
plugins/nemo-scaled-evals/deploy/compose/smoke.sh   ->  PASS

The plugin's health endpoint reported mode: scaled-evals-cp from the freshly built image,
confirming the running code matched this branch rather than a cached image.

Kubernetes, end to end from a clean slate

Deleted the target namespace so Postgres came back on a new volume, then redeployed. Startup
migrations built the schema from empty, and all four workloads rolled out on the same image:

deploy/k8s/apply.sh     ->  4/4 deployments rolled out
/v1/readyz              ->  status ok (postgres, schema, object_store, build_worker,
                            kubernetes-sandbox dispatch all ok)
deploy/k8s/smoke.sh     ->  PASS; image built by the managed cloud builder and the
                            registry digest matched
deploy/k8s/eval-smoke.sh -> 2 evaluations, both succeeded with reward 1.0
GET /v1/evaluations/{id}/archive -> HTTP 200 on both

The evaluation smoke runs the harness against the example task with an oracle agent that
applies the task's own reference solution, so a healthy run scores exactly 1.0 and needs no
model credentials. Both runs did. The script itself reported a failure on one invocation
because two concurrent runs contended for the same local port-forward and it treats a dropped
tunnel as fatal; the runs themselves completed server-side.

Not verified

• Multi-tenant behavior, since no identity provider is wired (limitation 1).
• Gym-based runtimes, which are opt-in and disabled by default.
• Any deployment target other than Compose and Google Kubernetes Engine.
• The two deployment smokes above ran before this branch was brought up to the current
  main. The delta since then is main's own commits, copyright headers, the
  type-annotation and naming fixes, and the PGPASSWORD change to the manifests. That last
  one is the only change to deployment behavior, it applies to Kubernetes and not to Compose,
  and it has not been re-run against a cluster. kubectl kustomize renders it, and libpq
  reads PGPASSWORD when the connection string omits a password, but treat the Kubernetes
  path as needing one more smoke before it is trusted.

[github-actions]

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	35422/45214	78.3%	62.8%
Integration Tests	21114/42989	49.1%	22.3%

[asutermo]

I would check to see if you're missing any changes from gitlab (e.g. the direct harbor dataset reference snippets). Otherwise, looking good

[asutermo]

not sure if the alembic change in gitlab is useful here or not. potentially could combine all of these migrations (unless you're going to point to the original DB in phase1)

[asutermo]

delete file

[asutermo]

delete file

[asutermo]

actually I just propose deleting this task in favor of a hello-world one