fix(evaluator): derived --config, region fallback, base-id regex, llm-context - #2027
Merged
Conversation
…egex, llm-context Follow-up fixes from the derived bug bash: - Reject --config with --type derived instead of silently ignoring it (the derived arm ran before the config loader). - Resolve the base metric's region via ConfigIO.resolveRegionFallback so a profile-only region works on a fresh project with no saved deploy targets (the old path mapped over an empty targets array and dropped the fallback). - Tighten BASE_EVALUATOR_ID_PATTERN to Builtin.<Metric> / ThirdParty.<Provider>.<Metric>; rejects malformed ids (ThirdParty.DeepEval, ThirdParty..ToolUse, trailing dot) that --level previously let through. - Sync llm-compacted schema so generated project guidance includes the derived arm.
resolveRegionFallback already checks env vars first and returns undefined rather than throwing (it's called bare in resolveAWSDeploymentTargets), so the catch that re-read the env vars was dead.
aidandaly24
approved these changes
Aug 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up fixes from the derived-evaluators bug bash (P2/P3), stacked on
Dervived_evaluators.Fixes
--configfor derived —--type derived --config x.jsondemanded--base-evaluator-id, and explicit derived flags + a nonexistent--configsilently ignored the file. Cause: the derived arm ran before the config loader. Now--configis rejected for--type derivedwith a clear message (use--base-evaluator-id+--model).aws-targets.json) when the region came only from the AWS profile.resolveAWSDeploymentTargets()maps over an empty array and drops the fallback. Now usesConfigIO.resolveRegionFallback()(made public) which returns the scalar env→profile fallback.BASE_EVALUATOR_ID_PATTERNaccepted malformed ids (ThirdParty.DeepEval,ThirdParty..ToolUse,ThirdParty.DeepEval.) which--levelthen persisted. Tightened toBuiltin.<Metric>/ThirdParty.<Provider>.<Metric>..llm-context—llm-compactedschema still described config asllmAsAJudge | codeBased; synced in thederivedarm so generated project guidance matches runtime Zod.Verification
agentcore-control.ts:546SDK-model error unrelated).--type derived --config→ rejected;ThirdParty.DeepEval(+--level) → rejected; valid derived still writes correct config.