feat: Add EvaluationPlugin for agent invocation evaluation and retry

[afarntrog]

Description

Adds a new EvaluationPlugin that integrates with the Strands agents plugin system to evaluate agent output after each invocation and automatically retry with improved system prompts on failure.

Changes

New: strands_evals.plugins module

• EvaluationPlugin — A Strands Plugin that wraps agent invocations to:
  • Run configured evaluators against agent output after each call
  • On evaluation failure, use an LLM to generate an improved system prompt based on the failure reasons
  • Automatically retry the invocation with the improved prompt (configurable max_retries)
  • Restore the original system prompt after all attempts complete
  • Return a structured ImprovementSuggestion (reasoning + system prompt) from the improvement LLM
• improvement_suggestion.py — Prompt template for the improvement suggestion agent that analyzes evaluation failures and proposes system prompt modifications
• Expected output and trajectory can be set at construction time or overridden per-invocation via invocation_state

Other changes

• Bumped strands-agents minimum version from >=1.0.0 to >=1.28.0 (required for the plugin interface)
• Added ruff line-length exemption for plugins/prompt_templates/

Testing

• 480 lines of unit tests covering: initialization, agent wrapping, evaluator execution, retry behavior, system prompt restoration, improvement suggestion generation, partial failure handling, max retries, and multi-evaluator scenarios

---

Let me know if you want any tweaks to this.

Related Issues

Documentation PR

Type of Change

New feature

Testing

How have you tested the change? Verify that the changes do not break functionality or introduce warnings in consuming repositories: agents-docs, agents-tools, agents-cli

• I ran hatch run prepare

Checklist

• I have read the CONTRIBUTING document
• I have added any necessary tests that prove my fix is effective or my feature works
• I have updated the documentation accordingly
• I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
• My changes generate no new warnings
• Any dependent changes have been merged and published

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[afarntrog]

We need to do this monkey patching because hooks alone will not give us the retry pattern we need:

What EvaluationPlugin needs to do:

1. Intercept an agent invocation
2. Let it run
3. Evaluate the result
4. If evaluation fails, modify the system prompt and run the agent again
5. Repeat up to max_retries times

What hooks give you:

• BeforeInvocationEvent fires before the agent runs. You can modify messages. No result yet.
• AfterInvocationEvent fires after the agent runs. You get the result. But you're inside _run_loop's finally block, and stream_async still holds the invocation lock. If you call agent() from here → deadlock.

Hooks give you "before" and "after", not "around". There's no way to say "evaluate this result and, if it's bad, run the whole thing again" from within a hook. The retry requires calling agent(), which requires the lock, which is held by the invocation that's firing your hook. __class__ swap helps because:
This replaces __call__ — every call to agent(prompt) now goes through _invoke_with_evaluation, which runs evaluators and handles retries.

[afarntrog]

Although there can be/is some overlap with steering the primary difference between these are:

• Steering: hooks into the agent's inner loop (per model call, per tool call: AfterModelCallEvent)
• EvaluationPlugin: wraps the agent's outer loop (per full invocation: AfterInvocationEvent)

[afarntrog]

see strands-agents/sdk-python#1711