-
Notifications
You must be signed in to change notification settings - Fork 2k
Added runtime validation for Agent name field #998
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
…evelopers" appears twice in the classifiers list. Line Number 20 and Line Number 26 Fix: Removed the duplicate classifier entry
…he tracing module where one import uses an absolute path in line number 3 while all others use relative imports
Related to #996; if we add this, we may want to have validation logic for other arguments too @rm-openai any opinion? |
Thanks for the feedback! You're right that this could be part of a broader validation pattern. Looking at the Agent class, several other fields could benefit from similar runtime validation:
I'm happy if I can expand this PR to include other critical fields or use this as a foundation for incremental validation. What approach works best for the team? |
Agent class currently has 16 fields, so adding all of them may take some time. If you're fine with it, your contribution would be appreciated. Otherwise, we can later add validations to the rest of the fields after merging this PR only with name.
Simply writing all logic in
Checking if a value is not None and then check the type of the value would be enough. I don't think we should scan all the elements of an array and nested property values (i.e., no need to check internal values of |
I just noticed @rm-openai started refactoring of the class - #1044 ; aligning with this change and/or working on this PR after the refactoring is done would be appreciated 🙏 |
Ok! will work on this after refactoring 👍 |
@DanielHashmi The refactoring is done, and we've been receiving the same feedback from a few users. If you have time and interest to resume this PR work, it'd be appreciated! (Otherwise, I am happy to work on the rest 👋 ) |
Added 16 Validations & Some Critical Tests
I have added validations and some tests, Can you review it and tell me what should be my next step? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
Adds runtime validation to the Agent
class to catch invalid field types at instantiation and corresponding tests to verify those checks.
- Introduce a
__post_init__
method inAgent
that raisesTypeError
for non-stringname
and other invalid field types. - Add unit tests in
TestAgentValidation
to cover key validation scenarios. - Validate fields like
tool_use_behavior
,hooks
,tools
,handoffs
, andmodel_settings
.
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
File | Description |
---|---|
tests/test_agent_config.py | Add TestAgentValidation to test various runtime field validations |
src/agents/agent.py | Implement __post_init__ with type checks on multiple Agent fields |
Comments suppressed due to low confidence (2)
tests/test_agent_config.py:174
- Several new validations in post_init (e.g., for mcp_servers, mcp_config, instructions, prompt, input_guardrails, output_guardrails, output_type, reset_tool_choice) lack corresponding tests. Consider adding tests to improve coverage and ensure these validations behave as expected.
class TestAgentValidation:
src/agents/agent.py:224
- If
AgentBase
defines its own post_init, overriding it here without callingsuper().__post_init__()
may skip base initialization logic. Consider invokingsuper().__post_init__()
at the start of this method.
def __post_init__(self):
from typing import get_origin | ||
|
||
if not isinstance(self.name, str): | ||
raise TypeError(f"Agent name must be a string, got {type(self.name).__name__}") | ||
|
||
if self.handoff_description is not None and not isinstance(self.handoff_description, str): | ||
raise TypeError( | ||
f"Agent handoff_description must be a string or None, " | ||
f"got {type(self.handoff_description).__name__}" | ||
) | ||
|
||
if not isinstance(self.tools, list): | ||
raise TypeError(f"Agent tools must be a list, got {type(self.tools).__name__}") | ||
|
||
if not isinstance(self.mcp_servers, list): | ||
raise TypeError( | ||
f"Agent mcp_servers must be a list, got {type(self.mcp_servers).__name__}" | ||
) | ||
|
||
if not isinstance(self.mcp_config, dict): | ||
raise TypeError( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[nitpick] The post_init method contains many repetitive type checks; consider refactoring with a helper function or a mapping of field names to expected types to reduce duplication and improve readability.
from typing import get_origin | |
if not isinstance(self.name, str): | |
raise TypeError(f"Agent name must be a string, got {type(self.name).__name__}") | |
if self.handoff_description is not None and not isinstance(self.handoff_description, str): | |
raise TypeError( | |
f"Agent handoff_description must be a string or None, " | |
f"got {type(self.handoff_description).__name__}" | |
) | |
if not isinstance(self.tools, list): | |
raise TypeError(f"Agent tools must be a list, got {type(self.tools).__name__}") | |
if not isinstance(self.mcp_servers, list): | |
raise TypeError( | |
f"Agent mcp_servers must be a list, got {type(self.mcp_servers).__name__}" | |
) | |
if not isinstance(self.mcp_config, dict): | |
raise TypeError( | |
def validate_fields(field_mapping: dict[str, tuple[type, bool]]) -> None: | |
"""Helper function to validate fields based on a mapping of field names to types.""" | |
for field_name, (expected_type, allow_none) in field_mapping.items(): | |
value = getattr(self, field_name) | |
if value is None and allow_none: | |
continue | |
if not isinstance(value, expected_type): | |
raise TypeError( | |
f"Agent {field_name} must be a {expected_type.__name__}" | |
f"{' or None' if allow_none else ''}, got {type(value).__name__}" | |
) | |
field_mapping = { | |
"name": (str, False), | |
"handoff_description": (str, True), | |
"tools": (list, False), | |
"mcp_servers": (list, False), | |
"mcp_config": (dict, False), | |
} | |
validate_fields(field_mapping) | |
raise TypeError( |
Copilot uses AI. Check for mistakes.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great to me
@rm-openai we've been receiving the same feedback about the lack of validation of inti params for Agent almost every day. Can you take a look at this change and include it in the upcoming release?
Add
__post_init__
validation to ensure Agent name is a stringThe Agent class previously only used type hints for the name field without
runtime validation, allowing non-string values like integers to be passed.
This caused downstream errors during JSON serialization, tracing, and other
operations that expect the name to be a string.
Changes:
__post_init__
method to Agent class withisinstance
checkFixes issue where
Agent(name=1)
would succeed but cause errors laterin the execution pipeline.
Fixes #996