possible issues in
"user_simulation": user_simulation,
over here
# be the {enabled, llm, system_prompt} shape — NOT our raw
# dataset shape — or .get("enabled") is False and no customer runs.
"user_simulation": user_simulation,
# the action/db/nl checks that define success (read by reward)
"evaluation_criteria": r.get("evaluation_criteria"),
# soft cap on turns; default 20 if the row didn't carry one
"max_turns_hint": (r.get("meta") or {}).get("max_turns_hint", 20),
},
In test banking.py
len() = 4 print(user_simulation) {'enabled': True, 'system_prompt': "You are playing the role of a customer contacting a customer service representative agent. Your character is a management consultant named Sarah Bosch who earns $100,000 annually. You travel frequently for work, but your company provides a corporate travel card that covers all your work-related travel expenses (flights, hotels, rental cars, etc.). \n\nYou're looking for a credit card to use for your everyday purchases. You want the card that gives you the highest cash back available in the company profile. You will not accept a credit card that has any annual fees unless it is the ONLY option available. \n\nIf you find out that there are no credit cards that fit your needs, you are happy to immediately take your business elsewhere and end the conversation.\n\nYou have access to a tool that allow you to apply for credit cards by specifying the card type. You're seeking advice on which personal credit card would be the best fit for your situation and spending patterns.\n\nYou receive a Rho-Bank+ subscription for free through your company. ONLY MENTION THIS if you are asked about this.\n\nAfter you receive enough information to make a decision, immediately apply for a credit card on your own, and there is no need to respond to the agent, or ask for instructions on how to apply.\n\nYou are currently on the line with a customer support agent. Only provide additional details about your situation when the agent asks for them. Don't dump all your information at once. Never respond as a customer service representative/assistant. You are playing the role of the customer.", 'llm': 'fireworks_ai/accounts/fireworks/models/gpt-oss-120b', 'llm_args': {'temperature': 0.0}}
Evaluation Criteria was this
print(r.get("evaluation_criteria")) {'actions': [{'name': 'apply_for_credit_card', 'arguments': {'card_type': 'Gold Rewards Card', 'customer_name': 'Sarah Bosch', 'annual_income': 100000, 'rho_bank_subscription': True}, 'requestor': 'user', 'action_id': '001_0'}], 'communicate_info': [], 'reward_basis': ['DB']}
(r.get("meta") or {}).get("max_turns_hint", 20) 34
What the heck is this max_turn_hint never used in Evals or baseline evaluation.
]The variable r holds this
1, in NameError: name 't' is not defined print(r) {'id': 'task_001', 'bucket': 'Medium', 'user_prompt_template': "You are playing the role of a customer contacting a customer service representative agent. Your character is a management consultant named Sarah Bosch who earns $100,000 annually. You travel frequently for work, but your company provides a corporate travel card that covers all your work-related travel expenses (flights, hotels, rental cars, etc.). \n\nYou're looking for a credit card to use for your everyday purchases. You want the card that gives you the highest cash back available in the company profile. You will not accept a credit card that has any annual fees unless it is the ONLY option available. \n\nIf you find out that there are no credit cards that fit your needs, you are happy to immediately take your business elsewhere and end the conversation.\n\nYou have access to a tool that allow you to apply for credit cards by specifying the card type. You're seeking advice on which personal credit card would be the best fit for your situation and spending patterns.\n\nYou receive a Rho-Bank+ subscription for free through your company. ONLY MENTION THIS if you are asked about this.\n\nAfter you receive enough information to make a decision, immediately apply for a credit card on your own, and there is no need to respond to the agent, or ask for instructions on how to apply.\n\nYou are currently on the line with a customer support agent. Only provide additional details about your situation when the agent asks for them. Don't dump all your information at once. Never respond as a customer service representative/assistant. You are playing the role of the customer.", 'user_persona': None, 'environment_context': {'domain': 'banking_knowledge', 'initial_state': None, 'required_documents': ['doc_credit_cards_gold_rewards_card_001', 'doc_credit_cards_silver_rewards_card_001', 'doc_credit_cards_bronze_rewards_card_001', 'doc_credit_cards_platinum_rewards_card_001'], 'user_tools': ['apply_for_credit_card']}, 'user_simulation': {'instructions': "You are playing the role of a customer contacting a customer service representative agent. Your character is a management consultant named Sarah Bosch who earns $100,000 annually. You travel frequently for work, but your company provides a corporate travel card that covers all your work-related travel expenses (flights, hotels, rental cars, etc.). \n\nYou're looking for a credit card to use for your everyday purchases. You want the card that gives you the highest cash back available in the company profile. You will not accept a credit card that has any annual fees unless it is the ONLY option available. \n\nIf you find out that there are no credit cards that fit your needs, you are happy to immediately take your business elsewhere and end the conversation.\n\nYou have access to a tool that allow you to apply for credit cards by specifying the card type. You're seeking advice on which personal credit card would be the best fit for your situation and spending patterns.\n\nYou receive a Rho-Bank+ subscription for free through your company. ONLY MENTION THIS if you are asked about this.\n\nAfter you receive enough information to make a decision, immediately apply for a credit card on your own, and there is no need to respond to the agent, or ask for instructions on how to apply.\n\nYou are currently on the line with a customer support agent. Only provide additional details about your situation when the agent asks for them. Don't dump all your information at once. Never respond as a customer service representative/assistant. You are playing the role of the customer.", 'recommended_user_model': 'fireworks_ai/accounts/fireworks/models/gpt-oss-120b', 'temperature': 0.0}, 'evaluation_criteria': {'actions': [{'name': 'apply_for_credit_card', 'arguments': {'card_type': 'Gold Rewards Card', 'customer_name': 'Sarah Bosch', 'annual_income': 100000, 'rho_bank_subscription': True}, 'requestor': 'user', 'action_id': '001_0'}], 'communicate_info': [], 'reward_basis': ['DB']}, 'meta': {'num_expected_actions': 1, 'avg_messages': '23.0', 'max_messages': '38', 'best_progress': '1.000', 'solve_fraction': '0.333', 'max_turns_hint': 34}}
Dataset adapter will run this function here
for p in ds_arg:
data_jsonl.extend(load_jsonl(p))
# Apply override for max rows if present
if max_dataset_rows is not None:
data_jsonl = data_jsonl[:max_dataset_rows]
data = dataset_adapter(data_jsonl)
This is in evaluation_test.py file inside eval_protocol directory of the library.
There is Eval metadata is pver here
print(eval_metadata)
name='test_banking' description="REWARD HOOK (step 3): score ONE finished rollout.\n\n EP calls this once per rolled-out row AFTER the trajectory is complete. The\n incoming row already has the full conversation (row.messages) and our\n input_metadata. We delegate ALL scoring to score_evaluation_row, which:\n 1. loads the real tau2 Task by row_id,\n 2. rebuilds a tau2 SimulationRun from row.messages,\n 3. runs tau2's evaluate_simulation -> reward_info (action_checks, db_check…),\n 4. applies the gated partial-credit math,\n 5. sets row.evaluation_result = EvaluateResult(score, reason) and returns row.\n\n Keeping the body a one-liner means there is ONE source of truth for the reward\n (the workspace/ module), reused identically by local eval and training.\n " version='voice-user-sim-v1.0.dev203+g775e159.dirty' status=Status(code=<Code.RUNNING: 101>, message='Evaluation is running', details=[]) num_runs=2 aggregation_method='mean' passed_threshold=EvaluationThreshold(success=0.001, standard_error=None) passed=None
This is what is happening
for row in data:
# generate a stable row_id for each row
if row.input_metadata.row_id is None:
# Generate a stable, deterministic row_id using the row's hash and num_combinations
index = hash(row)
max_index = num_combinations() - 1
# Ensure index is a non-negative integer within [0, max_index]
index = abs(index) % (max_index + 1)
row.input_metadata.row_id = generate_id(seed=0, index=index)
completion_params = kwargs["completion_params"] if "completion_params" in kwargs else None
# Create eval metadata with test function info and current commit hash
eval_metadata = EvalMetadata(
name=test_func.__name__,
description=test_func.__doc__,
status=Status.eval_running(),
num_runs=num_runs,
aggregation_method=aggregation_method,
passed_threshold=passed_threshold,
passed=None,
)
for row in data:
row.input_metadata.completion_params = (
completion_params if completion_params is not None else {}
)
# Add mode to session_data
if row.input_metadata.session_data is None:
row.input_metadata.session_data = {}
row.input_metadata.session_data["mode"] = mode
# Initialize eval_metadata for each row
row.eval_metadata = eval_metadata.model_copy(deep=True)
row.execution_metadata.experiment_id = experiment_id
row.execution_metadata.invocation_id = invocation_id
# has to be done in the pytest main process since it's
# used to determine whether this eval has stopped
row.pid = os.getpid()
The MCP Gym server is run over here
if MCPGymRolloutProcessor._shared_server is None:
raise RuntimeError(
"Shared MCP server not started. Call with server_mode='shared' and start_server=True first."
)
# Bind this instance to the shared server for this call
self.server = MCPGymRolloutProcessor._shared_server
else:
# Default "per_run" behavior: fresh server per call, reused only for retries
if start_server:
# Create fresh MCP server and environments for this run
if config.server_script_path is None:
raise ValueError("server_script_path is required for MCPGymRolloutProcessor")
self.server = MCPServerManager(config.server_script_path, port=port, **server_kwargs)
try:
self.server.start()
except Exception as e:
if self.server:
server.start does the task.
Authorization Headers Settings 1. KB_search KB_search
transfer_to_human_agents transfer_to_human_agents
get_current_time get_current_time
get_user_information_by_id get_user_information_by_id
get_user_information_by_name get_user_information_by_name
get_user_information_by_email get_user_information_by_email
change_user_email change_user_email
get_referrals_by_user get_referrals_by_user
get_credit_card_transactions_by_user get_credit_card_transactions_by_user
get_credit_card_accounts_by_user get_credit_card_accounts_by_user
log_verification log_verification
give_discoverable_user_tool give_discoverable_user_tool
unlock_discoverable_agent_tool unlock_discoverable_agent_tool
call_discoverable_agent_tool call_discoverable_agent_tool
list_discoverable_agent_tools list_discoverable_agent_tools
Lopoks like the MCP is all looking fine.