Problem
The agent intermittently sends confidence as a numeric probability (e.g., 0.9, "0.95") instead of the required enum string (LOW, MEDIUM, HIGH). When this happens, the validator in collect_ndjson_output.cjs rejects the entire message, causing valid tool calls (with correct rationale, issue_type, field_name, etc.) to be silently dropped.
Evidence
From a test run, ingestion step:
##[warning] Validation errors found:
- Line 1: set_issue_type 'confidence' must be one of: LOW, MEDIUM, HIGH
- Line 3: set_issue_field 'confidence' must be a string
- Line 4: set_issue_field 'confidence' must be a string
- Line 5: set_issue_field 'confidence' must be a string
- Line 6: set_issue_field 'confidence' must be a string
The agent sent "confidence": "0.95" and "confidence": 0.9 instead of "HIGH" / "MEDIUM".
Two-part fix
1. Strengthen the INTENT description suffix
Current (PR #42776):
INTENT: Include rationale (max 280 chars) and confidence (LOW/MEDIUM/HIGH) with each call.
Proposed:
INTENT: Include rationale (string, max 280 chars) and confidence (string, exactly one of: LOW, MEDIUM, HIGH) with each call.
File: actions/setup/js/generate_safe_outputs_tools.cjs (the hasRuntimeFeature(runtimeFeatures, "issue_intents") block)
2. Make validation lenient for optional intent fields
When confidence or rationale fails validation, strip the invalid field and pass the message through rather than rejecting the entire message. These are optional enrichment fields — a set_issue_type call with a valid issue_type should still execute even if confidence is malformed.
File: actions/setup/js/collect_ndjson_output.cjs (validation loop) and/or safe_output_type_validator.cjs
Context
Problem
The agent intermittently sends
confidenceas a numeric probability (e.g.,0.9,"0.95") instead of the required enum string (LOW,MEDIUM,HIGH). When this happens, the validator incollect_ndjson_output.cjsrejects the entire message, causing valid tool calls (with correctrationale,issue_type,field_name, etc.) to be silently dropped.Evidence
From a test run, ingestion step:
The agent sent
"confidence": "0.95"and"confidence": 0.9instead of"HIGH"/"MEDIUM".Two-part fix
1. Strengthen the INTENT description suffix
Current (PR #42776):
Proposed:
File:
actions/setup/js/generate_safe_outputs_tools.cjs(thehasRuntimeFeature(runtimeFeatures, "issue_intents")block)2. Make validation lenient for optional intent fields
When
confidenceorrationalefails validation, strip the invalid field and pass the message through rather than rejecting the entire message. These are optional enrichment fields — aset_issue_typecall with a validissue_typeshould still execute even ifconfidenceis malformed.File:
actions/setup/js/collect_ndjson_output.cjs(validation loop) and/orsafe_output_type_validator.cjsContext
confidenceandrationaleare optional fields in the tool schema (safe_outputs_tools.json)