Skip to content

Harden issue-intent confidence handling: clearer suffix + lenient validation #43164

Description

@alondahari

Problem

The agent intermittently sends confidence as a numeric probability (e.g., 0.9, "0.95") instead of the required enum string (LOW, MEDIUM, HIGH). When this happens, the validator in collect_ndjson_output.cjs rejects the entire message, causing valid tool calls (with correct rationale, issue_type, field_name, etc.) to be silently dropped.

Evidence

From a test run, ingestion step:

##[warning] Validation errors found:
  - Line 1: set_issue_type 'confidence' must be one of: LOW, MEDIUM, HIGH
  - Line 3: set_issue_field 'confidence' must be a string
  - Line 4: set_issue_field 'confidence' must be a string
  - Line 5: set_issue_field 'confidence' must be a string
  - Line 6: set_issue_field 'confidence' must be a string

The agent sent "confidence": "0.95" and "confidence": 0.9 instead of "HIGH" / "MEDIUM".

Two-part fix

1. Strengthen the INTENT description suffix

Current (PR #42776):

INTENT: Include rationale (max 280 chars) and confidence (LOW/MEDIUM/HIGH) with each call.

Proposed:

INTENT: Include rationale (string, max 280 chars) and confidence (string, exactly one of: LOW, MEDIUM, HIGH) with each call.

File: actions/setup/js/generate_safe_outputs_tools.cjs (the hasRuntimeFeature(runtimeFeatures, "issue_intents") block)

2. Make validation lenient for optional intent fields

When confidence or rationale fails validation, strip the invalid field and pass the message through rather than rejecting the entire message. These are optional enrichment fields — a set_issue_type call with a valid issue_type should still execute even if confidence is malformed.

File: actions/setup/js/collect_ndjson_output.cjs (validation loop) and/or safe_output_type_validator.cjs

Context

Metadata

Metadata

Labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions