You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The ARM API Reviewer agent posts its findings as inline PR review comments. Step 8 of the agent definition specifies a "Review-body preamble", a visible ## ARM API Review block stating that the comments come from the agent, that they are critic-verified, and against which commit. Separately, every posted comment ends with a hidden HTML marker <!-- posted-by: arm-api-reviewer-agent ... -->.
In practice the visible preamble is frequently missing. When the agent posts each finding as its own single-comment review with an empty review body, the only provenance left on the PR is the hidden HTML marker, which does not render in Markdown. To a reader, the findings then look like unattributed, context-free comments posted under the reviewer personal GitHub handle.
A spec author hit this on PR #44941: they could not tell where the comments came from and assumed they were spurious. The root cause is that Step 8 describes the preamble but does not (a) require posting through a single review that carries the preamble as its body, and (b) verify after posting that a visible preamble actually landed.
The original run (2026-07-24, operator sandipsh) submitted two COMMENTED reviews with empty bodies (bodyLen=0), so no visible preamble was posted. Affected comments: r3647833902, r3647834008.
A later run (2026-07-27, operator ravimeda) correctly added a review whose body begins with ## ARM API Review. This shows the intended behavior is well defined but not enforced, so whether attribution appears depends on the run.
Prevalence. Of 42 PRs sampled that carry the agent hidden marker, 18 had no visible attribution anywhere (review body or top-level comment) when first checked, and 17 still have none as of this writing (PR #44941 was remediated by the later run). The gap spans more than one operator, so it is systemic to the workflow rather than a single mistake.
On PR #44941 the two original comments are also textually corrupted (for example ^Gpi-version in place of api-version, and stray backslashes where $filter and $top should appear). This is a posting-time escaping defect, where the comment body was built by shell string interpolation instead of JSON serialization. It shares the same Step 8 posting area but is a distinct defect from the missing attribution and can be fixed alongside it.
Make the visible preamble mandatory in Step 8. Post all inline findings from a run through a single review (create_pull_request_review) whose body is the ## ARM API Review preamble. Do not post findings as separate empty-body reviews.
Add a fallback. When the posting path cannot set a review body, post the preamble as a top-level PR comment so visible attribution always exists.
Add a post-condition check. After posting, re-fetch the review and top-level comments and assert that at least one visible body contains the preamble. If none does, post the fallback top-level comment before the run finishes.
Consider a short visible attribution line on each inline comment, not only the hidden marker, so a stray single comment still carries provenance.
Add eval coverage. Add a check to the eval harness that fails when a simulated posting run omits the visible preamble.
Backfill (optional but recommended). Run a one-time reconciliation that adds a preamble or a top-level attribution comment to the open PRs listed under Evidence.
Acceptance criteria
Every run that posts at least one finding also posts a visible ## ARM API Review preamble (review body or top-level comment), confirmed by a re-fetch post-condition.
No run leaves findings whose only provenance is the hidden HTML marker.
The eval harness has a check that fails when a posting run omits the visible preamble.
The open PRs listed under Evidence either have a visible preamble added or are documented as remediated.
Background
The ARM API Reviewer agent posts its findings as inline PR review comments. Step 8 of the agent definition specifies a "Review-body preamble", a visible
## ARM API Reviewblock stating that the comments come from the agent, that they are critic-verified, and against which commit. Separately, every posted comment ends with a hidden HTML marker<!-- posted-by: arm-api-reviewer-agent ... -->.In practice the visible preamble is frequently missing. When the agent posts each finding as its own single-comment review with an empty review body, the only provenance left on the PR is the hidden HTML marker, which does not render in Markdown. To a reader, the findings then look like unattributed, context-free comments posted under the reviewer personal GitHub handle.
A spec author hit this on PR #44941: they could not tell where the comments came from and assumed they were spurious. The root cause is that Step 8 describes the preamble but does not (a) require posting through a single review that carries the preamble as its body, and (b) verify after posting that a visible preamble actually landed.
Evidence
Trigger case, PR #44941:
sandipsh) submitted twoCOMMENTEDreviews with empty bodies (bodyLen=0), so no visible preamble was posted. Affected comments: r3647833902, r3647834008.ravimeda) correctly added a review whose body begins with## ARM API Review. This shows the intended behavior is well defined but not enforced, so whether attribution appears depends on the run.Prevalence. Of 42 PRs sampled that carry the agent hidden marker, 18 had no visible attribution anywhere (review body or top-level comment) when first checked, and 17 still have none as of this writing (PR #44941 was remediated by the later run). The gap spans more than one operator, so it is systemic to the workflow rather than a single mistake.
Currently missing visible attribution:
sandipsh: Cts/johnkoh/tspconfig sdk #44899, Refactor(iotoperationsdataprocessor): migrate to unified folder structure #44830, Add upgradedToV2 to PublicIPAddress and PublicIPPrefix (2025-09-01) #44807, Review request for Microsoft.DBforPostgreSQL to add public preview release of version 2026-07-01-preview #44803, Adding DisableDefaultServerHeaderInResponse Property under Global Con… #44796, [Microsoft.Management] Add listDescendants, listMembers, and reshape listAncestors on serviceGroups (2026-03-01-preview) #44769, Updated the response code for the Service Gateway API and added a new SKU for the Load Balancer. #44745, [OCR] Add Open Capacity Reservation SDK support #44376, [AppService] [Web] Add stable 2026-07-15 with AppServicePlan startWebApps/stopWebApps #43926, Add Bulkactions 2026-07-06-preview version #43925, Add Policy ARM resource for CognitiveServices (2026-07-01-preview) #43898, Add hybridnetwork specification changes from PR #25860 #43894, Move and Remove clientName decorator in models.tsp #43814, [Hardware Security Modules] Updated CloudHsmClusterSkuName from closed enum to extensible enum #43745, [ProgramEnrollment] Add Microsoft.ProgramEnrollment TypeSpec API spec (2026-03-01-preview) #43694razvanbadea-msft: [Event Hubs] Add 2026-07-01-preview API version #44750, [Microsoft.Web] Introduce new api version "2026-08-01" and new resource type "Microsoft.Web/sites/silos", "Microsoft.Web/serverfarms/silos", "Microsoft.Web/sites/managedTrafficConfig/default", ""Microsoft.Web/sites/lastKnownGood/default" #44330Related observation (track separately)
On PR #44941 the two original comments are also textually corrupted (for example
^Gpi-versionin place ofapi-version, and stray backslashes where$filterand$topshould appear). This is a posting-time escaping defect, where the comment body was built by shell string interpolation instead of JSON serialization. It shares the same Step 8 posting area but is a distinct defect from the missing attribution and can be fixed alongside it.References
Requested work
create_pull_request_review) whose body is the## ARM API Reviewpreamble. Do not post findings as separate empty-body reviews.Acceptance criteria
## ARM API Reviewpreamble (review body or top-level comment), confirmed by a re-fetch post-condition.