Reduce tool denial failures in three high-frequency agentic workflows#42273
Merged
Conversation
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Reduce structural complexity to resolve tool denial limit
Reduce tool denial failures in three high-frequency agentic workflows
Jun 29, 2026
Copilot stopped reviewing on behalf of
pelikhan due to an error
June 29, 2026 16:17
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adds explicit tool-budget guidance to multiple GitHub Actions workflow prompt files and introduces Go tests that assert these instructions are present, helping ensure agents emit partial output instead of failing silently when tool-call limits are reached.
Changes:
- Add “tool budget awareness” and batch-size/early-emission instructions to workflow prompt markdown files.
- Add Go tests that read workflow prompt files and assert required budget-related guidance is included.
- Update workflow lock metadata hashes to reflect prompt content changes.
Show a summary per file
| File | Description |
|---|---|
| pkg/workflow/prompts_test.go | Adds assertions that specific workflow prompts include tool-budget guidance. |
| pkg/workflow/daily_safe_output_integrator_workflow_test.go | Adds a test to verify the safe-output integrator prompt includes budget/batch guidance. |
| .github/workflows/layout-spec-maintainer.md | Documents new lockfile scan batch limit, early emission, and tool-budget awareness. |
| .github/workflows/layout-spec-maintainer.lock.yml | Updates generated lock metadata hashes after prompt changes. |
| .github/workflows/daily-safe-output-integrator.md | Documents batch-size and tool-budget awareness for integrator runs. |
| .github/workflows/daily-safe-output-integrator.lock.yml | Updates generated lock metadata hashes after prompt changes. |
| .github/workflows/daily-formal-spec-verifier.md | Adds tool-budget awareness instruction with a fallback order. |
| .github/workflows/daily-formal-spec-verifier.lock.yml | Updates generated lock metadata hashes after prompt changes. |
Review details
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 8/8 changed files
- Comments generated: 4
- Review effort level: Low
| 9. **Sort alphabetically**: Keep lists organized and easy to scan | ||
| 10. **Real SHA values**: When listing actions, use actual commit SHAs found | ||
| 11. **Batch size**: When scanning lock files for patterns, run at most **50 lock files total** across all category queries (actions, artifacts, job names, paths) combined per run. If there are more than 50 lock files, sample the first 50 and note the remainder in the PR body. | ||
| 12. **Early emission**: After completing the scan and generating the document, emit the `create-pull-request` safe output immediately. Do not delay to gather additional data beyond the batch limit. |
| ) | ||
|
|
||
| func TestDailySafeOutputIntegratorHasToolBudgetAwareness(t *testing.T) { | ||
| sourceContent, err := os.ReadFile("../../.github/workflows/daily-safe-output-integrator.md") |
Comment on lines
+385
to
+387
| if !strings.Contains(workflow, "If you are approaching the tool call limit, emit a partial result immediately rather than continuing to gather more data.") { | ||
| t.Fatal("Expected daily-formal-spec-verifier workflow to contain tool budget awareness instruction") | ||
| } |
Comment on lines
+372
to
+382
| func TestDailyFormalSpecVerifierHasToolBudgetAwareness(t *testing.T) { | ||
| repoRoot, err := findRepoRoot() | ||
| if err != nil { | ||
| t.Fatalf("Failed to find repo root: %v", err) | ||
| } | ||
|
|
||
| workflowFile := filepath.Join(repoRoot, ".github", "workflows", "daily-formal-spec-verifier.md") | ||
| content, err := os.ReadFile(workflowFile) | ||
| if err != nil { | ||
| t.Fatalf("Failed to read workflow file: %v", err) | ||
| } |
Contributor
|
🎉 This pull request is included in a new release. Release: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three daily/weekly workflows (
daily-safe-output-integrator,daily-formal-spec-verifier,layout-spec-maintainer) were routinely hitting the 5/5 tool denial limit before emitting any safe output, causing complete run loss.Changes
Scope reduction (batch size limits)
Early emission
create-pull-requestimmediately after the scan — do not defer to gather more dataTool budget awareness (all three workflows)
Added an explicit instruction to each workflow's constraints/guardrails:
For Formal Spec Verifier, the instruction includes a priority ordering to resolve ambiguity between two fallback modes: emit partial
create_issueif formalization is substantially complete (≥5 predicates, ≥3 test functions), otherwise emitreport_incompleteimmediately.Tests
Added one test per workflow asserting the budget-awareness and batch-limit guidance is present in the source:
TestDailySafeOutputIntegratorHasToolBudgetAwarenessTestDailyFormalSpecVerifierHasToolBudgetAwarenessTestLayoutSpecMaintainerHasToolBudgetAwareness