[aw] Smoke Gemini failed

[github-actions]

Workflow Failure

Workflow: Smoke Gemini
Branch: copilot/fix-billing-header-injection
Run: https://github.com/github/gh-aw-firewall/actions/runs/25384800437
Pull Request: #2571

⚠️ Engine Failure: The gemini engine terminated unexpectedly.

Last agent output:

    at async file:///opt/hostedtoolcache/node/24.14.1/x64/lib/node_modules/@google/gemini-cli/bundle/gemini.js:9701:26
    at async main (file:///opt/hostedtoolcache/node/24.14.1/x64/lib/node_modules/@google/gemini-cli/bundle/gemini.js:14721:5) {
  cause: {
    code: 429,
    message: 'Resource has been exhausted (e.g. check quota).',
    details: []
  },
  retryDelayMs: undefined
}
{"type":"result","timestamp":"2026-05-05T15:21:10.451Z","status":"error","error":{"type":"unknown","message":"[API Error: Resource has been exhausted (e.g. check quota).]"},"stats":{"total_tokens":200203,"input_tokens":194923,"output_tokens":1696,"cached":85207,"input":109716,"duration_ms":0,"tool_calls":60,"models":{"gemini-2.5-flash-lite":{"total_tokens":7233,"input_tokens":6780,"output_tokens":65,"cached":0,"input":6780},"gemini-3-flash-preview":{"total_tokens":192970,"input_tokens":188143,"output_tokens":1631,"cached":85207,"input":102936}}}}

Action Required

Assign this issue to Copilot using the agentic-workflows sub-agent to automatically debug and fix the workflow failure.

Debug with any coding agent

Use this prompt with any coding agent (GitHub Copilot, Claude, Gemini, etc.):

Debug the agentic workflow failure using https://raw.githubusercontent.com/github/gh-aw/main/debug.md

The failed workflow run is at https://github.com/github/gh-aw-firewall/actions/runs/25384800437

Manually invoke the agent

Debug this workflow failure using your favorite Agent CLI and the agentic-workflows prompt.

• Start your agent
• Load the agentic-workflows prompt from .github/agents/agentic-workflows.agent.md or https://github.com/github/gh-aw/blob/main/.github/agents/agentic-workflows.agent.md
• Type debug the agentic workflow smoke-gemini failure in https://github.com/github/gh-aw-firewall/actions/runs/25384800437

Tip

Stop reporting this workflow as a failure

To stop a workflow from creating failure issues, set report-failure-as-issue: false in its frontmatter:

safe-outputs:
  report-failure-as-issue: false

Generated from Smoke Gemini · ◷

• expires on May 12, 2026, 3:21 PM UTC

[github-actions]

Agent job 25386662796 failed.

⚠️ Engine Failure: The gemini engine terminated unexpectedly.

Last agent output:

    at async (redacted)
    at async main ((redacted) {
  cause: {
    code: 429,
    message: 'Resource has been exhausted (e.g. check quota).',
    details: []
  },
  retryDelayMs: undefined
}
{"type":"result","timestamp":"2026-05-05T15:51:31.273Z","status":"error","error":{"type":"unknown","message":"[API Error: Resource has been exhausted (e.g. check quota).]"},"stats":{"total_tokens":7251,"input_tokens":6780,"output_tokens":67,"cached":0,"input":6780,"duration_ms":0,"tool_calls":0,"models":{"gemini-2.5-flash-lite":{"total_tokens":7251,"input_tokens":6780,"output_tokens":67,"cached":0,"input":6780},"gemini-3-flash-preview":{"total_tokens":0,"input_tokens":0,"output_tokens":0,"cached":0,"input":0}}}}

Generated from Smoke Gemini · ◷

[github-actions]

Agent job 25388379106 failed.

⚠️ Engine Failure: The gemini engine terminated unexpectedly.

Last agent output:

    at async (redacted)
    at async main ((redacted) {
  cause: {
    code: 429,
    message: 'Resource has been exhausted (e.g. check quota).',
    details: []
  },
  retryDelayMs: undefined
}
{"type":"result","timestamp":"2026-05-05T16:23:17.569Z","status":"error","error":{"type":"unknown","message":"[API Error: Resource has been exhausted (e.g. check quota).]"},"stats":{"total_tokens":7277,"input_tokens":6800,"output_tokens":96,"cached":0,"input":6800,"duration_ms":0,"tool_calls":0,"models":{"gemini-2.5-flash-lite":{"total_tokens":7277,"input_tokens":6800,"output_tokens":96,"cached":0,"input":6800},"gemini-3-flash-preview":{"total_tokens":0,"input_tokens":0,"output_tokens":0,"cached":0,"input":0}}}}

Generated from Smoke Gemini · ◷

[github-actions]

Agent job 25390039951 failed.

⚠️ Engine Failure: The gemini engine terminated unexpectedly.

Last agent output:

    at async (redacted)
    at async main ((redacted) {
  cause: {
    code: 429,
    message: 'Resource has been exhausted (e.g. check quota).',
    details: []
  },
  retryDelayMs: undefined
}
{"type":"result","timestamp":"2026-05-05T16:57:16.879Z","status":"error","error":{"type":"unknown","message":"[API Error: Resource has been exhausted (e.g. check quota).]"},"stats":{"total_tokens":7248,"input_tokens":6794,"output_tokens":63,"cached":0,"input":6794,"duration_ms":0,"tool_calls":0,"models":{"gemini-2.5-flash-lite":{"total_tokens":7248,"input_tokens":6794,"output_tokens":63,"cached":0,"input":6794},"gemini-3-flash-preview":{"total_tokens":0,"input_tokens":0,"output_tokens":0,"cached":0,"input":0}}}}

Generated from Smoke Gemini · ◷

[github-actions]

Agent job 25390913760 failed.

🔒 Lock File Out of Sync: The workflow could not start because its compiled lock file no longer matches the source markdown.

This means the workflow's .md file was edited but gh aw compile was not run afterwards to regenerate the corresponding .lock.yml file. The agent is prevented from running against a stale configuration to avoid unexpected behaviour.

To fix, recompile the workflow:

gh aw compile

Then commit and push the updated .lock.yml file.

More ways to recompile

Using the gh-aw MCP server (if configured):

{ "tool": "compile", "arguments": { "validate": true } }

Recompile all workflows at once:

gh aw compile --all

Verify the result:

gh aw compile --validate

How to investigate the mismatch

The workflow run logs contain a verbose debug pass that shows exactly what was hashed. Search the Check workflow lock file step logs for lines starting with [hash-debug] to see:

• The raw frontmatter text that was used as input
• Any imported files that were included in the hash
• The canonical JSON that was fed to SHA-256
• The resulting hash value

This makes it easy to spot accidental whitespace changes, encoding differences, or import path drift.

This mismatch can also happen on a fresh install (even without any manual edits) if gh aw add <path>@<ref> could not resolve the ref to an exact commit SHA during installation. In that case, rerun the add command with an exact SHA (or retry when API/rate-limit conditions recover), then recompile:

gh aw add <path>@<exact-sha>
gh aw compile

How to disable this check

[!CAUTION]
Disabling this check means the agent can run against an out-of-date compiled workflow. Only disable it if you have an alternative mechanism to keep lock files in sync.

Set stale-check: false in the on: section of your workflow frontmatter:

on:
  issues:
    types: [opened]
  stale-check: false

After editing, recompile the workflow: gh aw compile

Generated from Smoke Gemini · ◷

[github-actions]

Agent job 25391268568 failed.

⚠️ Engine Failure: The gemini engine terminated unexpectedly.

Last agent output:

    at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
    at async Models.generateContentStream ((redacted)
    at async (redacted)
    at async (redacted)
    at async retryWithBackoff ((redacted)
    at async GeminiChat.makeApiCallAndProcessStream ((redacted)
    at async GeminiChat.streamWithRetries ((redacted)
    at async Turn.run ((redacted)
    at async GeminiClient.processTurn ((redacted)
{"type":"result","timestamp":"2026-05-05T17:23:42.569Z","status":"error","error":{"type":"unknown","message":"[API Error: exception TypeError: fetch failed sending request]"},"stats":{"total_tokens":0,"input_tokens":0,"output_tokens":0,"cached":0,"input":0,"duration_ms":0,"tool_calls":0,"models":{"gemini-2.5-flash-lite":{"total_tokens":0,"input_tokens":0,"output_tokens":0,"cached":0,"input":0},"gemini-3.1-pro-preview":{"total_tokens":0,"input_tokens":0,"output_tokens":0,"cached":0,"input":0}}}}

Generated from Smoke Gemini · ◷

[github-actions]

Agent job 25380878455 failed.

⚠️ Engine Failure: The gemini engine terminated unexpectedly.

Last agent output:

    at async (redacted)
    at async main ((redacted) {
  cause: {
    code: 429,
    message: 'Resource has been exhausted (e.g. check quota).',
    details: []
  },
  retryDelayMs: undefined
}
{"type":"result","timestamp":"2026-05-05T18:32:18.514Z","status":"error","error":{"type":"unknown","message":"[API Error: Resource has been exhausted (e.g. check quota).]"},"stats":{"total_tokens":7221,"input_tokens":6787,"output_tokens":66,"cached":0,"input":6787,"duration_ms":0,"tool_calls":0,"models":{"gemini-2.5-flash-lite":{"total_tokens":7221,"input_tokens":6787,"output_tokens":66,"cached":0,"input":6787},"gemini-3-flash-preview":{"total_tokens":0,"input_tokens":0,"output_tokens":0,"cached":0,"input":0}}}}

Generated from Smoke Gemini · ◷

[github-actions]

Agent job 25380680673 failed.

⚠️ Engine Failure: The gemini engine terminated unexpectedly.

Last agent output:

    at async (redacted)
    at async main ((redacted) {
  cause: {
    code: 429,
    message: 'Resource has been exhausted (e.g. check quota).',
    details: []
  },
  retryDelayMs: undefined
}
{"type":"result","timestamp":"2026-05-05T18:32:25.634Z","status":"error","error":{"type":"unknown","message":"[API Error: Resource has been exhausted (e.g. check quota).]"},"stats":{"total_tokens":7220,"input_tokens":6769,"output_tokens":69,"cached":0,"input":6769,"duration_ms":0,"tool_calls":0,"models":{"gemini-2.5-flash-lite":{"total_tokens":7220,"input_tokens":6769,"output_tokens":69,"cached":0,"input":6769},"gemini-3-flash-preview":{"total_tokens":0,"input_tokens":0,"output_tokens":0,"cached":0,"input":0}}}}

Generated from Smoke Gemini · ◷

[github-actions]

Agent job 25382431173 failed.

⚠️ Engine Failure: The gemini engine terminated unexpectedly.

Last agent output:

    at async (redacted)
    at async main ((redacted) {
  cause: {
    code: 429,
    message: 'Resource has been exhausted (e.g. check quota).',
    details: []
  },
  retryDelayMs: undefined
}
{"type":"result","timestamp":"2026-05-05T18:32:46.425Z","status":"error","error":{"type":"unknown","message":"[API Error: Resource has been exhausted (e.g. check quota).]"},"stats":{"total_tokens":7242,"input_tokens":6796,"output_tokens":65,"cached":0,"input":6796,"duration_ms":0,"tool_calls":0,"models":{"gemini-2.5-flash-lite":{"total_tokens":7242,"input_tokens":6796,"output_tokens":65,"cached":0,"input":6796},"gemini-3-flash-preview":{"total_tokens":0,"input_tokens":0,"output_tokens":0,"cached":0,"input":0}}}}

Generated from Smoke Gemini · ◷

[github-actions]

Agent job 25382310320 failed.

⚠️ Engine Failure: The gemini engine terminated unexpectedly.

Last agent output:

    at async (redacted)
    at async main ((redacted) {
  cause: {
    code: 429,
    message: 'Resource has been exhausted (e.g. check quota).',
    details: []
  },
  retryDelayMs: undefined
}
{"type":"result","timestamp":"2026-05-05T18:33:19.055Z","status":"error","error":{"type":"unknown","message":"[API Error: Resource has been exhausted (e.g. check quota).]"},"stats":{"total_tokens":7250,"input_tokens":6778,"output_tokens":64,"cached":0,"input":6778,"duration_ms":0,"tool_calls":0,"models":{"gemini-2.5-flash-lite":{"total_tokens":7250,"input_tokens":6778,"output_tokens":64,"cached":0,"input":6778},"gemini-3-flash-preview":{"total_tokens":0,"input_tokens":0,"output_tokens":0,"cached":0,"input":0}}}}

Generated from Smoke Gemini · ◷

[github-actions]

Agent job 25381143603 failed.

⚠️ Engine Failure: The gemini engine terminated unexpectedly.

Last agent output:

    at async (redacted)
    at async main ((redacted) {
  cause: {
    code: 429,
    message: 'Resource has been exhausted (e.g. check quota).',
    details: []
  },
  retryDelayMs: undefined
}
{"type":"result","timestamp":"2026-05-05T18:33:48.169Z","status":"error","error":{"type":"unknown","message":"[API Error: Resource has been exhausted (e.g. check quota).]"},"stats":{"total_tokens":7258,"input_tokens":6786,"output_tokens":85,"cached":980,"input":5806,"duration_ms":0,"tool_calls":0,"models":{"gemini-2.5-flash-lite":{"total_tokens":7258,"input_tokens":6786,"output_tokens":85,"cached":980,"input":5806},"gemini-3-flash-preview":{"total_tokens":0,"input_tokens":0,"output_tokens":0,"cached":0,"input":0}}}}

Generated from Smoke Gemini · ◷

[github-actions]

Agent job 25381146558 failed.

⚠️ Engine Failure: The gemini engine terminated unexpectedly.

Last agent output:

    at async (redacted)
    at async main ((redacted) {
  cause: {
    code: 429,
    message: 'Resource has been exhausted (e.g. check quota).',
    details: []
  },
  retryDelayMs: undefined
}
{"type":"result","timestamp":"2026-05-05T18:34:09.936Z","status":"error","error":{"type":"unknown","message":"[API Error: Resource has been exhausted (e.g. check quota).]"},"stats":{"total_tokens":7269,"input_tokens":6801,"output_tokens":72,"cached":0,"input":6801,"duration_ms":0,"tool_calls":0,"models":{"gemini-2.5-flash-lite":{"total_tokens":7269,"input_tokens":6801,"output_tokens":72,"cached":0,"input":6801},"gemini-3-flash-preview":{"total_tokens":0,"input_tokens":0,"output_tokens":0,"cached":0,"input":0}}}}

Generated from Smoke Gemini · ◷

[github-actions]

Agent job 25394646857 failed.

⚠️ Engine Failure: The gemini engine terminated unexpectedly.

Last agent output:

    at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
    at async Models.generateContentStream ((redacted)
    at async (redacted)
    at async (redacted)
    at async retryWithBackoff ((redacted)
    at async GeminiChat.makeApiCallAndProcessStream ((redacted)
    at async GeminiChat.streamWithRetries ((redacted)
    at async Turn.run ((redacted)
    at async GeminiClient.processTurn ((redacted)
{"type":"result","timestamp":"2026-05-05T18:35:49.545Z","status":"error","error":{"type":"unknown","message":"[API Error: exception TypeError: fetch failed sending request]"},"stats":{"total_tokens":0,"input_tokens":0,"output_tokens":0,"cached":0,"input":0,"duration_ms":0,"tool_calls":0,"models":{"gemini-2.5-flash-lite":{"total_tokens":0,"input_tokens":0,"output_tokens":0,"cached":0,"input":0},"gemini-3.1-pro-preview":{"total_tokens":0,"input_tokens":0,"output_tokens":0,"cached":0,"input":0}}}}

Generated from Smoke Gemini · ◷

[github-actions]

Agent job 25395040735 failed.

Generated from Smoke Gemini · ◷

[github-actions]

Agent job 25394958644 failed.

⚠️ Engine Failure: The gemini engine terminated unexpectedly.

Last agent output:

    at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
    at async Models.generateContentStream ((redacted)
    at async (redacted)
    at async (redacted)
    at async retryWithBackoff ((redacted)
    at async GeminiChat.makeApiCallAndProcessStream ((redacted)
    at async GeminiChat.streamWithRetries ((redacted)
    at async Turn.run ((redacted)
    at async GeminiClient.processTurn ((redacted)
{"type":"result","timestamp":"2026-05-05T18:39:46.589Z","status":"error","error":{"type":"unknown","message":"[API Error: exception TypeError: fetch failed sending request]"},"stats":{"total_tokens":0,"input_tokens":0,"output_tokens":0,"cached":0,"input":0,"duration_ms":0,"tool_calls":0,"models":{"gemini-2.5-flash-lite":{"total_tokens":0,"input_tokens":0,"output_tokens":0,"cached":0,"input":0},"gemini-3.1-pro-preview":{"total_tokens":0,"input_tokens":0,"output_tokens":0,"cached":0,"input":0}}}}

Generated from Smoke Gemini · ◷