Skip to content

Commit df394aa

Browse files
authored
feat: update workflows to AI credits billing terminology
2 parents c611242 + c0051b7 commit df394aa

7 files changed

Lines changed: 105 additions & 81 deletions

File tree

.github/workflows/agentic-token-audit.md

Lines changed: 29 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
description: Daily audit of token usage across all agentic workflows with historical trend tracking
2+
description: Daily audit of AI credit spend across all agentic workflows with historical trend tracking
33
on:
44
schedule:
55
- cron: "daily around 12:00 on weekdays"
@@ -71,15 +71,15 @@ steps:
7171
timeout-minutes: 25
7272
---
7373

74-
# Daily Agentic Workflow Token Usage Audit
74+
# Daily Agentic Workflow AI Credit Spend Audit
7575

76-
You are the Agentic Workflow Token Auditor — a workflow that tracks daily token consumption across all agentic workflows in this repository and maintains a historical record for trend analysis.
76+
You are the Agentic Workflow Auditor — a workflow that tracks daily AI credit spend and token consumption across all agentic workflows in this repository and maintains a historical record for trend analysis.
7777

7878
## Mission
7979

80-
1. Parse the pre-downloaded agentic workflow logs and compute per-workflow token usage metrics.
80+
1. Parse the pre-downloaded agentic workflow logs and compute per-workflow AI credit spend and token usage metrics.
8181
2. Persist today's snapshot to repo-memory so the optimizer (and future runs of this audit) can read historical data.
82-
3. Publish a concise audit issue summarizing today's usage and trend highlights.
82+
3. Publish a concise audit issue summarizing today's AI credit spend and trend highlights.
8383

8484
## Data Sources
8585

@@ -103,8 +103,9 @@ Each element of `.runs` is a `RunData` object with (among others):
103103
|---|---|---|
104104
| `workflow_name` | string | Human-readable name |
105105
| `workflow_path` | string | `.github/workflows/....lock.yml` |
106+
| `ai_credits` | float | AI credits consumed (primary billing metric; 1 AIC = $0.01 USD) |
106107
| `token_usage` | int | Total tokens (`omitempty` — treat missing/null as 0) |
107-
| `effective_tokens` | int | Normalized token metric |
108+
| `effective_tokens` | int | Legacy normalized token metric (deprecated; use `ai_credits` for billing) |
108109
| `action_minutes` | float | Billable GitHub Actions minutes |
109110
| `turns` | int | Number of agent turns |
110111
| `duration` | string | Human-readable duration |
@@ -128,9 +129,9 @@ Write a Python script to `/tmp/gh-aw/token-audit/process_audit.py` and run it. T
128129
1. Load `/tmp/gh-aw/token-audit/workflow-logs.json` and extract `.runs`.
129130
2. Filter to `status == "completed"` runs only.
130131
3. Group by `workflow_name` and compute per-workflow aggregates:
131-
- `run_count`, `total_tokens`, `avg_tokens`, `total_turns`, `avg_turns`, `total_action_minutes`, `error_count`, `warning_count`
132-
4. Compute an overall summary: total runs, total tokens, total action minutes.
133-
5. Sort workflows descending by `total_tokens`.
132+
- `run_count`, `total_ai_credits`, `avg_ai_credits`, `total_tokens`, `avg_tokens`, `total_turns`, `avg_turns`, `total_action_minutes`, `error_count`, `warning_count`
133+
4. Compute an overall summary: total runs, total AI credits, total tokens, total action minutes.
134+
5. Sort workflows descending by `total_ai_credits`.
134135
6. Save the result to `/tmp/gh-aw/token-audit/audit_snapshot.json` with this shape:
135136

136137
```json
@@ -139,13 +140,16 @@ Write a Python script to `/tmp/gh-aw/token-audit/process_audit.py` and run it. T
139140
"period_days": 30,
140141
"overall": {
141142
"total_runs": N,
143+
"total_ai_credits": F,
142144
"total_tokens": N,
143145
"total_action_minutes": F
144146
},
145147
"workflows": [
146148
{
147149
"workflow_name": "...",
148150
"run_count": N,
151+
"total_ai_credits": F,
152+
"avg_ai_credits": F,
149153
"total_tokens": N,
150154
"avg_tokens": N,
151155
"total_turns": N,
@@ -159,15 +163,15 @@ Write a Python script to `/tmp/gh-aw/token-audit/process_audit.py` and run it. T
159163
}
160164
```
161165

162-
Handle null/missing `token_usage` by treating them as 0.
166+
Handle null/missing `ai_credits` and `token_usage` by treating them as 0.
163167

164168
## Phase 2 — Persist Snapshot to Repo-Memory
165169

166170
1. Read the snapshot from `/tmp/gh-aw/token-audit/audit_snapshot.json`.
167171
2. Copy it to `/tmp/gh-aw/repo-memory/default/YYYY-MM-DD.json` (today's UTC date).
168172
3. This file is what the optimizer workflow reads to identify high-usage workflows.
169173

170-
Also maintain a rolling summary file at `/tmp/gh-aw/repo-memory/default/rolling-summary.json` that contains an array of daily overall totals (date, total_tokens, total_runs, total_action_minutes) for the last 90 entries. Load the existing file, append today's entry, trim to 90, and save.
174+
Also maintain a rolling summary file at `/tmp/gh-aw/repo-memory/default/rolling-summary.json` that contains an array of daily overall totals (date, total_ai_credits, total_tokens, total_runs, total_action_minutes) for the last 90 entries. Load the existing file, append today's entry, trim to 90, and save.
171175

172176
Do not append a synthetic zero-valued entry to `rolling-summary.json` when either of these conditions is true:
173177

@@ -180,8 +184,8 @@ Report those two cases differently in the issue as described below so the empty-
180184

181185
Create up to two chart images in `/tmp/gh-aw/token-audit/charts/` using Python, `matplotlib`, and `seaborn` with `whitegrid` styling:
182186

183-
1. **Token usage by workflow** (`token_by_workflow.png`): a horizontal bar chart of the top 15 workflows by total tokens from `audit_snapshot.json`.
184-
2. **Historical token trend** (`token_trend.png`): a line chart from `rolling-summary.json`.
187+
1. **AI credit spend by workflow** (`ai_credits_by_workflow.png`): a horizontal bar chart of the top 15 workflows by total AI credits from `audit_snapshot.json`.
188+
2. **Historical AI credit trend** (`ai_credits_trend.png`): a line chart from `rolling-summary.json`.
185189

186190
Chart requirements:
187191

@@ -191,8 +195,8 @@ Chart requirements:
191195
- Save only PNG files.
192196
- If there are fewer than 2 rolling-summary points, skip the trend chart and explain why in the issue.
193197
- After generating each chart, call `upload_asset` with its file path.
194-
- In the issue template below, replace `UPLOAD_URL_WORKFLOW_PLACEHOLDER` with the URL returned for `token_by_workflow.png`.
195-
- In the issue template below, replace `UPLOAD_URL_TREND_PLACEHOLDER` with the URL returned for `token_trend.png`.
198+
- In the issue template below, replace `UPLOAD_URL_WORKFLOW_PLACEHOLDER` with the URL returned for `ai_credits_by_workflow.png`.
199+
- In the issue template below, replace `UPLOAD_URL_TREND_PLACEHOLDER` with the URL returned for `ai_credits_trend.png`.
196200
- If a chart is skipped, omit that image markdown line entirely instead of leaving a placeholder behind.
197201

198202
## Phase 4 — Publish Audit Issue
@@ -213,45 +217,46 @@ Create an issue with these sections:
213217
214218
- **Period**: last 24 hours (YYYY-MM-DD to YYYY-MM-DD)
215219
- **Total runs**: N
220+
- **Total AI credits**: N.NN AIC
216221
- **Total tokens**: N (formatted with commas)
217222
- **Total Actions minutes**: X.X min
218223
- **Active workflows**: N
219224
220-
### 🏆 Top 5 Workflows by Token Usage
225+
### 🏆 Top 5 Workflows by AI Credit Spend
221226
222-
| Workflow | Runs | Total Tokens | Avg Tokens |
227+
| Workflow | Runs | Total AI Credits | Avg AI Credits |
223228
|---|---|---|---|
224229
| ... | ... | ... | ... |
225230
226231
### 📈 Trends
227232
228233
Embed chart images using uploaded asset URLs when available:
229234
230-
![Token Usage by Workflow](UPLOAD_URL_WORKFLOW_PLACEHOLDER)
235+
![AI Credit Spend by Workflow](UPLOAD_URL_WORKFLOW_PLACEHOLDER)
231236
232-
![Historical Token Trend](UPLOAD_URL_TREND_PLACEHOLDER)
237+
![Historical AI Credit Trend](UPLOAD_URL_TREND_PLACEHOLDER)
233238
234-
Summarize token changes from `rolling-summary.json` when historical data is available.
239+
Summarize AI credit and token changes from `rolling-summary.json` when historical data is available.
235240
236241
<details>
237242
<summary><b>Full Per-Workflow Breakdown</b></summary>
238243
239-
[Complete table of all workflows sorted by total tokens]
244+
[Complete table of all workflows sorted by total AI credits]
240245
241246
</details>
242247
243248
### 💡 Observations
244249
245-
- Identify any workflow with >30% of total tokens as a "heavy hitter"
250+
- Identify any workflow with >30% of total AI credits as a "heavy hitter"
246251
- Note workflows with high error/warning counts relative to runs
247-
- Flag any workflow whose avg tokens per run exceeds 100,000
252+
- Flag any workflow whose avg AI credits per run exceeds 1.00 AIC
248253
249254
**Data snapshot**: `memory/token-audit/YYYY-MM-DD.json`
250255
```
251256

252257
## Important Notes
253258

254-
- Use `// 0` (null coalescing) in jq and `.get(field, 0)` in Python for nullable numeric fields.
259+
- Use `// 0` (null coalescing) in jq and `.get(field, 0)` in Python for nullable numeric fields (`ai_credits`, `token_usage`).
255260
- Distinguish between these two cases in the issue:
256261
- the raw `.runs` array is empty
257262
- the raw `.runs` array is non-empty but none of the runs are `status == "completed"`

.github/workflows/agentic-token-optimizer.md

Lines changed: 16 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
description: Daily optimizer that identifies a high-token-usage agentic workflow, audits its runs, and recommends efficiency improvements including inline sub-agent refactors when warranted
2+
description: Daily optimizer that identifies high-AI-credit-spend agentic workflows, audits their runs, and recommends efficiency improvements including inline sub-agent refactors when warranted
33
on:
44
schedule:
55
- cron: "daily around 14:00 on weekdays"
@@ -71,6 +71,7 @@ steps:
7171
| select(.status == "completed")
7272
| {
7373
workflow_name: .workflow_name,
74+
ai_credits: (.ai_credits // 0),
7475
tokens: (.token_usage // 0),
7576
turns: (.turns // 0),
7677
action_minutes: (.action_minutes // 0)
@@ -80,12 +81,14 @@ steps:
8081
| map({
8182
workflow_name: .[0].workflow_name,
8283
run_count: length,
84+
total_ai_credits: (map(.ai_credits) | add),
85+
avg_ai_credits: ((map(.ai_credits) | add) / length),
8386
total_tokens: (map(.tokens) | add),
8487
avg_tokens: ((map(.tokens) | add) / length),
8588
total_turns: (map(.turns) | add),
8689
total_action_minutes: (map(.action_minutes) | add)
8790
})
88-
| sort_by(.total_tokens)
91+
| sort_by(.total_ai_credits)
8992
| reverse
9093
| .[:10]
9194
)
@@ -107,9 +110,9 @@ steps:
107110
fi
108111
---
109112

110-
# Agentic Workflow Token Usage Optimizer
113+
# Agentic Workflow AI Credit Spend Optimizer
111114

112-
You are the Agentic Workflow Token Optimizer. Pick one high-token workflow, audit recent runs, and create a conservative optimization issue with measurable improvements. Your recommendations may include prompt, tool, reliability, setup-prefix, and inline sub-agent improvements when the evidence supports them.
115+
You are the Agentic Workflow Optimizer. Pick one high-AI-credit-spend workflow, audit recent runs, and create a conservative optimization issue with measurable improvements. Your recommendations may include prompt, tool, reliability, setup-prefix, and inline sub-agent improvements when the evidence supports them.
113116

114117
## Objectives
115118

@@ -148,23 +151,24 @@ Prefer `--jq` on `gh api` calls over a separate `| jq` step when the filter is s
148151
## Data Inputs
149152

150153
- `/tmp/gh-aw/token-audit/all-runs.json`: full 7-day run data (`gh aw logs --json`).
151-
- `/tmp/gh-aw/token-audit/top-workflows.json`: pre-aggregated top 10 workflows by total tokens.
154+
- `/tmp/gh-aw/token-audit/top-workflows.json`: pre-aggregated top 10 workflows by total AI credits.
152155
- `/tmp/gh-aw/repo-memory/default/YYYY-MM-DD.json`: daily audit snapshots.
153156
- `/tmp/gh-aw/repo-memory/default/optimization-log.json`: prior optimizations (if present).
154157

155-
Treat missing numeric fields (`token_usage`, `turns`, `action_minutes`) as `0`.
158+
Treat missing numeric fields (`ai_credits`, `token_usage`, `turns`, `action_minutes`) as `0`.
156159

157160
## Phase 1 — Select Target
158161

159162
- Start from `top-workflows.json`.
160163
- Exclude workflows optimized in the last 14 days (use `optimization-log.json`).
161164
- Exclude workflows with "Token" in the name to avoid self-targeting.
162-
- Choose the highest token workflow that remains.
165+
- Choose the highest AI-credit-spend workflow that remains.
163166
- If no snapshot/history exists, derive candidates directly from `all-runs.json`.
164167

165168
Then collect run-level data for the selected workflow:
166169

167170
- run count
171+
- total and average AI credits
168172
- total and average tokens
169173
- total and average turns
170174
- conclusions/error patterns
@@ -176,8 +180,8 @@ Use this compact analysis matrix:
176180
| Area | Required checks | Output |
177181
|---|---|---|
178182
| Tool usage | Compare configured tools from workflow source vs observed usage across multiple runs | Keep / Consider removing / Remove |
179-
| Token efficiency | Evaluate token totals, effective tokens, cache efficiency, turns | Top token waste drivers |
180-
| Reliability | Repeated errors, warnings, retries, missing tools | Token waste from failures |
183+
| AI credit spend | Evaluate AI credits, token totals, cache efficiency, turns | Top spend drivers |
184+
| Reliability | Repeated errors, warnings, retries, missing tools | AI credit waste from failures |
181185
| Prompt efficiency | Redundant instructions, overlong sections, avoidable iteration | Prompt reduction opportunities |
182186
| Structural optimization | Repeated setup/tool-call prefixes and sections suited for inline sub-agents | Extract setup / Add sub-agent / Keep in main agent |
183187

@@ -279,10 +283,10 @@ Create one issue with:
279283

280284
- **Target workflow + reason selected**
281285
- **Analysis period + runs analyzed**
282-
- **Token profile table** (total tokens, avg tokens/run, avg turns/run, cache efficiency)
286+
- **Spend profile table** (total AI credits, avg AI credits/run, total tokens, avg turns/run, cache efficiency)
283287
- **Ranked recommendations** with:
284288
- title
285-
- estimated token savings per run
289+
- estimated AI credit savings per run
286290
- concrete action
287291
- evidence from observed runs
288292
- **Optional structural optimizations** for shared setup prefixes and inline sub-agents when supported by the analysis
@@ -300,7 +304,7 @@ Create one issue with:
300304

301305
Append one entry to `/tmp/gh-aw/repo-memory/default/optimization-log.json`:
302306

303-
`{"date":"YYYY-MM-DD","workflow_name":"...","total_tokens_analyzed":N,"runs_audited":N,"recommendations_count":N,"subagent_candidates":N,"estimated_savings_per_run":N}`
307+
`{"date":"YYYY-MM-DD","workflow_name":"...","total_ai_credits_analyzed":F,"total_tokens_analyzed":N,"runs_audited":N,"recommendations_count":N,"subagent_candidates":N,"estimated_ai_credit_savings_per_run":F}`
304308

305309
Use `subagent_candidates` for the count of inline sub-agent candidates you actually recommend in the issue body.
306310

CHANGELOG.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,12 @@ Changelog entries should describe what changed for users of these workflows, not
2323
- Omit vague filler like "various fixes" when the actual effect can be named.
2424
- Do not include contributor handles, reviewer names, or other attribution in release entries.
2525

26+
## [0.3.0] - 2026-06-08
27+
28+
### Changed
29+
30+
- Updated both workflows to track and report AI credits (AIC) as the primary billing metric, aligned with the GitHub Copilot AI credits billing model introduced on June 1, 2026 (1 AIC = $0.01 USD). The daily audit snapshot and rolling summary now include `total_ai_credits`; audit and optimizer issues now lead with AI credit spend. Effective tokens remain available as a legacy compatibility field in the run data but are no longer the primary reporting metric.
31+
2632
## [0.2.2] - 2026-06-01
2733

2834
### Changed

README.md

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,16 @@
11
# `agentic-ops`
2-
> "Audit token spend, surface waste, and optimize agentic workflows with confidence!"
2+
> "Audit AI credit spend, surface waste, and optimize agentic workflows with confidence!"
33
4-
`agentic-ops` is a focused bundle of GitHub Agentic Workflows for teams scaling agentic automation and wanting better visibility into usage, trends, and optimization opportunities. Instead of guessing which workflows are token-heavy or where token waste is hiding, this package gives you an audit trail, historical reporting, and conservative recommendations you can review before making changes.
4+
`agentic-ops` is a focused bundle of GitHub Agentic Workflows for teams scaling agentic automation and wanting better visibility into usage, trends, and optimization opportunities. Instead of guessing which workflows are driving the highest AI credit spend or where waste is hiding, this package gives you an audit trail, historical reporting, and conservative recommendations you can review before making changes.
55

66
## Introduction
77

88
It is built for platform engineers, developer productivity teams, and repository maintainers who are scaling agentic workflows and need a practical way to keep them efficient. The bundle helps solve a common problem with AI automation: token usage grows quickly, but the signals for where to improve are scattered across workflow runs and logs. With `agentic-ops`, you get repeatable workflows that make usage measurable, optimization opportunities actionable, and efficiency work easier to operationalize.
99

1010
## Key Features
1111

12-
- **Clear operational visibility** with a daily token audit that captures usage, trends, and workflow-level hotspots.
13-
- **Actionable optimization guidance** that identifies high-token workflows and proposes safe, conservative improvements.
12+
- **Clear operational visibility** with a daily audit that captures AI credit spend, usage trends, and workflow-level hotspots.
13+
- **Actionable optimization guidance** that identifies high-AI-credit-spend workflows and proposes safe, conservative improvements.
1414
- **Faster efficiency improvements** by helping teams find waste before it becomes recurring operational overhead.
1515
- **Built for real GitHub workflows** using GitHub Agentic Workflows, so installation and adoption fit naturally into existing repositories.
1616
- **Useful historical context** through shared snapshots that support trend analysis instead of one-off debugging.
@@ -41,17 +41,17 @@ Required configuration after installation:
4141

4242
After installation, you can use the included workflows to:
4343

44-
- run a daily audit of workflow token usage
45-
- identify the workflows consuming the most tokens
44+
- run a daily audit of AI credit spend and workflow token usage
45+
- identify the workflows consuming the most AI credits
4646
- generate optimization recommendations grounded in recent run data
4747
- surface workflows that are good candidates for inline sub-agent refactors
4848

4949
Included workflows:
5050

5151
| Workflow | What it does |
5252
| ----- | --- |
53-
| [`Daily Agentic Workflow Token Usage Audit`](https://github.com/githubnext/agentic-ops/blob/main/workflows/agentic-token-audit.md?plain=1) | Collects recent agentic workflow usage and creates a daily audit snapshot. |
54-
| [`Agentic Workflow Token Usage Optimizer`](https://github.com/githubnext/agentic-ops/blob/main/workflows/agentic-token-optimizer.md?plain=1) | Analyzes token-heavy workflows and proposes conservative token-reduction changes, including inline sub-agent opportunities when they are a strong fit. |
53+
| [`Daily Agentic Workflow AI Credit Spend Audit`](https://github.com/githubnext/agentic-ops/blob/main/workflows/agentic-token-audit.md?plain=1) | Collects recent agentic workflow usage and creates a daily AI credit spend snapshot. |
54+
| [`Agentic Workflow AI Credit Spend Optimizer`](https://github.com/githubnext/agentic-ops/blob/main/workflows/agentic-token-optimizer.md?plain=1) | Analyzes high-AI-credit-spend workflows and proposes conservative efficiency changes, including inline sub-agent opportunities when they are a strong fit. |
5555

5656
## License
5757

aw.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
manifest-version: "1"
22
name: agentic-ops
3-
description: Reusable agentic workflows for auditing and optimizing workflow token usage.
3+
description: Reusable agentic workflows for auditing AI credit spend and optimizing workflow efficiency.
44
emoji: "🤖"
55
files:
66
- workflows/agentic-token-audit.md

0 commit comments

Comments
 (0)