Skip to content

benchtest: scaffold #428

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sugh01 wants to merge 9 commits into
base: main
Choose a base branch
from
Open

benchtest: scaffold #428

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
61c721d
benchtest: scaffold
sugh01 Jan 23, 2026
3a1a012
benchtest: fix file paths
sugh01 Jan 23, 2026
305d016
benchtest: Add mock benchmark data for testing the comparison pipelin…
sugh01 Jan 23, 2026
f15b8c0
test: Improve mock data generation to create proper JSON files inline
sugh01 Jan 23, 2026
97cd560
ci: Add check for cmd directory existence before running compare script
sugh01 Jan 23, 2026
5fea3f5
fix: Replace heredoc with echo statements to fix YAML syntax error
sugh01 Jan 23, 2026
ea7440b
fix: Add issues:write permission for PR comments
sugh01 Jan 23, 2026
3e51ca7
Merge branch 'main' into bench-test-compare
sugh01 Jan 23, 2026
420e371
Rename handlers_bench_test.go to handlers_benchmark_test.go
sugh01 Jan 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
217 changes: 217 additions & 0 deletions .github/workflows/benchmark-compare.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,217 @@
name: Benchmark Comparison

on:
pull_request:
paths:
- 'services/asset/**'
- '.github/workflows/benchmark-compare.yaml'
- 'cmd/parse-benchmarks/**'
- 'cmd/compare-benchmarks/**'

permissions:
contents: read
pull-requests: write
issues: write

env:
GO_VERSION: '1.25.2'

jobs:
# Run benchmarks on the PR feature branch
benchmark-feature:
name: Benchmark Feature Branch
runs-on: ubuntu-latest
timeout-minutes: 35
steps:
- name: Checkout feature branch
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4
with:
fetch-depth: 0

- name: Set up Go
uses: actions/setup-go@d35c59abb061a4a6fb18e82ac0862c26744d6ab5 # v5
with:
go-version: ${{ env.GO_VERSION }}
cache: false

- name: Display branch info
run: |
echo "Branch: $(git rev-parse --abbrev-ref HEAD)"
echo "Commit: $(git rev-parse HEAD)"

- name: Run benchmarks
run: |
echo "=== Running benchmarks on feature branch ==="
go test -bench=. -benchmem -benchtime=5s -timeout=30m -run=^$ \
./services/asset/httpimpl \
| tee feature-benchmark-output.txt

- name: Parse results
continue-on-error: true
run: |
if [ -d ./cmd/parse-benchmarks ]; then
go run ./cmd/parse-benchmarks \
-input feature-benchmark-output.txt \
-output feature-benchmarks.json \
-commit "${{ github.sha }}" \
-branch "PR-${{ github.event.pull_request.number }}" \
-pr "${{ github.event.pull_request.number }}"
else
echo "Parse script not yet available, will use mock data"
fi

- name: Upload artifact
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
with:
name: feature-benchmarks
path: feature-benchmarks.json
retention-days: 30

# Run benchmarks on main branch as baseline
benchmark-main:
name: Benchmark Main Branch
runs-on: ubuntu-latest
timeout-minutes: 35
steps:
- name: Checkout main branch
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4
with:
ref: main
fetch-depth: 0

- name: Set up Go
uses: actions/setup-go@d35c59abb061a4a6fb18e82ac0862c26744d6ab5 # v5
with:
go-version: ${{ env.GO_VERSION }}
cache: false

- name: Display branch info
run: |
echo "Branch: $(git rev-parse --abbrev-ref HEAD)"
echo "Commit: $(git rev-parse HEAD)"

- name: Run benchmarks
run: |
echo "=== Running benchmarks on main branch ==="
go test -bench=. -benchmem -benchtime=5s -timeout=30m -run=^$ \
./services/asset/httpimpl \
| tee main-benchmark-output.txt

- name: Parse results
continue-on-error: true
run: |
if [ -d ./cmd/parse-benchmarks ]; then
go run ./cmd/parse-benchmarks \
-input main-benchmark-output.txt \
-output main-benchmarks.json \
-commit "$(git rev-parse HEAD)" \
-branch "main"
else
echo "Parse script not yet available, will use mock data"
fi

- name: Upload artifact
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
with:
name: main-benchmarks
path: main-benchmarks.json
retention-days: 30

# Compare results and post report
compare:
name: Compare & Report
runs-on: ubuntu-latest
needs: [benchmark-feature, benchmark-main]
if: always()
steps:
- name: Checkout code
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4

- name: Set up Go
uses: actions/setup-go@d35c59abb061a4a6fb18e82ac0862c26744d6ab5 # v5
with:
go-version: ${{ env.GO_VERSION }}
cache: false

- name: Download feature benchmarks
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
with:
name: feature-benchmarks
continue-on-error: true

- name: Download main benchmarks
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
with:
name: main-benchmarks
continue-on-error: true

- name: Use mock benchmarks if artifacts unavailable
run: |
if [ ! -f feature-benchmarks.json ]; then
echo "Creating mock feature benchmarks"
cp .github/workflows/mock-main-benchmarks.json feature-benchmarks.json
sed -i 's/"main"/"feature"/g; s/59718/60000/g; s/abc1234/def5678/g' feature-benchmarks.json
fi
if [ ! -f main-benchmarks.json ]; then
echo "Creating mock main benchmarks"
cp .github/workflows/mock-main-benchmarks.json main-benchmarks.json
fi
echo "Benchmarks ready for comparison"
ls -la *.json

- name: Compare benchmarks
id: compare
continue-on-error: true
run: |
echo "=== Comparing Feature vs Main ==="
if [ -d ./cmd/compare-benchmarks ]; then
go run ./cmd/compare-benchmarks \
-current feature-benchmarks.json \
-baseline main-benchmarks.json \
-output comparison-report.md \
-threshold 5.0
else
echo "Compare script not yet available, generating placeholder report"
{
echo "## Benchmark Comparison Report"
echo ""
echo "**Status:** Mock data (actual scripts not yet deployed)"
echo ""
echo "- Regressions: 0"
echo "- Improvements: 0"
echo "- Unchanged: 3"
echo ""
echo "*First run with real data coming soon*"
} > comparison-report.md
fi

- name: Display report
if: always()
run: |
echo "=== Benchmark Comparison Report ==="
cat comparison-report.md

- name: Post comment on PR
uses: actions/github-script@v7
if: always()
continue-on-error: true
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
const fs = require('fs');
const report = fs.readFileSync('comparison-report.md', 'utf8');

github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: report
});

- name: Fail on regressions
if: failure() || steps.compare.outcome == 'failure'
run: |
if grep -q "REGRESSION DETECTED" comparison-report.md; then
echo "❌ Performance regressions detected!"
exit 1
fi
31 changes: 31 additions & 0 deletions .github/workflows/mock-main-benchmarks.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
{
"version": "1.0",
"timestamp": "2026-01-23T13:00:00Z",
"git": {
"commit": "abc1234567890abcdef1234567890abcdef123456",
"branch": "main"
},
"benchmarks": [
{
"name": "BenchmarkGetTransactionJSON",
"ns_per_op": 59718,
"bytes_per_op": 12190,
"allocs_per_op": 120,
"iterations": 104190
},
{
"name": "BenchmarkGetTransactionBinary",
"ns_per_op": 40061,
"bytes_per_op": 5216,
"allocs_per_op": 65,
"iterations": 144621
},
{
"name": "BenchmarkGetTransactionHex",
"ns_per_op": 40518,
"bytes_per_op": 5876,
"allocs_per_op": 68,
"iterations": 146862
}
]
}
Loading
Loading