Skip to content

Add autoloop perf-comparison program for tsb vs pandas benchmarking - #87

Merged
mrjf merged 5 commits into
mainfrom
copilot/add-autoloop-performance-test
Apr 12, 2026
Merged

Add autoloop perf-comparison program for tsb vs pandas benchmarking#87
mrjf merged 5 commits into
mainfrom
copilot/add-autoloop-performance-test

Conversation

Copilot AI commented Apr 10, 2026

Copy link
Copy Markdown
Contributor

New autoloop program that iteratively benchmarks tsb (TypeScript/Bun) against pandas (Python), one function per iteration, with results displayed on the playground doc site.

Autoloop program

  • .autoloop/programs/perf-comparison/program.md — runs every 6h, metric is benchmarked_functions (higher is better). Each iteration picks an unbenchmarked function, writes matching TS + Python benchmarks, runs both, records results.

Benchmark infrastructure

  • benchmarks/tsb/ and benchmarks/pandas/ — paired benchmark scripts with identical datasets (100K elements, 5 warm-up + 50 measured iterations)
  • benchmarks/run_benchmarks.sh — orchestrator that discovers pairs, runs both, computes ratios, writes results.json
  • Seed benchmark: bench_series_creation for both runtimes

Playground page

  • playground/benchmarks.html — bar chart + detail table, dynamically loads results.json
  • playground/index.html — added Performance section linking to benchmarks
  • .github/workflows/pages.yml — copies benchmark results into deployed site

benchmarks page

Copilot AI and others added 2 commits April 10, 2026 14:51
…d playground page

- Create .autoloop/programs/perf-comparison/program.md autoloop program definition
- Add benchmarks/tsb/ and benchmarks/pandas/ with seed Series creation benchmarks
- Add benchmarks/run_benchmarks.sh to orchestrate both TS and Python benchmarks
- Add playground/benchmarks.html for displaying performance comparison metrics
- Update playground/index.html with Performance section linking to benchmarks
- Update pages.yml to copy benchmark results into deployed playground site

Agent-Logs-Url: https://github.com/githubnext/tsessebe/sessions/4b3aa7c2-4060-4c79-b63c-1717b2502bb1

Co-authored-by: mrjf <180956+mrjf@users.noreply.github.com>
- Add error handling for malformed JSON in benchmark output parsing
- Fix division-by-zero edge case in ratio calculation
- Use string-quoted type hints in Python for broader compatibility

Agent-Logs-Url: https://github.com/githubnext/tsessebe/sessions/4b3aa7c2-4060-4c79-b63c-1717b2502bb1

Co-authored-by: mrjf <180956+mrjf@users.noreply.github.com>
Copilot AI changed the title [WIP] Add new autoloop program to compare TypeScript and pandas performance Add autoloop perf-comparison program for tsb vs pandas benchmarking Apr 10, 2026
Copilot AI requested a review from mrjf April 10, 2026 14:57
Copilot finished work on behalf of mrjf April 10, 2026 14:57
@mrjf
mrjf marked this pull request as ready for review April 10, 2026 15:45
@mrjf

mrjf commented Apr 10, 2026

Copy link
Copy Markdown
Contributor

@copilot fix ci

- Remove unused `Label` import in tests/stats/rank.test.ts (lint error: noUnusedImports)
- Fix benchmarks/results.json formatting to satisfy biome formatter

Agent-Logs-Url: https://github.com/githubnext/tsessebe/sessions/d1b6f11d-7523-4655-b5fe-d8a5cb191ffe

Co-authored-by: mrjf <180956+mrjf@users.noreply.github.com>

Copilot AI commented Apr 10, 2026

Copy link
Copy Markdown
Contributor Author

@copilot fix ci

Fixed in 05c8606. Two issues:

  1. Unused Label import in tests/stats/rank.test.ts (pre-existing noUnusedImports lint error)
  2. benchmarks/results.json formatting didn't match biome's expected format

Lint, typecheck, and all 1130 tests pass now.

Copilot finished work on behalf of mrjf April 10, 2026 15:54
@github-actions

Copy link
Copy Markdown
Contributor

Warning

The push_to_pull_request_branch operation failed: Failed to apply patch. The code changes were not applied.

Evergreen Fix: Merge Conflict Resolved

I resolved the merge conflict in .github/workflows/pages.yml by merging main into this branch.

Conflict: Both this PR and main modified the same section of pages.yml. The PR added a benchmark results copy step; main (via #93) added Python setup and validation steps.

Resolution: Both step groups are included in sequence:

  1. Copy benchmark results to playground (from this PR)
  2. Set up Python 3.12 + install pandas/numpy (from main)
  3. Validate Python playground examples (from main)

Generated by Evergreen — PR Health Keeper · ● 1.4M ·

@mrjf

mrjf commented Apr 10, 2026

Copy link
Copy Markdown
Contributor

@copilot resolve the merge conflicts in this pull request

…performance-test

# Conflicts:
#	.github/workflows/pages.yml

Co-authored-by: mrjf <180956+mrjf@users.noreply.github.com>

Copilot AI commented Apr 10, 2026

Copy link
Copy Markdown
Contributor Author

@copilot resolve the merge conflicts in this pull request

Resolved in 6ad6a7a. The only conflict was in .github/workflows/pages.yml — main added Python validation steps at the same location where our branch added the benchmark results copy step. Kept both: benchmark copy runs first, then Python setup + validation.

Copilot finished work on behalf of mrjf April 10, 2026 17:38
@mrjf
mrjf merged commit d2d2836 into main Apr 12, 2026
3 checks passed
@mrjf
mrjf deleted the copilot/add-autoloop-performance-test branch April 12, 2026 11:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Neeed a new autoloop program to compare typescript and pandas performance

2 participants