[experiments] Daily Experiment Report — 2026-06-22

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-06-22

37 experiments across 34 workflows. 9 ready (all variants ≥ min_samples), 28 still collecting. ⚠️ 1 balance alert.

Note: gh aw experiments CLI could not be installed (network restriction). State read from experiments/* git branches. GitHub API unavailable — outcome p-values not computed.

⚡ Quick Stats

Metric	Value
Active experiments	37
🟢 Ready for analysis	9
🟡 Still collecting	28
⚠️ Balance alerts	1
Outcome significance	0 (GitHub API unavailable)

---

🟢 Ready for Analysis (9 experiments)

All variants ≥ min_samples. Outcome significance pending GitHub API access.

Experiment	Workflow	Variants	n (ctrl/var)	Balance	Rec
caveman	smoke-copilot	no / yes	97/96	✅ p=0.900	READY
subagent_model	smoke-copilot	small / large	86/87	✅ p=0.898	READY
sub_agent_strategy	smoke-gemini	single_agent / sub_agents	58/92	⚠️ p=0.006	EXTEND (fix balance)
sub_agent_strategy	smoke-antigravity	single_agent / sub_agents	58/63	✅ p=0.654	READY
caveman	smoke-copilot-aoai-apikey	yes / no	35/34	✅ p=0.871	READY
subagent_model	smoke-copilot-aoai-apikey	small / large	35/34	✅ p=0.871	READY
prompt_style	daily-community-attribution	concise / verbose	24/24	✅ p=1.000	READY
caveman	smoke-copilot-aoai-entra	no / yes	20/21	✅ p=0.850	READY
subagent_model	smoke-copilot-aoai-entra	small / large	20/21	✅ p=0.850	READY

📊 Charts for top 5 experiments

caveman · smoke-copilot

model · smoke-copilot_subagent

strategy · smoke-gemini_sub_agent

strategy · smoke-antigravity_sub_agent

caveman · smoke-copilot-aoai-apikey

🟡 Still Collecting — Top 10 by runs

Experiment	Workflow	Runs	min_s	Slowest variant
tone_variant	aw-failure-investigator	86	50	narrative: ████░░░░ 25/50
prompt_style	daily-astrostylelite-markdown-spellcheck	50	30	concise: ██████░░ 24/30
output_format	daily-issues-report	47	30	collapsible: ██████░░ 22/30
prompt_style	daily-news	42	30	concise: █████░░░ 18/30
reasoning_depth	daily-security-red-team	41	30	single_pass: █████░░░ 19/30
output_format	daily-compiler-quality	37	20	detailed: ██████░░ 15/20
output_format	daily-code-metrics	36	20	executive_summary: ███████░ 17/20
semgrep_output_format	daily-semgrep-scan	36	30	bullet_list: ███░░░░░ 10/30
output_format	deep-report	34	15	full_briefing: ██████░░ 11/15
prompt_compression	agent-performance-analyzer	33	14	verbose: ███████░ 12/14

+18 more — see all in git state branches.

---

🔍 Detailed Analysis — READY Experiments

caveman · smoke-copilot

Balance: ✅ balanced (chi2=0.01, p=0.9002) · min_samples=20 · total runs=193

• no ← ctrl: n=97 (50.3%) ████████ 97/20
• yes: n=96 (49.7%) ████████ 96/20

Rec: READY_FOR_ANALYSIS

---

subagent_model · smoke-copilot

Balance: ✅ balanced (chi2=0.01, p=0.8978) · min_samples=20 · total runs=173

• small ← ctrl: n=86 (49.7%) ████████ 86/20
• large: n=87 (50.3%) ████████ 87/20

Rec: READY_FOR_ANALYSIS

---

sub_agent_strategy · smoke-gemini

Balance: ⚠️ IMBALANCED (chi2=7.71, p=0.0055) · min_samples=30 · total runs=150

• single_agent ← ctrl: n=58 (38.7%) ████████ 58/30
• sub_agents: n=92 (61.3%) ████████ 92/30

⚠️ Assignment is imbalanced (p=0.0055 < 0.05). Fix randomization before interpreting outcomes.

Rec: EXTEND (fix balance first)

---

sub_agent_strategy · smoke-antigravity

Balance: ✅ balanced (chi2=0.21, p=0.6539) · min_samples=30 · total runs=121

• single_agent ← ctrl: n=58 (47.9%) ████████ 58/30
• sub_agents: n=63 (52.1%) ████████ 63/30

Rec: READY_FOR_ANALYSIS

---

caveman · smoke-copilot-aoai-apikey

Balance: ✅ balanced (chi2=0.01, p=0.8713) · min_samples=20 · total runs=69

• yes ← ctrl: n=35 (50.7%) ████████ 35/20
• no: n=34 (49.3%) ████████ 34/20

Rec: READY_FOR_ANALYSIS

---

Analysis: 37 experiments · 1616 total variant assignments · balance threshold p < 0.05
Run: 27944101270

Warning

Firewall blocked 2 domains

The following domains were blocked by the firewall during workflow execution:

• proxy.golang.org
• releaseassets.githubusercontent.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "proxy.golang.org"
    - "releaseassets.githubusercontent.com"

See Network Configuration for more information.

Generated by 🧪 daily-experiment-report · 214.5 AIC · ⌖ 17.3 AIC · ⊞ 11.2K · ◷

• expires on Jun 25, 2026, 2:09 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #40993.