Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 13 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
# SIA (Self-Improving AI)

[![arXiv](https://img.shields.io/badge/arXiv-2605.27276-b31b1b.svg)](https://arxiv.org/abs/2605.27276)
[![CI](https://github.com/hexo-ai/sia/actions/workflows/ci.yml/badge.svg)](https://github.com/hexo-ai/sia/actions/workflows/ci.yml)
[![PyPI version](https://img.shields.io/pypi/v/sia-agent.svg)](https://pypi.org/project/sia-agent/)
[![PyPI downloads](https://img.shields.io/pypi/dm/sia-agent.svg)](https://pypi.org/project/sia-agent/)
[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)
[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
[![PyPI version](https://img.shields.io/pypi/v/sia-agent.svg)](https://pypi.org/project/sia-agent/)

<p align="center">
<a href="https://star-history.com/#hexo-ai/sia&Date">
Expand All @@ -19,14 +21,14 @@ Official implementation of [**SIA: Self Improving AI with Harness & Weight Updat

SIA is a Self Improving AI framework to autonomously improve the performance of any AI system (Model / Agent) on a benchmark task.

> **Just want to try it?** Skip to [Run SIA locally](#2-run-sia-locally-with-built-in-tasks).
> **Just want to try it?** Skip to [Run SIA locally](#run-sia-locally-with-built-in-tasks).

## Introduction Videos

- [SIA setup](https://www.loom.com/share/be0534bc818d408bab937033c6457ec9)
- [SIA Runs Visualizer](https://www.loom.com/share/5b1dc2dc858b4493b4b348f0b88d5b9e)

### Architecture
## Architecture

<p align="center"><img src="docs/flow.png" alt="SIA orchestration flow" width="720"></p>
<p align="center"><i>Control flow between Meta, Target, and Feedback agents over successive generations.</i></p>
Expand All @@ -41,15 +43,15 @@ SIA operates by coordinating three main types of AI agents that work together to
This iterative process allows the system to autonomously refine and enhance its ability to solve scientific tasks.


### Benchmark Results
## Benchmark Results

<p align="center"><img src="docs/mlebench.png" alt="MLE Bench Results" width="720"><br><i>OpenAI MLE-Bench Hard: a gauntlet of real Kaggle ML competitions where agents must write, run, and iterate full ML pipelines. SIA ranks #1 across all generations tested.</i></p>

<p align="center"><img src="docs/lawbench.png" alt="LawBench Results" width="720"><br><i>LawBench: predict the criminal charge from Chinese court case descriptions across 191 charge categories. SIA-W+H reaches 70.1% Top-1 accuracy, beating the prior SOTA of 45%.</i></p>

<p align="center"><img src="docs/trimul_cuda.png" alt="TriMul CUDA Results" width="720"><br><i>AlphaFold-3 TriMul Triton Kernel: implement and optimize the Triangle Multiplicative Update as a Triton kernel, preserving correctness while hitting H100 latency targets. SIA-W+H achieves 14x speedup over baseline.</i></p>

<p align="center"><img src="docs/denoising.png" alt="Denoising Results" width="720"><br><i>scRNA-seq Denoising: impute missing gene expression values in single-cell RNA sequencing data. SIA-W+H scores 0.289 MSE<sub>norm</sub>, surpassing the prior SOTA of 0.220.</i></p>
<p align="center"><img src="docs/denoising.png" alt="Denoising Results" width="720"><br><i>scRNA-seq Denoising: impute missing gene expression values in single-cell RNA sequencing data. SIA-W+H scores 0.289 MSE<sub>norm</sub>, surpassing the prior SOTA of 0.240.</i></p>

---

Expand Down Expand Up @@ -103,6 +105,8 @@ Artifacts land in `runs/run_{run_id}/gen_{n}/`:
While a run is in progress a **live dashboard** auto-starts at
`http://127.0.0.1:8000` (disable with `--no-web`).

**Security:** the default `--sandbox none` mode runs agent-generated code with host access. For untrusted tasks or models, use `--sandbox docker`, which isolates execution in a Docker container with no network access. See [SECURITY.md](SECURITY.md) for the full security model.

### Common flags (`sia run`)

| Flag | Default | Description |
Expand Down Expand Up @@ -256,6 +260,10 @@ python my-task/data/public/evaluate.py --gen-dir runs/run_1/gen_1 # should wri

Full contract, return-format rules, and a complete example: [EVALUATION_GUIDE.md](EVALUATION_GUIDE.md).

## Contributing

Contributions are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup, test/lint/type-check commands, and guidance for adding new tasks.

---

## Further reading
Expand Down
Loading