Agent-Researcher: Automated ML Research with Gemini Agents

An automated research pipeline that uses Google Gemini LLM agents to conduct end-to-end machine learning experiments — from literature review through paper writing.

Built on top of the Agent Laboratory framework by Schmidgall et al., with modifications for Google Gemini model support and batch experiment orchestration.

---

🔬 What This Does

Agent-Researcher runs fully autonomous research workflows:

1. Literature Review — Agents search arXiv, read papers, and summarize relevant work
2. Plan Formulation — Postdoc and PhD agents collaboratively design experiments
3. Data Preparation — Software engineer agent prepares datasets (e.g., from HuggingFace)
4. Experimentation — MLE solver iteratively writes, runs, and debugs experiment code
5. Results Interpretation — Agents analyze outputs and draw conclusions
6. Report Writing — LaTeX paper is drafted, reviewed by simulated peer reviewers, and refined

🧪 Experiment Results

We ran the SGC (Simplifying Graph Convolutional Networks) reproduction task across multiple Gemini models:

Model	Runs	Best Test Accuracy	Report Score
gemini-3.5-flash	1	81.60% (SGC, K=2)	6.50/10
gemini-3.5-flash-lite	3	See logs	See logs
gemini-3.1-flash-lite	3	See logs	See logs

Full results including logs, generated papers, source code, and state saves are in the results/ directory.

---

🛠️ Setup

Prerequisites

• Python 3.11+
• A Google Gemini API key
• CUDA-enabled GPU (optional, for experiment code execution)

Installation

git clone https://github.com/bananatruck/Agent-Researcher.git
cd Agent-Researcher
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Configuration

1. Set your API key:

   export GEMINI_API_KEY="your-key-here"

2. Define experiments in YAML files under experiment_configs/:

   copilot-mode: False
   research-topic: "Your goal is to write a PyTorch implementation to reproduce..."
   llm-backend: "gemini-3.5-flash"
   lit-review-backend: "gemini-3.5-flash"
   language: "English"
   num-papers-lit-review: 3
   mlesolver-max-steps: 8
   papersolver-max-steps: 8
   compile-latex: False

3. Edit run_experiments.py to reference your YAML configs and set the number of runs:

   RUN_INDICES = (1, 2, 3)  # 3 runs per model

---

🚀 Running Experiments

python run_experiments.py

This will:

• Validate your API key and check model availability
• Register models in the local registry
• Run each experiment config for each run index
• Archive results to results/<name>_run_<N>/

Task Notes

Customize agent instructions in config.py under TASK_NOTE_LLM. This controls what the agents are told during each research phase (e.g., which SDK to use for inference, dataset size limits, language).

---

📁 Project Structure

├── run_experiments.py          # Batch experiment runner (Gemini SDK)
├── ai_lab_repo.py              # Core research pipeline entry point
├── config.py                   # API URLs, task notes, and settings
├── experiment_configs/         # YAML experiment configurations
│   ├── sgc_gemini_3_5_flash_lite.yaml
│   └── sgc_gemini_3_1_flash_lite.yaml
├── results/                    # Archived experiment outputs
│   ├── gemini_3_5_flash_run_1/
│   │   ├── research_dir/       # Generated paper, code, figures
│   │   ├── state_saves/        # Checkpoint .pkl files
│   │   └── *.log               # Full experiment log
│   └── ...
├── configs/models_pricing.json # Model registry with pricing
└── model_registry.py           # Runtime model resolution

---

🔧 Key Modifications from Upstream

This fork adds:

• run_experiments.py — Batch runner with Google GenAI SDK for model validation
• Gemini task notes — config.py updated with GenAI SDK instructions (replaces OpenAI references)
• Base URL fix — GOOGLE_GENERATIVE_API_BASE_URL corrected with /openai/ suffix
• Stdin deadlock fix — Subprocess input() calls no longer block the batch runner
• YAML-based configs — Declarative experiment definitions

---

📜 Attribution

This project is a fork of Agent Laboratory by Schmidgall et al.

@misc{schmidgall2025agentlaboratoryusingllm,
      title={Agent Laboratory: Using LLM Agents as Research Assistants},
      author={Samuel Schmidgall and Yusheng Su and Ze Wang and Ximeng Sun and Jialian Wu and Xiaodong Yu and Jiang Liu and Zicheng Liu and Emad Barsoum},
      year={2025},
      eprint={2501.04227},
      archivePrefix={arXiv},
      primaryClass={cs.HC},
      url={https://arxiv.org/abs/2501.04227},
}

📄 License

MIT License — see upstream for details.