|
| 1 | +--- |
| 2 | +title: "20x Faster TRL Fine-tuning with RapidFire AI" |
| 3 | +thumbnail: /blog/assets/rapidfireai/thumbnail.png |
| 4 | +authors: |
| 5 | +- user: kbigdelysh |
| 6 | + guest: true |
| 7 | + org: rapidfire-ai-inc |
| 8 | +- user: arunkk09 |
| 9 | + guest: true |
| 10 | + org: rapidfire-ai-inc |
| 11 | +- user: qgallouedec |
| 12 | +--- |
| 13 | + |
| 14 | +# 20x Faster TRL Fine-tuning with RapidFire AI |
| 15 | + |
| 16 | +Hugging Face TRL now officially integrates with RapidFire AI to accelerate your fine-tuning and post-training experiments. TRL users can now discover, install, and run RapidFire AI as the fastest way to compare multiple fine-tuning/post-training configurations to customize LLMs without major code changes and without bloating GPU requirements. |
| 17 | + |
| 18 | +## Why this matters |
| 19 | + |
| 20 | +When fine-tuning or post-training LLMs, teams often do not have the time and/or budget to compare multiple configs even though that can significantly boost eval metrics. RapidFire AI lets you launch multiple TRL configs concurrently--even on a single GPU--and compare them in near real time via a new adaptive, chunk-based scheduling and execution scheme. In internal benchmarks referenced in the TRL page, this delivers ~16–24× higher experimentation throughput than sequentially comparing configs one after another, enabling you to reach much better metrics much faster. |
| 21 | + |
| 22 | + |
| 23 | +*RapidFire AI establishes live three-way communication between your IDE, a metrics dashboard, and a multi-GPU execution backend* |
| 24 | + |
| 25 | +## What you get, out of the box |
| 26 | + |
| 27 | +- **Drop-in TRL wrappers** — Use `RFSFTConfig`, `RFDPOConfig`, and `RFGRPOConfig` as near-zero-code replacements for TRL's SFT/DPO/GRPO configs. |
| 28 | + |
| 29 | +- **Adaptive chunk-based concurrent training** — RapidFire AI shards the dataset into a given number of chunks and cycles configs at chunk boundaries to enable earlier apples-to-apples comparisons and also maximize GPU utilization. |
| 30 | + |
| 31 | +- **Interactive Control Ops (IC Ops)** — From the dashboard itself, you can Stop, Resume, Delete, and Clone-Modify, possibly with Warm-Start, any runs in flight to avoid wasting resources on underperforming configs and double-down on better performing configs--no job restarts, no juggling separate GPUs or clusters, no resource bloat. |
| 32 | + |
| 33 | + |
| 34 | +*Clone promising configurations with modified hyperparameters, optionally warm-starting from the parent's weights, all from the live dashboard* |
| 35 | + |
| 36 | +- **Multi-GPU orchestration** — The RapidFire AI scheduler automatically places and orchestrates configs across available GPUs on chunks of data via effcient shared-memory mechanisms. You focus on your models and eval metrics, not plumbing. |
| 37 | + |
| 38 | +- **MLflow-based dashboard** — Real-time metrics, logs, and IC Ops in one place as soon as you start your experiment. Support for more dashboards such as Trackio, W&B, and TensorBoard coming soon. |
| 39 | + |
| 40 | +## How it works |
| 41 | + |
| 42 | +RapidFire AI splits your dataset randomly into "chunks" and cycles LLM configurations through the GPUs at chunk boundaries. You get incremental signal on eval metrics across all configs much more quickly. The automatic checkpointing via an efficient shared-memory-based adapter/model spilling/loading mechanism keeps training smooth, stable, and consistent. Use IC Ops to adapt mid-flight to stop low-performers earlier and clone promising ones with tweaked config knobs, optionally warm-starting from the parent's weights. |
| 43 | + |
| 44 | + |
| 45 | +*Sequential vs. Task Parallel vs. RapidFire AI: The adaptive scheduler maximizes GPU utilization across multiple configs and GPUs. The bottom row shows IC Ops in action—stopping, cloning, and modifying runs mid-flight.* |
| 46 | + |
| 47 | +## Getting Started |
| 48 | + |
| 49 | +Install RapidFire AI and get running in under a minute: |
| 50 | + |
| 51 | +```bash |
| 52 | +pip install rapidfireai |
| 53 | + |
| 54 | +# Authenticate with Hugging Face |
| 55 | +huggingface-cli login --token YOUR_TOKEN |
| 56 | + |
| 57 | +# Workaround for current issue |
| 58 | +pip uninstall -y hf-xet |
| 59 | + |
| 60 | +# Initialize and start RapidFire AI |
| 61 | +rapidfireai init |
| 62 | +rapidfireai start |
| 63 | +``` |
| 64 | + |
| 65 | +The dashboard launches at `http://localhost:3000` where you can monitor and control all your experiments. |
| 66 | + |
| 67 | +## Supported TRL trainers |
| 68 | + |
| 69 | +- SFT with `RFSFTConfig` |
| 70 | +- DPO with `RFDPOConfig` |
| 71 | +- GRPO with `RFGRPOConfig` |
| 72 | + |
| 73 | +These are designed as drop-in replacements so that you can keep your TRL mental model while gaining far more concurrency and control for your fine-tuning/post-training applications. |
| 74 | + |
| 75 | +## Minimal TRL SFT example |
| 76 | + |
| 77 | +Here's what it looks like to train **multiple configurations concurrently** even on a single GPU: |
| 78 | + |
| 79 | +```python |
| 80 | +from rapidfireai import Experiment |
| 81 | +from rapidfireai.automl import List, RFGridSearch, RFModelConfig, RFLoraConfig, RFSFTConfig |
| 82 | +from datasets import load_dataset |
| 83 | +from transformers import AutoModelForCausalLM, AutoTokenizer |
| 84 | + |
| 85 | +# Setup: load your dataset and define formatting |
| 86 | +dataset = load_dataset("bitext/Bitext-customer-support-llm-chatbot-training-dataset") |
| 87 | +train_dataset = dataset["train"].select(range(128)).shuffle(seed=42) |
| 88 | + |
| 89 | +def formatting_function(row): |
| 90 | + return { |
| 91 | + "prompt": [ |
| 92 | + {"role": "system", "content": "You are a helpful customer support assistant."}, |
| 93 | + {"role": "user", "content": row["instruction"]}, |
| 94 | + ], |
| 95 | + "completion": [{"role": "assistant", "content": row["response"]}] |
| 96 | + } |
| 97 | + |
| 98 | +dataset = dataset.map(formatting_function) |
| 99 | + |
| 100 | +# Define multiple configs to compare |
| 101 | +config_set = List([ |
| 102 | + RFModelConfig( |
| 103 | + model_name="TinyLlama/TinyLlama-1.1B-Chat-v1.0", |
| 104 | + peft_config=RFLoraConfig(r=8, lora_alpha=16, target_modules=["q_proj", "v_proj"]), |
| 105 | + training_args=RFSFTConfig(learning_rate=1e-3, max_steps=128, fp16=True), |
| 106 | + ), |
| 107 | + RFModelConfig( |
| 108 | + model_name="TinyLlama/TinyLlama-1.1B-Chat-v1.0", |
| 109 | + peft_config=RFLoraConfig(r=32, lora_alpha=64, target_modules=["q_proj", "v_proj"]), |
| 110 | + training_args=RFSFTConfig(learning_rate=1e-4, max_steps=128, fp16=True), |
| 111 | + formatting_func=formatting_function, |
| 112 | + ) |
| 113 | +]) |
| 114 | + |
| 115 | +# Run all configs concurrently with chunk-based scheduling |
| 116 | +experiment = Experiment(experiment_name="sft-comparison") |
| 117 | +config_group = RFGridSearch(configs=config_set, trainer_type="SFT") |
| 118 | + |
| 119 | +def create_model(model_config): |
| 120 | + model = AutoModelForCausalLM.from_pretrained( |
| 121 | + model_config["model_name"], |
| 122 | + device_map="auto", torch_dtype="auto" |
| 123 | + ) |
| 124 | + tokenizer = AutoTokenizer.from_pretrained(model_config["model_name"]) |
| 125 | + return (model, tokenizer) |
| 126 | + |
| 127 | +experiment.run_fit(config_group, create_model, train_dataset, num_chunks=4, seed=42) |
| 128 | +experiment.end() |
| 129 | +``` |
| 130 | + |
| 131 | +**What happens when you run this?** |
| 132 | + |
| 133 | +Suppose you run the above on a 2-GPU machine. Instead of training sequentially (Config 1 → wait → Config 2 → wait), both configs train concurrently: |
| 134 | + |
| 135 | +| Approach | Time till Comparative Decision | GPU utilization | |
| 136 | +|----------|-----------------|-----------------| |
| 137 | +| Sequential (traditional) | ~15 minutes | 60% utilization | |
| 138 | +| RapidFire AI (concurrent) | ~5 minutes | 95%+ utilization | |
| 139 | + |
| 140 | +You can get to a comparative decision **3× sooner** on the same resources after both configs finish processing the first data chunk instead of waiting for them to see the whole dataset one after another. Open `http://localhost:3000` to watch live metrics and use IC Ops to stop, clone, or tweak runs in real-time based on what you're seeing. |
| 141 | + |
| 142 | +## Benchmarks: Real-World Speedups |
| 143 | + |
| 144 | +Here is what teams see on time to reach a comparable overall best training loss (across all tried configs) when switching from sequential comparisons to RapidFire AI-enabled hyperparallel experimentation: |
| 145 | + |
| 146 | +| Scenario | Sequential Time | RapidFire AI Time | Speedup | |
| 147 | +|----------|----------------|-------------------|---------| |
| 148 | +| 4 configs, 1 GPU | 120 min | 7.5 min | **16×** | |
| 149 | +| 8 configs, 1 GPU | 240 min | 12 min | **20×** | |
| 150 | +| 4 configs, 2 GPUs | 60 min | 4 min | **15×** | |
| 151 | + |
| 152 | +*Benchmarks on NVIDIA A100 40GB with TinyLlama-1.1B and Llama-3.2-1B models* |
| 153 | + |
| 154 | +## Get Started Today |
| 155 | + |
| 156 | +**🚀 Try it hands-on**: [Interactive Colab Notebook](http://tinyurl.com/rapidfireai-colab) — Zero setup, runs in your browser |
| 157 | + |
| 158 | +**📚 Full Documentation**: [oss-docs.rapidfire.ai](https://oss-docs.rapidfire.ai) — Complete guides, examples, and API reference |
| 159 | + |
| 160 | +**💻 GitHub**: [RapidFireAI/rapidfireai](https://github.com/RapidFireAI/rapidfireai) — Open source, production-ready |
| 161 | + |
| 162 | +**📦 Install via PyPI**: [pypi.org/project/rapidfireai](https://pypi.org/project/rapidfireai) — `pip install rapidfireai` |
| 163 | + |
| 164 | +**💬 Join the Community**: [Discord](https://discord.gg/6vSTtncKNN) — Get help, share results, request features |
| 165 | + |
| 166 | +--- |
| 167 | + |
| 168 | +RapidFire AI was built because the common status quo of trying one config at a time wastes both time and GPU cycles. With this official integration, every TRL user can fine-tune/post-train smarter, iterate faster, and ship better models. |
| 169 | + |
| 170 | +**Try the integration and let us know**: How much faster is your experimentation loop? What should we build next? We're just getting started, and your feedback shapes where we go from here. |
0 commit comments