🐍 Kairu

⚡ Real-time inference optimizer for LLMs — faster generation, smarter decoding, and live observability 📊✨

---

🌊 Meaning

Kairu (流れる) — to flow, to stream.

Inference should be fluid — not blocked by latency, inefficiency, or opaque performance.

---

🚀 What it is

Kairu wraps any HuggingFace model and adds:

• 🦅 Speculative decoding (EAGLE-style)

• ⏩ Dynamic early exit

• 💸 Token budget enforcement

• 📊 Live dashboard:

  • tokens/sec
  • latency
  • quality tradeoffs

---

❗ The problem

Speculative decoding works — but:

• locked inside heavy frameworks (vLLM, etc.)
• hard to experiment with
• no lightweight tooling
• no built-in observability

---

🧠 What you learn

• Speculative decoding internals (EAGLE, Medusa)
• KV cache management
• Streaming inference
• Performance optimization

---

🚀 Quick Start

pip install kairu

from kairu import wrap_model

model = wrap_model("your-model")
model.generate("Hello world")

---

🎯 Vision

Make LLM inference fast, transparent, and controllable.