Documentation | Community Slack

---

LLMariner (= LLM + Mariner) is an extensible open source platform to simplify the management of generative AI workloads. Built on Kubernetes, it enables you to efficiently handle both training and inference data within your own clusters. With OpenAI-compatible APIs, LLMariner leverages an ecosystem of tools, facilitating seamless integration for a wide range of AI-driven applications.

Architecture

LLMariner consists of a control-plane and one or more worker-planes. Both components can operate within a single cluster, but if you want to utilize GPU resources across multiple clusters, they can also be installed into separate clusters:

Control-Plane components:

Expose the OpenAI-compatible APIs and manage the overall state of LLMariner and receive a request from the client.

Worker-Plane components:

Run every worker cluster, process tasks using compute resources such as GPUs in response to requests from the control-plane.

Refer to the High-Level Architecture document and Technical Details document for more information.

Installation

Check out our installation guide, which covers several deployment methods, including setting up a test environment using the kind cluster and building a production-ready environment, among others.

Integration

LLMariner provides OpenAI-compatible APIs, making it easy to integrate with powerful tools such as assistant web UIs, code generation tools, and more. Here are some integration samples:

• Open WebUI: A self-hosted web UI that works with OpenAI-compatible APIs. See Open WebUI integration guide for details.
• Continue: An open-source AI code assistant inside of VS Code and JetBrains. See Continue integration guide for details.
• Weights & Biases (W&B): AI developer platform that can enable you to easily see the progress of your fine-tuning jobs, such as training epoch, loss, etc. See W&B integration guide for details.

Directory Structure

• cli: CLI
• deployments: Helm chart
• integration-examples: Examples of integration of other services with LLMariner
• provision: provisioning scripts
• tutorials: Tutorials

Talks

• Transform Your Kubernetes Cluster Into a GenAI Platform: Get Ready-to-Use LLM APIs Today! - Cloud Native & Kubernetes AI Day 2024 North America

Contributing

See Contributing Guide.

License

See LICENSE.