Merge branch 'main' of https://github.com/SearchSavior/OpenArc

Author: SearchSavior

.cursorignore

@@ -2,7 +2,7 @@
 
 assets/
 benchmarks/
-examples/
+# examples/
 openarc_bench.db
 openarc.log
 scratchpad.md

AGENTS.md

@@ -1,7 +1,6 @@
-OpenArc uses bleeding edge libraries and APIs you may not be familair with. When working through a task, use the deepwiki mcp server to get *contextual* information, and the command line to investigate python surfaces.
+OpenArc uses bleeding edge libraries and APIs you may not be familair with. When working through a task, use the deepwiki mcp server to get *contextual* information, and the command line to investigate python surfaces. Prefer the command line.
 
 - When making changes, dont worry about backward compatibility; we use git for this.
 - Use uv to install dependencies
-- Respect existing patterns in the codebase
-- When the backend is changed, make sure parameters in the frontend are updated.
-- Don't rush, and ask clarifying questions.
+- Don't rush, and ask clarifying questions.
+- In a prompt `ticks` represent grep targets

CONTRIBUTING.md

@@ -5,12 +5,29 @@ The goal with this project has been to make using Intel devices as accelerators
 ## Contributing
 
 - Open an issue before beginning work 
+- Joining discord before making any issues around contributing is encouraged
 
 ## Guidelines
 
 - Create separate PRs for each feature or fix. Avoid combining unrelated changes in a single PR
 - Consider allowing write access to your branch for faster reviews, as reviewers can push commits directly
 
 
+## Documentation
+
+To build and serve the documentation locally use: 
+
+```
+zensical serve -a localhost:8004
+```
+
+Then you can edit markdown files in `docs/` with updates in real time.
+
+When implementing any changes documentation updates are expected. Keeping docs up to date is an ongoing process.
+
+### Github Actions
+
+When changes are made to any file in `docs/` the live site is rebuilt and deployed automatically, meaning we can change it like code in PRs and on merge the workflow executes.
+
 
 

README.md

@@ -4,56 +4,47 @@
 [![Hugging Face](https://img.shields.io/badge/🤗%20Hugging%20Face-Echo9Zulu-yellow)](https://huggingface.co/Echo9Zulu)
 [![Devices](https://img.shields.io/badge/Devices-CPU%2FGPU%2FNPU-blue)](https://github.com/openvinotoolkit/openvino)
 [![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/SearchSavior/OpenArc)
+[![Docs](https://img.shields.io/badge/📖%20Docs-blue)](https://searchsavior.github.io/OpenArc/)
 
 > [!NOTE]
 > OpenArc is under active development.
 
-**OpenArc** is an inference engine for Intel devices. Serve LLMs, VLMs, Whisper, Kokoro-TTS, Qwen-TTS, Qwen-ASR, Embedding and Reranker models over OpenAI compatible endpoints, powered by OpenVINO on your device. Local, private, open source AI.  
+**OpenArc** is an inference engine for Intel devices. 
 
-Drawing on ideas from `llama.cpp`, `vLLM`, `transformers`, `OpenVINO Model Server`, `Ray`, `Lemonade`, and other projects cited below, OpenArc has been a way for me to learn about inference engines by trying to build one myself.
+Serve LLMs, VLMs, Whisper, Kokoro-TTS, Qwen-TTS, Qwen-ASR, Embedding and Reranker models over OpenAI compatible endpoints, powered by OpenVINO on your device. Local, private, open source AI.  
 
-Along the way a Discord community has formed around this project! If you are interested in using Intel devices for AI and machine learning, feel free to stop by. 
+Drawing on ideas from `llama.cpp`, `vLLM` and other inference engines, OpenArc is a community-driven effort meant to push the Arc ecosytem forward with great performance, high usability and a knowledgeable community of users who just want Arc to work for their usecases.
+
+Our Discord is probably the best place to keep up with all things Arc; If you are interested in using Intel devices for AI and machine learning, feel free to stop by!
 
 Thanks to everyone on Discord for their continued support!
 
 > [!NOTE]
-> Documentation has been ported to a Zensical site. It's still WIP, and the site isn't live.
-> To build and serve the docs after install:
-```
-zensical serve -a localhost:8004
-```
-## Table of Contents
+>  Docs now live [here](https://searchsavior.github.io/OpenArc/)
 
 
 
-- [Features](#features)
-- [Quickstart](#quickstart)
-  - [Linux](#linux)
-  - [Windows](#windows)
-  - [Docker](#docker)
-
 
 ## Features
-  - NEW! Containerization with Docker #60 by @meatposes
-  - NEW! Speculative decoding support for LLMs #57 by @meatposes
-  - NEW! Streaming cancellation support for LLMs and VLMs
+  - Containerization with Docker
+  - Speculative decoding
+  - Streaming cancellation support for LLMs and VLMs
   - Multi GPU Pipeline Paralell
   - CPU offload/Hybrid device
   - NPU device support
   - OpenAI compatible endpoints
       - `/v1/models`
-      - `/v1/completions`: `llm` only
+      - `/v1/completions`
       - `/v1/chat/completions`
-      - `/v1/audio/transcriptions`: `whisper`, `qwen3_asr`
-      - `/v1/audio/speech`: `kokoro` only       
-      - `/v1/embeddings`: `qwen3-embedding` #33 by @mwrothbe
-      - `/v1/rerank`: `qwen3-reranker` #39 by @mwrothbe
+      - `/v1/audio/transcriptions`
+      - `/v1/audio/speech`       
+      - `/v1/embeddings`
+      - `/v1/rerank`
   - `jinja` templating with `AutoTokenizers`
   - OpenAI Compatible tool calls with streaming and paralell 
     - tool call parser currently reads "name", "argument" 
   - Fully async multi engine, multi task architecture
   - Model concurrency: load and infer multiple models at once
-  - Automatic unload on inference failure
   - `llama-bench` style benchmarking for `llm` w/automatic sqlite database
   - metrics on every request
     - ttft
@@ -63,156 +54,17 @@ zensical serve -a localhost:8004
     - tpot
     - load time
     - stream mode
-  - More OpenVINO [examples](examples/)
+  - OpenVINO [examples](examples/)
   - OpenVINO implementation of [hexgrad/Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M)
   - OpenVINO implementation of Qwen3-TTS and Qwen3-ASR
+    - Voice Cloning
+    - Voice Design
+    - Custom Voice
 
 
 > [!NOTE] 
-> Interested in contributing? Please open an issue before submitting a PR!
-
-<div align="right">
-
-[↑ Top](#table-of-contents)
-
-</div>
-
-## Quickstart 
-
-<details id="linux">
-<summary><strong style="font-size: 1.2em;">Linux</strong></summary>
-
-<br>
-
-1. OpenVINO requires **device specifc drivers**.
- 
-- Visit [OpenVINO System Requirments](https://docs.openvino.ai/2025/about-openvino/release-notes-openvino/system-requirements.html#cpu) for the latest information on drivers.
-
-2. Install uv from [astral](https://docs.astral.sh/uv/getting-started/installation/#standalone-installer)
-
-3. After cloning use:
-
-```
-uv sync
-```
-
-4. Activate your environment with:
-
-```
-source .venv/bin/activate
-```
-
-Build latest optimum
-```
-uv pip install "optimum-intel[openvino] @ git+https://github.com/huggingface/optimum-intel"
-```
-
-Build latest OpenVINO and OpenVINO GenAI from nightly wheels
-```
-uv pip install --pre -U openvino-genai --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly
-```
-
-5. Set your API key as an environment variable:
-```
-	export OPENARC_API_KEY=<api-key>
-```
-
-6. To get started, run:
-
-```
-openarc --help
-```
-
-</details>
-
-<details id="windows">
-<summary><strong style="font-size: 1.2em;">Windows</strong></summary>
-
-<br>
-
-1. OpenVINO requires **device specifc drivers**.
- 
-- Visit [OpenVINO System Requirments](https://docs.openvino.ai/2025/about-openvino/release-notes-openvino/system-requirements.html#cpu) to get the latest information on drivers.
-
-2. Install uv from [astral](https://docs.astral.sh/uv/getting-started/installation/#standalone-installer)
+> Interested in contributing? Please open an issue before submitting a PR and join Discord so we can discuss!
 
-3. Clone OpenArc, enter the directory and run:
-  ```
-  uv sync
-  ```
-
-4. Activate your environment with:
-
-```
-.venv\Scripts\activate
-```
-
-Build latest optimum
-```
-uv pip install "optimum-intel[openvino] @ git+https://github.com/huggingface/optimum-intel"
-```
-
-Build latest OpenVINO and OpenVINO GenAI from nightly wheels
-```
-uv pip install --pre -U openvino-genai --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly
-```
-
-5. Set your API key as an environment variable:
-```
-setx OPENARC_API_KEY openarc-api-key
-```
-
-6. To get started, run:
-
-```
-openarc --help
-```
-
-</details>
-
-<details id="docker">
-<summary><strong style="font-size: 1.2em;">Docker</strong></summary>
-
-<br>
-
-Instead of fighting with Intel's own docker images, we built our own which is as close to boilerplate as possible. For a primer on docker [check out this video](https://www.youtube.com/watch?v=DQdB7wFEygo).
-
-
-**Build and run the container:**
-```bash
-docker-compose up --build -d
-```
-
-**Run the container:**
-```bash
-docker run -d -p 8000:8000 openarc:latest
-```
-**Enter the container:**
-```bash
-docker exec -it openarc /bin/bash
-```
-
-## Environment Variables
-
-```bash
-export OPENARC_API_KEY="openarc-api-key" # default, set it to whatever you want
-export OPENARC_AUTOLOAD_MODEL="model_name" # model_name to load on startup
-export MODEL_PATH="/path/to/your/models" # mount your models to `/models` inside the container
-docker-compose up --build -d
-```
-
-
-Take a look at the [docker-compose](docker-compose.yaml) and [Dockerfile](Dockerfile) ([Battlemage Dockerfile](Battlemage.Dockerfile)) for more details.
-
-</details>
-
-<br>
-
-> [!NOTE]
-> Need help installing drivers? [Join our Discord](https://discord.gg/Bzz9hax9Jq) or open an issue.
-
-> [!NOTE] 
-> uv has a [pip interface](https://docs.astral.sh/uv/pip/) which is a drop in replacement for pip, but faster. Pretty cool, and a good place to start learning uv.
 
 
 ## Acknowledgments