Speed.AI - LLM Benchmarking

Easy benchmarking of LLMs using Llama.cpp engine

About

Speed.AI is an Android app for benchmarking locally run LLM models using the Llama.cpp engine. You can also chat with LLMs and view detailed benchmarking data.

Setup

Clone the repository and initialize its submodules:

git clone https://github.com/TIC-13/llm-benchmarking-lcpp.git
cd llm-benchmarking-lcpp
git submodule update --init --recursive

Benchmark Ranking

Benchmarking results are sent to a ranking system, where you can compare performance across different devices. The ranking is shared with the Speed.AI - AI Benchmarking app.

To host your own ranking instance, check out these repositories:

• Frontend
• Backend

Hosting a ranking instance is optional; the app works independently.

Configuring the Environment

To connect the app to a ranking instance, configure the following environment variables in local.properties:

• API_ADDRESS: Backend address
• API_KEY: A base64-encoded 32-byte (AES-256) key (must match the backend key)
• RANKING_ADDRESS: URL of the ranking server

Adding New Models

To add a new model, edit LLMViewModel.kt and append its Hugging Face .gguf download URL to the huggingFaceUrls list.

Technologies Used

The inference engine in this app is taken from SmolChat.