Web-Based Distributed LLM Inference

This repository contains code for my master theses:

• An Architecture for Web-Based Distributed LLM Inference (Software Engineering & Internet Computing)
• Optimizing Distributed LLM Inference for Heterogeneous Workers through Dynamic Graph Partitioning (Data Science)

Summary

The theses focus on distributed LLM inference on heterogeneous devices using the web to reduce setup complexity.
The first thesis covers the overall architecture. The second thesis focuses on model partitioning.

Structure

• centralized_inference: Contains centralized reference implementations using ONNX Runtime. Used during experiments/prototyping to compare centralized and distributed variants.
• evaluation: Tools to run experiments, process logs, and generate tables/plots.
• foundation: Early experiments and prototyping.
• src: The systems code split into the server and clients/workers.
• util: Tools. Specifically, we use Olive to convert LLMs to ONNX.