Skip to content
vLLM

Open source alternatives to vLLM

200 open source apps like vLLM, closest match first.

Top picks

  1. V

    vllm-mlx

    Launch

    vllm-mlx is an inference server for large language models on Apple Silicon. It exposes OpenAI-compatible and Anthropic-compatible APIs, so existing clients and tools can talk to a model running on your Mac. The engine is built on MLX, Apple

    Free

  2. W

    Web LLM

    Launch

    WebLLM is an inference engine that runs large language models directly inside a web browser. It uses WebGPU for hardware acceleration, so text generation runs on the visitor's device without server-side processing. The project's stated goal

    Free

  3. L

    llama.cpp

    Launch

    C/C++ inference engine for large language models, supporting quantization, multi-GPU, Apple Silicon, and an OpenAI-compatible server across a wide range of hardware.

  4. P

    Paddler

    Launch

    Open-source LLM/VLM load balancer and serving platform for self-hosting LLMs (and VLMs) at scale 🏓🦙 Alternative to projects like llm-d, Docker Model Runner, etc but with less moving parts and simple deployments built around ggml ecosystem.

    FreeSelf-hosted

  5. O

    Ollama

    Launch

    Ollama is a tool for running open AI models on your own computer, from the terminal or through apps you build. It runs on macOS, Windows and Linux. When a model is too large for your hardware, cloud models are available too, paid for with u

    Self-hosted

  6. F

    A local LLM server built for concurrent work. Drop-in replacement for Ollama and/or OpenAI and Ollama APIs on one port. Requests that share a prompt reuse each other's KV cache instead of each prefilling it. Rust, wrapping llama.cpp.

    Free

More like vLLM

7
L

LocalAI

Run LLMs, speech, image generation

Launch
8
A

Ai00 Server

The all-in-one RWKV runtime box with embed, RAG, AI agents, and more

Launch
9
M

mlx-serve

Native LLM inference server for Apple Silicon

Launch
10
G

Ggrun

llama.cpp/ik_llama.cpp launcher: loads big MoE models across mismatched

Launch
11
I

Inference Gateway

An open-source, cloud-native, high-performance gateway unifying multiple LLM…

Launch
12
O

ollama

Get up and running with large language models, locally

Launch
13
O

Olla

High-performance lightweight proxy and load balancer for LLM infrastructure

Launch
14
L

Llamastash

Zero-overhead, terminal-native local-LLM runtime manager

Launch
15
T

TurboLLM

Run any local LLM engine, auto-tuned to your GPU

Launch
16
S

SMG

Engine-agnostic LLM gateway in Rust

Launch
17
L

LiteLLM

Acts as a unified proxy across 100+ LLMs

Launch
18
T

Time To First Token

A 10-week, 30-minutes-a-day roadmap for LLM inference serving and optimization

Launch
19
F

Fastllm-proxy

The lowest-overhead LLM router

Launch
20
T

ToshLLM

Run large language models locally on Intel Macs with AMD GPUs

Launch
21
L

Llamafile

Distribute and run LLMs with a single file

Launch
22
O

OpenGateLLM

An open-source, unified interface for running and managing self-hosted LLMs

Launch
23
W

Willow Inference Server

Open source, local, and self-hosted highly optimized language inference…

Launch
24
L

Llmaker

Selfhost modern LLM stacks. Run the whole fleet from your terminal

Launch
25
P

pooled

Peer-to-peer LLM inference in the browser: pool your devices

Launch
26
L

liter-llm

Universal LLM API client — 163 providers, 14 native language bindings…

Launch
27
L

LLM Gateway

Route, manage, and analyze LLM requests across multiple providers with one API

Launch
28
H

HyperQwen

Serve large Qwen models fast on the GPUs you actually own

Launch
29
B

BodhiApp

Run Open Source/Open Weight LLMs locally with OpenAI compatible APIs

Launch
30
L

LlamaChat

Create AI models on Mac, have natural conversations, and gain real-time insights

Launch
31
O

OlliteRT

Turn your Android phone into an OpenAI-compatible LLM inference server

Launch
32
O

OpenLLMetry

Open-source observability platform for LLMs using OpenTelemetry

Launch
33
W

whatcani.run

Find the best models and how to run them locally

Launch
34
T

The OSS LLMOps Stack

Modular, open source LLMOps stack that separates concerns: LiteLLM unifies…

Launch
35
T

Textgen

Open-source desktop app for local LLMs

Launch
36
L

Llamactl

Unified management and routing for llama.cpp, MLX and vLLM models

Launch
37
R

Routerly.ai

Self-hosted LLM gateway that routes requests across AI providers

Launch
38
L

Local Llms On Android

Run local LLMs like Gemma, Qwen, and LLaMA on Android for offline, private

Launch
39
L

Llama3.java

Llama 3+ inference in pure Java

Launch
40
V

VoidLLM

Privacy-first LLM proxy and AI gateway

Launch
41
T

TheWhisper

Optimized Whisper models for streaming and on-device use

Launch
42
G

GoModel

AI gateway written in Go with a unified OpenAI-compatible API

Launch
43
L

LLM X

LLMX; Easiest 3rd party Local LLM UI for the web!

Launch
44
C

ChatLLM-Web

Private local model studio, AI chat

Launch
45
P

Project Maya

Run GLM-5.3-Flash (321B MoE) privately on your own GPU(s)…

Launch
46
M

MLX Tune

Fine-tune LLMs on your Mac with Apple Silicon

Launch
47
L

Lucebox

LLM speculative inference server for heterogeneous hardware & consumer GPUs

Launch
48
O

OmniKey-AI-Unified-Key-Manager

A self-hosted LLM proxy and failover gateway supporting OpenAI & Gemini…

Launch
49
G

Graviton

500B+ LLMs on a Mac Mini

Launch
50
B

Baseten

Inference Platform: Deploy AI models in production

Launch

About vLLM

V

vLLM

Inference and serving engine for large language models, built

vLLM is an inference and serving engine for large language models, built for high throughput and efficient use of memory. It runs open models on your own hardware and exposes them through a drop-in OpenAI-compatible API, so applications that already call that API can switch to a model you serve yourself. Its main techniques are PagedAttention, which manages the attention memory in pages, and advanced scheduling with continuous batching, which keeps the GPU busy by adding new requests while others are still running. The site describes it as easy to deploy across a wide range of open-source models and hardware. Hardware support is listed for NVIDIA CUDA GPUs, AMD ROCm GPUs, Huawei Ascend NPUs, AWS Neuron, Google Cloud TPUs, Intel Gaudi and XPU, CPUs, and Apple Silicon, among others. The quick start lets you pick a stable or nightly build, a platform such as CUDA, ROCm, XPU or CPU, and a package route, then run an install command. Python 3.10 or newer is required, with 3.12 or newer recommended, and uv is suggested for a faster install. vLLM is a community project supported by donations and by compute from several companies. The site also links to docs, recipes, a performance page and a roadmap.

Category
AI
Website
vllm.ai