Skip to content
vLLM

Alternatives to vLLM

200 apps like vLLM, closest match first.

Top picks

  1. V

    vllm-mlx

    Launch

    vllm-mlx is an inference server for large language models on Apple Silicon. It exposes OpenAI-compatible and Anthropic-compatible APIs, so existing clients and tools can talk to a model running on your Mac. The engine is built on MLX, Apple

    Free

  2. W

    Web LLM

    Launch

    WebLLM is an inference engine that runs large language models directly inside a web browser. It uses WebGPU for hardware acceleration, so text generation runs on the visitor's device without server-side processing. The project's stated goal

    Free

  3. L

    llama.cpp

    Launch

    C/C++ inference engine for large language models, supporting quantization, multi-GPU, Apple Silicon, and an OpenAI-compatible server across a wide range of hardware.

  4. P

    Paddler

    Launch

    Open-source LLM/VLM load balancer and serving platform for self-hosting LLMs (and VLMs) at scale 🏓🦙 Alternative to projects like llm-d, Docker Model Runner, etc but with less moving parts and simple deployments built around ggml ecosystem.

    FreeSelf-hosted

  5. L

    LLMKube

    Launch

    LLMKube is a Kubernetes operator for running language models on your own hardware. You write a Model resource and an InferenceService resource in YAML, and the operator serves the model behind an OpenAI-compatible endpoint once it is ready.

    FreeSelf-hosted

  6. O

    Ollama

    Launch

    Ollama is a tool for running open AI models on your own computer, from the terminal or through apps you build. It runs on macOS, Windows and Linux. When a model is too large for your hardware, cloud models are available too, paid for with u

    Self-hosted

More like vLLM

7
F

Fox

A local LLM server built for concurrent work

Launch
8
L

LocalAI

Run LLMs, speech, image generation

Launch
9
A

Ai00 Server

The all-in-one RWKV runtime box with embed, RAG, AI agents, and more

Launch
10
M

mlx-serve

Native LLM inference server for Apple Silicon

Launch
11
G

Ggrun

llama.cpp/ik_llama.cpp launcher: loads big MoE models across mismatched

Launch
12
I

Inference Gateway

An open-source, cloud-native, high-performance gateway unifying multiple LLM…

Launch
13
O

ollama

Get up and running with large language models, locally

Launch
14
O

Olla

High-performance lightweight proxy and load balancer for LLM infrastructure

Launch
15
L

Llamastash

Zero-overhead, terminal-native local-LLM runtime manager

Launch
16
T

TurboLLM

Run any local LLM engine, auto-tuned to your GPU

Launch
17
S

SMG

Engine-agnostic LLM gateway in Rust

Launch
18
L

LiteLLM

Acts as a unified proxy across 100+ LLMs

Launch
19
T

Time To First Token

A 10-week, 30-minutes-a-day roadmap for LLM inference serving and optimization

Launch
20
F

Fastllm-proxy

The lowest-overhead LLM router

Launch
21
T

ToshLLM

Run large language models locally on Intel Macs with AMD GPUs

Launch
22
L

Llamafile

Distribute and run LLMs with a single file

Launch
23
O

OpenGateLLM

An open-source, unified interface for running and managing self-hosted LLMs

Launch
24
W

Willow Inference Server

Open source, local, and self-hosted highly optimized language inference…

Launch
25
L

Llmaker

Selfhost modern LLM stacks. Run the whole fleet from your terminal

Launch
26
P

pooled

Peer-to-peer LLM inference in the browser: pool your devices

Launch
27
L

liter-llm

Universal LLM API client — 163 providers, 14 native language bindings…

Launch
28
L

LLM Gateway

Route, manage, and analyze LLM requests across multiple providers with one API

Launch
29
H

HyperQwen

Serve large Qwen models fast on the GPUs you actually own

Launch
30
B

BodhiApp

Run Open Source/Open Weight LLMs locally with OpenAI compatible APIs

Launch
31
L

LlamaChat

Create AI models on Mac, have natural conversations, and gain real-time insights

Launch
32
H

huggingface-local-models

Select models to run locally

Get
33
O

OlliteRT

Turn your Android phone into an OpenAI-compatible LLM inference server

Launch
34
O

OpenLLMetry

Open-source observability platform for LLMs using OpenTelemetry

Launch
35
W

whatcani.run

Find the best models and how to run them locally

Launch
36
C

Clinfera

AI inference chat session from custom data using OpenCL and NOT CUDA!

Launch
37
J

jetson-inference-mem-tune

Pick the serving stack and per-runtime memory flags

Get
38
T

The OSS LLMOps Stack

Modular, open source LLMOps stack that separates concerns: LiteLLM unifies…

Launch
39
T

Textgen

Open-source desktop app for local LLMs

Launch
40
L

Llamactl

Unified management and routing for llama.cpp, MLX and vLLM models

Launch
41
R

Routerly.ai

Self-hosted LLM gateway that routes requests across AI providers

Launch
42
L

Local Llms On Android

Run local LLMs like Gemma, Qwen, and LLaMA on Android for offline, private

Launch
43
L

Llama3.java

Llama 3+ inference in pure Java

Launch
44
A

AgentOp

Run a real LLM in your browser. No install, no server

Launch
45
V

VoidLLM

Privacy-first LLM proxy and AI gateway

Launch
46
T

TheWhisper

Optimized Whisper models for streaming and on-device use

Launch
47
G

GoModel

AI gateway written in Go with a unified OpenAI-compatible API

Launch
48
L

LLM X

LLMX; Easiest 3rd party Local LLM UI for the web!

Launch
49
J

jetson-llm-serve

Stand up vLLM or SGLang serving on Jetson

Get
50
C

ChatLLM-Web

Private local model studio, AI chat

Launch

About vLLM

V

vLLM

Inference and serving engine for large language models, built

vLLM is an inference and serving engine for large language models, built for high throughput and efficient use of memory. It runs open models on your own hardware and exposes them through a drop-in OpenAI-compatible API, so applications that already call that API can switch to a model you serve yourself. Its main techniques are PagedAttention, which manages the attention memory in pages, and advanced scheduling with continuous batching, which keeps the GPU busy by adding new requests while others are still running. The site describes it as easy to deploy across a wide range of open-source models and hardware. Hardware support is listed for NVIDIA CUDA GPUs, AMD ROCm GPUs, Huawei Ascend NPUs, AWS Neuron, Google Cloud TPUs, Intel Gaudi and XPU, CPUs, and Apple Silicon, among others. The quick start lets you pick a stable or nightly build, a platform such as CUDA, ROCm, XPU or CPU, and a package route, then run an install command. Python 3.10 or newer is required, with 3.12 or newer recommended, and uv is suggested for a faster install. vLLM is a community project supported by donations and by compute from several companies. The site also links to docs, recipes, a performance page and a roadmap.

Category
AI
Website
vllm.ai