Skip to content
V

vLLM

Inference and serving engine for large language models, built

#16in AI
2In stores

Description

vLLM is an inference and serving engine for large language models, built for high throughput and efficient use of memory. It runs open model…

Preview

Highlights

  • OpenAI-compatible API for serving open models
  • PagedAttention and continuous batching for high throughput
  • Runs on NVIDIA, AMD, Intel, TPU, Apple Silicon and CPU
  • Stable and nightly builds with a guided install
  • Requires Python 3.10 or newer, with uv recommended

Ratings & Reviews

No ratings yet

Be the first to rate vLLM.

People also visit

V

vllm-mlx

High-performance OpenAI and Anthropic compatible LLM inference server

AI
Launch
W

Web LLM

High-performance In-browser LLM Inference Engine

AI
Launch
L

llama.cpp

C/C++ inference engine for large language models, supporting quantization

AI
Launch
P

Paddler

Open-source LLM/VLM load balancer and serving platform for self-hosting LLMs

AI
Launch
L

LLMKube

Kubernetes operator for self-hosted LLM inference with pluggable runtimes

AI
Launch
O

Ollama

Load and run large LLMs locally to use in your terminal or build your apps

AI
Launch

vLLM FAQ

What is vLLM for?

Serving large language models with high throughput and efficient memory use, through an OpenAI-compatible API.

Which hardware does it support?

NVIDIA CUDA GPUs, AMD ROCm GPUs, Huawei Ascend NPUs, AWS Neuron, Google Cloud TPUs, Intel Gaudi and XPU, CPUs and Apple Silicon, according to the site.

How do I install it?

Choose a stable or nightly build and your platform on the quick start, then run the install command. The site recommends uv, and Python 3.12 or newer.

Can my existing code use it?

Yes, if your code calls the OpenAI API. The site describes a drop-in compatible endpoint, so you can point the client at your own server.

Is it free to use?

The site says the project is a community project funded by donations and supported by compute from several companies. The price and license are not shown in the scanned text.

What are the best vLLM alternatives?

Popular alternatives to vLLM include vllm-mlx, Web LLM and llama.cpp. See all vLLM alternatives

Developer Stuff

Badges