Description
vLLM is an inference and serving engine for large language models, built for high throughput and efficient use of memory. It runs open models on your own hardware and exposes them through a drop-in OpenAI-compatible API, so applications that already call that API can switch to a model you serve yourself. Its main techniques are PagedAttention, which manages the attention memory in pages, and advanced scheduling with continuous batching, which keeps the GPU busy by adding new requests while others are still running. The site describes it as easy to deploy across a wide range of open-source models and hardware. Hardware support is listed for NVIDIA CUDA GPUs, AMD ROCm GPUs, Huawei Ascend NPUs, AWS Neuron, Google Cloud TPUs, Intel Gaudi and XPU, CPUs, and Apple Silicon, among others. The quick start lets you pick a stable or nightly build, a platform such as CUDA, ROCm, XPU or CPU, and a package route, then run an install command. Python 3.10 or newer is required, with 3.12 or newer recommended, and uv is suggested for a faster install. vLLM is a community project supported by donations and by compute from several companies. The site also links to docs, recipes, a performance page and a roadmap.…
Preview
Highlights
- OpenAI-compatible API for serving open models
- PagedAttention and continuous batching for high throughput
- Runs on NVIDIA, AMD, Intel, TPU, Apple Silicon and CPU
- Stable and nightly builds with a guided install
- Requires Python 3.10 or newer, with uv recommended
Ratings & Reviews
No ratings yet
Be the first to rate vLLM.
People also visit
vLLM FAQ
What is vLLM for?
Serving large language models with high throughput and efficient memory use, through an OpenAI-compatible API.
Which hardware does it support?
NVIDIA CUDA GPUs, AMD ROCm GPUs, Huawei Ascend NPUs, AWS Neuron, Google Cloud TPUs, Intel Gaudi and XPU, CPUs and Apple Silicon, according to the site.
How do I install it?
Choose a stable or nightly build and your platform on the quick start, then run the install command. The site recommends uv, and Python 3.12 or newer.
Can my existing code use it?
Yes, if your code calls the OpenAI API. The site describes a drop-in compatible endpoint, so you can point the client at your own server.
Is it free to use?
The site says the project is a community project funded by donations and supported by compute from several companies. The price and license are not shown in the scanned text.
What are the best vLLM alternatives?
Popular alternatives to vLLM include vllm-mlx, Web LLM and llama.cpp. See all vLLM alternatives
