vLLM

Overview

vLLM is an open-source library for LLM inference and serving. It uses PagedAttention for KV-cache memory management, continuous batching, chunked prefill and prefix caching, and supports quantization including FP8, INT8, INT4, GPTQ and AWQ. It also offers speculative decoding and an OpenAI-compatible API server. The project supports 200+ model architectures and runs on NVIDIA, AMD and Intel GPUs as well as x86, ARM and PowerPC CPUs.

Use cases

• Serve LLMs with high throughput
• Reduce inference memory use
• Run OpenAI-compatible API endpoints
• Deploy models on GPUs and CPUs

Sources

• vLLM README
• vLLM Apache 2.0 licence