llama.cpp

Overview

llama.cpp aims to enable LLM and VLM inference with minimal setup and state-of-the-art performance on a wide range of hardware, both locally and in the cloud. It is a plain C/C++ implementation without dependencies, built on the ggml library, with Apple silicon support through ARM NEON, Accelerate and Metal, AVX/AVX2/AVX512/AMX on x86 and RVV and other extensions on RISC-V. The project provides 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit and 8-bit integer quantization for faster inference and reduced memory use, custom CUDA kernels for NVIDIA GPUs with HIP and MUSA variants for other GPUs, Vulkan and SYCL backends, and CPU+GPU hybrid inference to partially accelerate models larger than total VRAM. Prebuilt binaries, Docker images and an OpenAI-compatible server are available under the MIT licence.

Use cases

• Run LLMs locally on CPU and GPU hardware
• Quantize models to 1.5-8 bit to reduce memory use
• Speed up inference with custom CUDA and Metal kernels
• Use hybrid CPU+GPU inference for models larger than VRAM

Sources

• llama.cpp repository
• llama.cpp MIT licence