TensorRT-LLM

Overview

TensorRT-LLM is an open-sourced library for optimizing LLM and Visual Gen inference on NVIDIA GPUs. It provides state-of-the-art optimizations including custom kernels for common inference operations such as attention, GEMMs and MoE, algorithmic runtime optimizations such as prefill-decode disaggregation, wide expert parallelism and speculative decoding, and quantization support for FP8 and FP4 models. Architected on PyTorch, it exposes a high-level Python LLM API that supports inference setups from a single GPU to multi-GPU and multi-node deployments with built-in parallelism strategies, and integrates with NVIDIA Dynamo and the Triton Inference Server. The library is released under the Apache 2.0 licence.

Use cases

• Optimize LLM inference on NVIDIA GPUs
• Reduce latency with custom kernels and runtime optimizations
• Deploy multi-GPU and multi-node LLM serving
• Run quantized FP8 and FP4 models

Sources

• TensorRT-LLM repository
• TensorRT-LLM Apache 2.0 licence