TensorRT, TensorRT-LLM and Triton Explained
By Karthick Selvam · Published
Three NVIDIA technologies for fast, scalable AI inference.
TensorRT
An SDK that optimises trained models for inference on NVIDIA GPUs through layer and tensor fusion, kernel auto-tuning and reduced precision (FP16, INT8, FP8 and more). Models are typically imported via ONNX.
TensorRT-LLM
An open-source library focused on large language model inference, with features such as in-flight batching, paged KV caching, quantisation and multi-GPU tensor and pipeline parallelism.
Triton Inference Server
Open-source serving software that hosts models from many frameworks (TensorRT, PyTorch, ONNX Runtime, Python and others) with dynamic batching, concurrent execution, ensembles, HTTP/gRPC APIs and Prometheus metrics. NVIDIA now positions Triton as part of its Dynamo inference platform.
How they combine
Optimise a model with TensorRT or TensorRT-LLM, then serve it with Triton (or package it as a NIM microservice).
Frequently asked questions
Are these tools free?
TensorRT-LLM and Triton are open source; TensorRT is free to download under NVIDIA's licence. Enterprise support is available via NVIDIA AI Enterprise.
Related programmes
- GPU Programming with CUDA
- Accelerated Data Science with NVIDIA RAPIDS
- Building and Serving LLM Apps with NVIDIA NIM & NeMo
Turn this into a team capability
Our corporate programmes cover topics like this hands-on, mapped to your stack.
NVIDIA & GPU Computing guides
What Is CUDA?
CUDA is NVIDIA's parallel computing platform and programming model for running general-purpose code on NVIDIA GPUs.
What Is NVIDIA RAPIDS?
RAPIDS is a suite of open-source libraries from NVIDIA for GPU-accelerated data science and analytics.
What Are NVIDIA NIM and NeMo?
NIM and NeMo are parts of NVIDIA's software stack for generative AI: NIM for serving models, NeMo for building and customising them.