Accelerated Inference for Large Transformer Models Using FasterTransformer and Triton Inference Server

jwitsoe · August 3, 2022, 5:00pm

Originally published at: Accelerated Inference for Large Transformer Models Using NVIDIA Triton Inference Server | NVIDIA Technical Blog

Learn about FasterTransformer, one of the fastest libraries for distributed inference of transformers of any size, including benefits of using the library.

j.monnet · August 10, 2023, 1:24pm

Which NVIDIA hardware ressources do I need to deploy ChatGPT-J

Topic		Replies	Views
Deploying GPT-J and T5 with FasterTransformer and Triton Inference Server Technical Blog	7	1000	April 19, 2023
Deploying AI Deep Learning Models with NVIDIA Triton Inference Server Technical Blog	0	392	December 18, 2020
Fast and Scalable AI Model Deployment with NVIDIA Triton Inference Server Technical Blog	0	414	November 9, 2021
Deploying Models from TensorFlow Model Zoo Using NVIDIA DeepStream and NVIDIA Triton Inference Server DeepStream SDK	3	8906	February 29, 2024
Deploying Models from TensorFlow Model Zoo Using NVIDIA DeepStream and NVIDIA Triton Inference Server Technical Blog	13	1178	May 25, 2022
Tensorflow inference using TRT converted model TensorRT	10	1054	May 25, 2021
Accelerate Generative AI Inference Performance with NVIDIA TensorRT Model Optimizer, Now Publicly Available Technical Blog	4	272	July 16, 2024
Optimizing Inference on Large Language Models with NVIDIA TensorRT-LLM, Now Publicly Available Technical Blog	8	1659	January 25, 2024
Demystifying AI Inference Deployments for Trillion Parameter Large Language Models Technical Blog	2	166	July 11, 2024
Scaling LLMs with NVIDIA Triton and NVIDIA TensorRT-LLM Using Kubernetes Technical Blog llama	1	25	October 22, 2024