In tensorrt 10.8 release note, Release Notes — NVIDIA TensorRT Documentation . In Limitations, there is a description [image] but the subsequent Best Practices section contains the following description: [image] It seems that these two descriptions are conflicting. So, does TensorRT 10.8 supp…

Does tensorrt10.8 support pointwise op between matmul and softmax when using fused mha

AI & Data Science Deep Learning (Training & Inference) TensorRT

lijinghaiwhu February 7, 2025, 3:24am 1

In tensorrt 10.8 release note, Release Notes — NVIDIA TensorRT Documentation.
In Limitations, there is a description

but the subsequent Best Practices section contains the following description:

It seems that these two descriptions are conflicting. So, does TensorRT 10.8 support inserting pointwise operations between matmul and softmax when using fp8?

Topic		Replies	Views
igpu (Tegra) on px2 support for tensorrt int8 (Tensorrt 3.0.2） General	4	752	April 25, 2018
TensorRT mix precision with supported hard drives TensorRT tensorrt	3	626	April 19, 2021
It seems Pow operator in tensorrt reduce the precision Automatically TensorRT	2	547	May 27, 2022
A typo in TensorRT 8.5.3 release notes TensorRT	1	338	February 1, 2023
Is there any layer that fp16 supports but int8 does not？ TensorRT	5	494	December 1, 2021
Int8 callibration with point cloud TensorRT	1	414	February 27, 2023
Quantization in TensorRt Jetson Nano tensorrt	6	1932	March 2, 2022
Analyzing sampleInt8 accuracy TensorRT	2	1005	June 5, 2019
FP16 --half=true option doesn't work on GTX 1080 TI although it runs ./sample_int8 INT8 GPU-Accelerated Libraries	2	4948	August 23, 2017
Different between TensorRT and Pytouch Cuda use (mixed precision) TensorRT tensorrt , cuda	2	399	December 1, 2022

Does tensorrt10.8 support pointwise op between matmul and softmax when using fused mha

Related topics