We tried to use GEMM with INT8 (using cuBLAS GEMMEX API), but we met the following issues, In our typical settings, M=768, N=786432, K=128, GEMM with INT8 (volta_sgemm_int8_128x128_nt) is much slower than FP16 (turing_h1688gemm_128x128_ldg8_nt), 21.443ms vs. 8.6957ms. I changed to CUDA version fr…

cuBLAS GEMM INT8 is much slower than FP16 in T4

mnicely June 30, 2021, 4:54pm 3

If you think you’re getting incorrect results, I suggest filing a bug. How to report a bug

Topic		Replies	Views
The best input layout settings in CuBlas GPU-Accelerated Libraries cublas	4	235	August 27, 2024
cuBLAS INT8 tensor core mode vs. FP16 mode GPU-Accelerated Libraries cublas	13	5496	December 5, 2022
New cuBLAS 12.0 Features and Matrix Multiplication Performance on NVIDIA Hopper GPUs Technical Blog	0	528	February 1, 2023
cublasGemmEx doesn't work with INT8 utilizing __dp4a instruction on NVIDIA 1080TI CUDA Programming and Performance	12	3656	September 25, 2017
cuBLAS INT8 tensor core mode vs. FP16 mode GPU-Accelerated Libraries	0	886	February 15, 2019
why the Tesla T4 peak performance test result mismatch with the official doc CUDA Programming and Performance	8	2488	October 19, 2019
Blackwell Integer CUDA Programming and Performance	139	2861	June 26, 2025
Is it correct that my Pascal card is calling Maxwell_gemm kernels through cublas? And if so, why is cublas unusably slow for me? CUDA Programming and Performance	6	949	August 23, 2018
Discrepancy Between Claimed and Actual Sparse INT8 Performance of Tensor Cores on Jetson AGX Orin Jetson AGX Orin tensorrt , performance	15	400	September 11, 2024
cuBLAS convolution does not use Tensor Cores GPU-Accelerated Libraries cublas	6	2237	June 8, 2021