Sparse tensor math speedup on Ampere

AI & Data Science Deep Learning (Training & Inference) TensorRT
,
1

Hi I was trying out sparse matrix multiplication with the 2-4 pattern (I.e. atleast 2 zeroes per 4 elements).

For tests I used TensorRT to multiply 2 square matrices of size 2048 and 1024. I observed pretty low gains (<=13%) if any for these sizes.

Is that expected behaviour? Or should I be seeing noticeable difference between dense and sparse kernels?

Tests were run on an Jetson AGX Orin 32GB(at 50W | power mode 3). With Jetpack 5.1.

Thanks.

2

The tests were run by converting torch nnlinear layers to onnx and converting the onnx files with trtexec and by setting --sparsity=force and --sparsity=disable to compare sparse and dense performance respectively.

The onnx files used for the tests are attached here.