Why does Throughput improve when profiling my TensorRT model inference performance using ncu

Developer Tools Nsight Compute
1

When I use the command “/usr/local/TensorRT_8.6.1/bin/trtexec --loadEngine=kep_deblur_slow_720x1280_fp16.trt --avgRuns=1000 --staticPlugins=/usr/local/TensorRT_8.6.1/plugin/libnvinfer_plugin.so --streams=8 --useCudaGraph”
the output is:
[07/03/2024-17:14:06] [I] Throughput: 7.13357 qps
[07/03/2024-17:14:06] [I] Latency: min = 147.254 ms, max = 1021.66 ms, mean = 843.216 ms, median = 1014.76 ms, percentile(90%) = 1020.67 ms, percentile(95%) = 1021.03 ms, percentile(99%) = 1021.66 ms

And when I use the command “ncu /usr/local/TensorRT_8.6.1/bin/trtexec --loadEngine=kep_deblur_slow_720x1280_fp16.trt --avgRuns=1000 --staticPlugins=/usr/local/TensorRT_8.6.1/plugin/libnvinfer_plugin.so --streams=8 --useCudaGraph”
the output is:
[07/03/2024-17:08:38] [I] Throughput: 12.2562 qps
[07/03/2024-17:08:38] [I] Latency: min = 82.3564 ms, max = 563.388 ms, mean = 409.962 ms, median = 443.677 ms, percentile(90%) = 481.955 ms, percentile(95%) = 549.321 ms, percentile(99%) = 563.388 ms

Why is the Throughput increased from 7.13357 qps to 12.2562 qps when running ncu? Is the Throughput output inaccurate when using ncu?

2

Hi, @919898756

When using either kernel or application replay, individual kernels are serialized. You can select range or app-range replay modes.

Refer 2. Kernel Profiling Guide — NsightCompute 12.5 documentation

In Range Replay , all requested metrics in NVIDIA Nsight Compute are grouped into one or more passes. In contrast to Kernel Replay and Application Replay, Range Replay captures and replays complete ranges of CUDA API calls and kernel launches within the profiled application. Metrics are then not associated with individual kernels but with the entire range. This allows the tool to execute kernels without serialization and thereby supports profiling kernels that should be run concurrently for correctness or performance reasons.

3

Sorry, I don’t understand what you mean. Is there something wrong with the way I use ncu? Is it very promising that I can increase the throughput of my model to 12qps.

4

You can use ‘ncu --replay-mode range’ to profile.