An ERROR from nsight-system 2023.4.3.13-234333569073v0 of Jetpack 6.0 DP

[ningreka]

I try to profile the trtexec with nsys on the jetson orin nano, but I got this error.

FATAL ERROR: /dvs/p4/build/sw/devtools/Agora/Rel/QuadD_Tegra/QuadD/Target/quadd_d/quadd_d/jni/EventSource/Trace.cpp(1853): Throw in function QuadDCommon::Uuid QuadDDaemon::EventSource::Trace::GetCudaDeviceUuidForTimestamp(QuadDCommon::ProcessId, const ConstTraceEvent&, uint64_t) const
Dynamic exception type: boost::wrapexcept<QuadDCommon::InternalErrorException>
std::exception::what: InternalErrorException
[QuadDCommon::tag_message*] = Can't find UUID for CUDA device 0 (PID 2101)

I found the SD Card image doesn’t contain the nsys, so I installed it with the sdk-manager.

I just use the following command to profile.

sudo nsys profile /usr/src/tensorrt/bin/trtexec --loadEngine=xxx.engine

I checked some similar error reported on forums, such as Unable to capture “Can’t find UUID for CUDA device”

Is the error caused by version incompatible between nsys and the driver in the new jetpack?

[AastaLLL]

Hi,

We can reproduce the same issue on JetPack 6.0 DP and now are working on this issue.

Please note that the Nsight System can work correctly on JetPack 5.1.2.
You can downgrade the software version in case you need the profiling result now.

Thanks.

[AastaLLL]

Hi,

On JetPack 6DP, adding a delay can also avoid this issue.
For example:

$ /usr/src/tensorrt/bin/trtexec --onnx=/usr/src/tensorrt/data/mnist/mnist.onnx --saveEngine=mnist.trt
$ sudo nsys profile --delay=3 /usr/src/tensorrt/bin/trtexec --loadEngine=mnist.trt --iterations=1000

Thanks.

[ningreka]

Hi,

Thank you for your solution, that solved pretty much all of my problem.

[AastaLLL]

Hi,

We found the cause and is fixed in our next release.
For now, please try the below WAR:

1. Open the configuration file whose path can be found through the nsys -z command
2. Add “CuptiUsePerThreadBuffer=false”

Thanks.