Nsight trace cuda failure on orin

Autonomous Vehicles DRIVE AGX Orin DRIVE AGX Orin General
1

i want to run nsys to tracec inference performance on orin,my nsight system version is NVIDIA Nsight Systems version 2022.3.4.34-133b775.

my cmd is
nsys profile --trace=nvtx,cublas,cudla,cusparse,cudnn,nvmedia --output=model_gn.nvvp ./trtexec --loadEngine=ins2.10_2m_bs1_dla.engine --iterations=10 --idleTime=500 --duration=0

but it occured error :
Generating ‘/tmp/nsys-report-86aa.qdstrm’
FATAL ERROR: /build/agent/work/323cb361ab84164c/QuadD/Common/GpuTraits/Src/GpuTicksConverter.cpp(376): Throw in function QuadDCommon::TimestampType GpuTraits::GpuTicksConverter::ConvertToCpuTime(const QuadDCommon::Uuid&, uint64_t&) const
Dynamic exception type: boost::wrapexceptQuadDCommon::NotFoundException
std::exception::what: NotFoundException
[QuadDCommon::tag_message*] = No GPU associated to the given UUID

but when i don’t trace cuda,it work noramlly,i have no idea to solve it.

2

Dear @deweichen,
Is it DLA model? May I know what are the params used to generate trt engine?
So it worked with trace=nvtx,nvmedia?
Also, may I know if you have tried nsys GUI?

4

hello @SivaRamaKrishnaNV ,
is it DLA model, the cmd i generate trt engine is ./trtexec --onnx=…/…/model/ins2.10_2m_bs1.onnx --int8 --useDLACore=0 --allowGPUFallback --saveEngine=ins2.10_2m_bs1_dla.engine --separateProfileRun
yes it work normally with --trace=nvtx,cublas,cudla,cusparse,cudnn,nvmedia
i haven’t tried nsys GUI yet,just tried nsys CLI

5

Dear @deweichen,
Could you share the model to reproduce it on our end.
BTW did you get chance to test via nsys GUI?

6

hello, I couldn’t support model, i think u can reproduce it on your end thorw any model on orin.
i also have tried via nsys GUI,it also didn’t work.

7

@deweichen please see if you can narrow down the issue caused by any parameters. I found cudla isn’t in 1.3.1. CLI Profile Command Switch Options.

8

I know it’s been quite a while since this problem was reported. The problem is that Nsight Systems couldn’t trace CUDA when running as non-root on Jetson Orin. It should work fine when running as root.

The next release of JetPack will include a version of Nsight Systems (lilely 2022.5.2) that can trace CUDA when running as non-root.

10

Dear @deweichen ,
Did you check list of supported parameters as vick suggested earlier. Does removing cudla helped?

12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.