Nsight trace cuda failure on orin

i want to run nsys to tracec inference performance on orin,my nsight system version is NVIDIA Nsight Systems version 2022.3.4.34-133b775.

my cmd is
nsys profile --trace=nvtx,cublas,cudla,cusparse,cudnn,nvmedia --output=model_gn.nvvp ./trtexec --loadEngine=ins2.10_2m_bs1_dla.engine --iterations=10 --idleTime=500 --duration=0

but it occured error :
Generating ‘/tmp/nsys-report-86aa.qdstrm’
FATAL ERROR: /build/agent/work/323cb361ab84164c/QuadD/Common/GpuTraits/Src/GpuTicksConverter.cpp(376): Throw in function QuadDCommon::TimestampType GpuTraits::GpuTicksConverter::ConvertToCpuTime(const QuadDCommon::Uuid&, uint64_t&) const
Dynamic exception type: boost::wrapexceptQuadDCommon::NotFoundException
std::exception::what: NotFoundException
[QuadDCommon::tag_message*] = No GPU associated to the given UUID

but when i don’t trace cuda,it work noramlly,i have no idea to solve it.

Dear @deweichen,
Is it DLA model? May I know what are the params used to generate trt engine?
So it worked with trace=nvtx,nvmedia?
Also, may I know if you have tried nsys GUI?

hello @SivaRamaKrishnaNV ,
is it DLA model, the cmd i generate trt engine is ./trtexec --onnx=…/…/model/ins2.10_2m_bs1.onnx --int8 --useDLACore=0 --allowGPUFallback --saveEngine=ins2.10_2m_bs1_dla.engine --separateProfileRun
yes it work normally with --trace=nvtx,cublas,cudla,cusparse,cudnn,nvmedia
i haven’t tried nsys GUI yet,just tried nsys CLI

Dear @deweichen,
Could you share the model to reproduce it on our end.
BTW did you get chance to test via nsys GUI?

hello, I couldn’t support model, i think u can reproduce it on your end thorw any model on orin.
i also have tried via nsys GUI,it also didn’t work.

@deweichen please see if you can narrow down the issue caused by any parameters. I found cudla isn’t in 1.3.1. CLI Profile Command Switch Options.

I know it’s been quite a while since this problem was reported. The problem is that Nsight Systems couldn’t trace CUDA when running as non-root on Jetson Orin. It should work fine when running as root.

The next release of JetPack will include a version of Nsight Systems (lilely 2022.5.2) that can trace CUDA when running as non-root.

Dear @deweichen ,
Did you check list of supported parameters as vick suggested earlier. Does removing cudla helped?