Nsight systems iostream error

[virtual.ramblings]

I’m trying to profile a deep learning workload on Jetson AGX Orin using nsight systems. Towards the end of the run, I see the following error. Please advise.

ERROR: /build/agent/work/323cb361ab84164c/QuadD/Common/ProcessWaiter/Src/POSIX/ProcessWaiterImpl.cpp(318): Throw in function void QuadDCommon::ProcessWaiter::ProcessWaiterImpl::WriteTerminationStatusToFile(const QuadDCommon::ProcessWaiter::TerminationStatus&)
Dynamic exception type: boost::wrapexceptQuadDCommon::WriteStreamException
std::exception::what: WriteStreamException
[QuadDCommon::tag_message*] = Failed to write termination status to file
[QuadDCommon::tag_file_name*] = /home/dream-orin3/nsight-tmp/nsys_termination_status_24054
[QuadDCommon::tag_errno_text*] = Dynamic exception type: std::__ios_failure
std::exception::what: basic_ios::clear: iostream error

[virtual.ramblings]

Any solutions to this?

[kayccc]

Your topic was posted in the wrong category. I am moving this to the Jetson AGX Orin category for visibility.

[AastaLLL]

Hi,

Could you share the command you used with us?

Based on the error log:

Failed to write termination status to file

Did you run the command with sudo? GPU profiling requires root privilege on Jetson.

Thanks.

[virtual.ramblings]

Here is the command I used:
nsys profile --trace=cuda,nvtx,cublas,cudla,cusparse,cudnn,nvmedia --output=.nvvp python3 <workload.py> duration=0

Do CUDA traces require sudo access too? In the past, I have used the same command without sudo but didn’t run into the iostream error. This error started showing up a few days ago.

[AastaLLL]

Hi

Yes, please sudo nsys ... to get the GPU trace for profiling.

Thanks.