Nsys run time error "Unknown API driver activity" profiling Numba with nsys on CUDA 11.3

Developer Tools Nsight Systems Profiling Linux Targets
1

Hi there,

I have CUDA 11.3, the Nsight System in that folder is version 2021.1.3.14.b695ea9.

I am running Numba programs fine. But when I try to profile with nsys I get the following error:

**** Analysis failed with:
Status: TargetProfilingFailed
Props {
Items {
Type: DeviceId
Value: “Local (CLI)”
}
}
Error {
Type: RuntimeError
SubError {
Type: ProcessEventsError
Props {
Items {
Type: ErrorText
Value: “/build/agent/work/20a3cfcd1c25021d/QuadD/Host/Analysis/Modules/TraceProcessEvent.cpp(48): Throw in function const string& {anonymous}::GetCudaCallbackName(bool, uint32_t, const QuadDAnalysis::MoreInjection&)\nDynamic exception type: boost::exception_detail::clone_implQuadDCommon::InvalidArgumentException\nstd::exception::what: InvalidArgumentException\n[QuadDCommon::tag_error_text*] = Unknown API driver activity\n[boost::errinfo_errno_*] = 647, "Unknown error 647"\n”
}
}
}
}

As per Nsys profile error: invalidArgumentException, unknown API driver activity - #3 by hwilper I double-checked I am using the correct nsys. It is the one in the cuda 11.3 folder.

Tried running it on a very simple matmul program and kernel. Same issue.

I can’t imagine it could be a version conflict since it is the nsys installed with that toolkit.

Driver is 510.47.03 for a Tesla M60 on Ubuntu 20.04

Python is 3.10.12 and Numba is 0.56.4

2

Hi John,

Technically Nsight Systems does not officially support Maxwell, so you might have to use the NVVP (nvprof) profiler from the toolkit instead. And 11.3 CTK should work with any driver more recent than r450.

However, I am actually going to recommend that you get the latest Nsys and try again. I don’t know of any reason why it shouldn’t work.