Recently, I tried to profile a multi-process application using Nsight Systems within a singularity container. The command I used within the container is “nsys profile -o SOME_DIR APP_COMMAND APP_ARGS”. I am very sure that CUDA kernels were launched in the application, as I monitored GPU usage from nvidia-smi. However, GPU behavior was not captured in the report. On the other hand, there was no problem profiling single-process applications within the singular container. Further, from a discussion with colleagues, I realized that the same multi-process application was profiled successfully in a docker container using Nsight Systems. So it seems that singularity is the reason. I have attached the failed profile report in this post. Can anyone point out how to generate successful reports in a singularity container?
result_with_smaq.qdrep (2.0 MB)