Profile command cannot be used more than once with the same agent

Developer Tools Nsight Systems Profiling Linux Targets
1

I want to profile the process of my CNN model inference, But I got this info:

Dynamic exception type: std::runtime_error
std::exception::what: Profile command cannot be used more than once with the same agent

even when I use a very simple code in test.py file like:

print(1+1)

And I use the command “nsys profile python test.py”, the info is still the same

2

If I write a simple C file ‘test.c’ below and then generate a binary ‘test.out’

#include<stdio.h>
int main(){
	printf("%s\n", "test");
	return 0;
}

use this command to profile:

nsys profile ./test.out

It has no such runtime_error and runs successfully:

Importing the qdstrm file using /opt/nvidia/nsight-systems/2019.4.2/host-linux-x64/QdstrmImporter.

Importing...

Importing [==================================================100%]
Saving report to file "/home/edge/Workspace/nsys/report1.qdrep"
Report file saved.
Please discard the qdstrm file and use the qdrep file instead.

Removed /home/edge/Workspace/nsys/report1.qdstrm as it was successfully imported.
Please use the qdrep file instead.
3

Is there any problems with python script? I see an example in document like:

nsys profile --trace=cuda,cudnn,cublas,osrt,nvtx --delay=60 python my_dnn_script.py
4

Hi IronRoyGrey,

What version of Nsight Systems are you using? You can determine the version of nsys you are using with the “nsys --version” command.

I used the latest version, “NVIDIA Nsight Systems version 2019.6.1.101-8582829”, shipped in mid-December and available at Gameworks Download Center | NVIDIA Developer. With the latest version, I could not reproduce your issue using the test.py script you provided.

The command line you provided, “nsys profile --trace=cuda,cudnn,cublas,osrt,nvtx --delay=60 python my_dnn_script.py”, looks fine. It should work.

Bob Knight

5

Hi, I encountered the same problem.

my config:
NVIDIA Nsight Systems version 2019.6.1.101-8582829
Driver Version: 440.33.01 CUDA Version: 10.2
(those marked as gconfig below)

In my case, IronRoyGrey’s test.c report the same problem.
And I even can not reproduce this problem in another machine with the same OS, gconfig and GPUs.

Does anyone have some advice?

6

Did this issue get resolved?

7

Upgrading to NVIDIA Nsight Systems version 2020.3.1.72-e5b8014 resolved this issue for me.