I ran a tensorflow program, generated a .nvvp file by -o flag, but when i try to import it in the Visual Profiler, it gives me an error of “unsupported file type”
2 Likes
I guess you used nvprof -o to generate an output file, right? It’s not a nvvp but pdm file, anyway if you generated the output from nvprof, you should use nvvp->import file->nvprof to import it, if it’s a nvvp file, you just need to use nvvp to open it.
Yes, I created the file with the -o flag, when i try to import it into nvvp like you mentioned i get the error i mentioned earlier.
What about if you do not use -o and just see what will print in the console ? Any error happens ?
I am able to successfully print out the trace using --log-file , and even pass a csv argument to parse it later in Python, but given that nvvp already exists, and it provides a very intuitive GUI, i would still like to use that.
Can you share the command you used for profile and the file you generated ?
We can check in our side to see what’s wrong with the file.
I can’t share the log file generated, is there another way to debug?
Now i can’t even export using -o, I get the following error: Failed to create CUPTI subcriber, same as here //devtalk.nvidia.com/default/topic/1038689/visual-profiler/failed-to-create-cupti-subcriber-/post/5277826/#5277826
Hi, dhingratul
I will check with dev about the Failed to create CUPTI subcriber error.
I think the import issue may also caused by this.
For the “Failed to create CUPTI subscriber error”, here is the explanation from dev:
It’s not allowed to have multiple GPU profilers operating at the same time. Visual Profiler and nvprof are not intended to be used with applications that make use of the CUPTI.
This is documented in the profiler documentation but this is covered under events and metrics limitations:
Here are a couple of reasons why Visual Profiler may fail to gather metric or event information. More than one tool is trying to access the GPU. To fix this issue please make sure only one tool is using the GPU at any given point. Tools include the CUDA command line profiler, Parallel NSight Analysis Tools and Graphics Tools, and applications that use either CUPTI or PerfKit API (NVPM) to read event values.
I am using tensorflow, which according to the @harryz_, uses CUPTI. I am looking for a workaround to fix this. Also, I am able to get the profile a tensorflow code in one virtual environment but fails to do so in another one. The one that fails, preceded by the following log statement
"2018-08-30 09:23:50.653258: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcupti.so.9.0 locally
"
However, for the virtual env that i am able to run the profiler, this message doesn’t appear.
@harryz i, I tried following your first answer, and modified the extension of my file to pdm (or .nvprof as suggested here).
When importing via “Import > Nvprof > Single process > Timeline data file”, I get the following error: Nvprof Import Failed
Unable to import nvprof generated profile data.
You are trying to import unsupported data. Either use updated Visual Profiler or generate data using compatible version.
I am generating the profile file on my server with cuda 10.1 and using nvvp on my local machine which also has the cuda toolkit 10.1 installed (but no drivers since I don’t have a GPU locally).
I installed nvvp via sudo apt-get install -y nvidia-visual-profiler. The version is 7.5.
However, I don’t know where I can find a compatibility matrix to install the correct version of nvvp matching my server’s nvprof. And I don’t even know if versions are really the problem, nor how to install an earlier version of nvvp.
Actually, I just managed to import the profile with nsight instead of nvprof. So I should be fine.