nvvp not showing timeline

Hi,

when I use nvvp to profile my OpenACC application (PGI 15.7-15.10) it does not produce the typical timeline even though it is able to generate the required “raw data” (pressing the Examine GPU Usage button and looking at the Details tab I get a table with all kernel launches and timing data etc.). Initially I thought it was because my program still produces a memory error in the final destruction phase and nvvp warns about it. But when I put an exit(0) before that the program terminates normally and nvvp does not complain. However, still no timeline.
I have tried with nvvp from cuda 5.5 and 6.6 and PGI compiler version 15.7,15.9, and 15.10.
Is there any known incompatibility or am I doing something wrong?

Thanks,
LS

Hi LS,

I use NVVP all the time with OpenACC and not had any issues generating a timeline so I’m not sure what’s wrong. Let me ask our profiler folks if they might have ideas.

What are the step your doing to produce the profile? Are you running directly under NVVP or generating the profile via NVprof? If nvprof, are you using the “-o” flag?

  • Mat

Hi Mat,

I am running directly under NVVP. I haven’t tried creating a profile with nvprof from the command line and then loading it into NVVP for displaying the timeline. But I guess that’s another option I could try even though one would think this should not make a difference.

Thanks,
LS

Hi LS,

The comments back are that CUDA 5.5 is old so they ask if you can try CUDA 7.5. CUDA 6.5 should be fine since I’ve used it myself.

Also, they want to make sure you selected ‘Run application to generate GPU timeline’.

The other possibility is that the code doesn’t have any GPU code in it. When you run from the command line with the environment variable PGI_ACC_NOTIFY=1 enabled, do you get any output indicating that kernels are being executed?

  • MAt

Hi Mat,

I have been using NVVP from CUDA 6.5 installation (haven’t installed CUDA 7.5 yet). Or are you referring to the compilation process? I only specify -ta=tesla,host so according to the manual I would expect that it is using CUDA 6.5, right?
I have used PGI_ACC_NOTIFY a lot (it is very useful) and I am seeing all my kernels being launched. In the DETAILS tab of NVVP I also see all the details about each kernel launch, but it just doesn’t show the timeline …

In the meantime I was also able to fix the memory corruption error, but it doesn’t affect my timeline problem.

Thanks,
LS

Hi Mat

I have further investigated this issue and was wondering whether you have any idea where else I could look at.

I copied the binary to a host in another office location, used nvvp from the same cuda toolkit (6.5) and there I can see the timeline. Already just after I hit the button to generate the timeline, the time axis gets displayed while the program is still running.

Since my PGI license is bound to the machine where it doesn’t work, it gets very inconvenient always having to transfer the binary across the Atlantic for profiling. Besides I don’t have access to the same kind of GPUs at each site.
So, is there anything I could check for in the environment (e.g. JAVA_HOME), user settings etc. that might affect the behavior of nvvp that I am observing? Does nvvp write any logfiles that might mention why no timeline can be generated?

Thanks,
LS

Hi LS,

My best guess is that your CUDA driver is older on the local machine and why it works on the system across the Atlantic. What driver versions are installed on each system?

As for diagnostics, I asked on of PGI profiler engineers who works with the NVVP folks. He didn’t think there is any diagnostic logging, at least he didn’t know of any. I can poke the NVVP folks if necessary, but lets check on the driver versions first.

  • Mat

Hi Mat

you are right in that the driver is older on the machine I am seeing the timeline problem than on the one it works, however, it differs only in the minor version number:
driver version 340.65 (on machine where timeline works, K20c installed)
driver version 340.29 (on machine where timeline is not displayed, K20X installed).

I also checked the latest driver versions available and found the following:
340.93 (recommended for Cuda Toolkit 6.5)
352.63 (recommended for Cuda Toolkit 7.5)
If I want to have the option to use Cuda Toolkit version 6.5,7.0, and 7.5, which driver version should be installed on the machine? Any recommendations?

Thanks,
LS

Any recommendations?

I’d recommend installing the latest available driver.

Hi Matt

I upgraded to the latest Nvidia driver and it still didn’t work. Then I decided to rm -rf the directories .nv and nvvp_workspace in my home directory. After that nvvp displayed the timeline again :-)
So some settings/state must have been messed up. But I did not see any setting in the nvvp GUI that is supposed to control the timeline behavior. Did I overlook something?

Thanks,
LS

Ok, my assumption then is that you minimized the timeline window. In that case, you need to click the “restore” icon in the left-most tool bar.