Error in reading profiler output

Accelerated Computing CUDA CUDA Programming and Performance
1

Hi,

I’m quite new to CUDA and “brandnew” to CUDA Visual Profiler (both version 2.2).

I have a problem profiling a CUDA enabled progam with cudaprof. The application works fine, with and without CUDA. Unfortunately the error is rather imprecise.
After I ran the application using cudaprof, I get a messagebox saying “Error in reading profiler output.” without any further information.

Profiling the examples coming with the SDK works fine.

Do I have to make the application profiling-capable in some way? Could there be anything wrong with folders or env variables? Is there a profiler log or can I debug the problem somehow?

I also tried to set CUDA_PROFILE=1. Do I have to recompile the application afterwards?

I’m using cudaprof on a 64-bit CentOS system (Red Hat 4.1.2) and a Geforce GTS 8800.

Greetings,
Tobias

2

Nobody out there with an idea about this? :unsure:

3

Not sure if it relates, but make sure that the GPU is done calculating before the profiler stops collecting information.

I’ve often seen that in that case the profiler will not show the results or have partial results.

eyal

4

Thank you for the answer.

You mean setting “Max. Execution Time” higher than the actual computing time? That is definately given and unfortunately doesn’t solve the problem.

Any other ideas? Is there a way to find out more about the error message? Debug it in some way?

Cheers

5

I had a similar problem. For me, the problem was caused by accidentally calling cudaThreadSynchronize() after the final cudaThreadExit(). I guess this error could also arise if you don’t use cudaThreadExit() at all.

Greetings,
Mattes

6

Hi tobsen,

unfortunately I’ve got the same problem as you have.

I use cuda 2.2 with fedora 11 for 32-bits architecture.

Sorry Mattes, but your solution doesn’t work for me.

Btw. CUDA reference manual says: cudaThreadExit() is implicitly called on host thread exit.

what do you mean by this ?

Is cudaEventSynchroniz / cudaThreadSynchronize not enough ?

Greg

7

Hi again,

I worked it out, at least it works for me now.

The problem was, I didn’t set up the environment properly.

Without that you can’t even run profiler from command line.

I think it should help in your case tobsen, as well.

Add this ‘export’ command to your profile file, so you don’t need to execute it manually.

Hope it helps you,

Greg

8

I have exactly the same problem on Vista 64. In the environment variables tab, what do I add for the variable, as I was under the impression LD_LIBRARY_path is a linux library?

9

Yes, this is for linux environment. I don’t know exactly how it’s in windows with Profiler. I guess you don’t install it separately, for you won’t have any problems with running it.

Try to look for tips in all files that comes with toolkit, sdk. I found solution in VisualProfilerReleaseNotes.txt.

Greg

10

I have the same problem, even after set the LD_LIBRARY_PATH.
If I set the “run in separate window”, the cuda code can be run, and generate some csv file in the working directory, but at last, get the error too.

BTW: Ubuntu 9.10 64bit, cuda 3.0

11

For those still seeing:

Program run #1 completed.
Empty header line found in CSV file.
Error in reading profiler output.

Something to try is redirecting output to a file rather than to stdout or stderr.

For example, in mac osx, I use a script file run.sh that contains these 5 lines:

#!/bin/bash
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib/
export CUDA_PROFILE=1
export CUDA_PROFILE_CSV=1
/Users/moi/my_program >& out

without the “>& out”, I get the above error; with it, all runs fine.

12

I had the same problem running on windows 7 64 bit. For me the solution was to change launch path (under settings) to: “C:/Program Files/MATLAB/R2009b/bin/win64/MATLAB.exe” instead of previously “C:/Program Files/MATLAB/R2009b/bin/matlab.exe”.

/D

13

I just hit this error. For me the .csv fils contained the header, but no result rows. This indicates that my app did not properly shut down CUDA, which writes the data. It turns out my app has a host thread that runs all my CUDA, and another thread that called exit(). So the CUDA thread just got killed without properly shutting down CUDA. My solution was to shut down that thread before the main thread.

14

It went away for me if I was strict about releasing all my resources before shutting down my app (I’m actually running openCL rather than CUDA, so that was all the programs, kernels, buffers, images and events I was using). There is a note about this in the release notes:

15

I was able to run the profiling for CUDA examples but when I tried on the OpenCL example. I always get the Error in reading profiler output. Only occurring for the Mac Snow Leopard.

Whlle I was able to run the profiling on OpenCL project in ubuntu 9.04. Any suggestion?

16

I was able to run the profiling for CUDA examples but when I tried on the OpenCL example. I always get the Error in reading profiler output. Only occurring for the Mac Snow Leopard.

Whlle I was able to run the profiling on OpenCL project in ubuntu 9.04. Any suggestion?

17

I was able to run the profiling for CUDA examples but when I tried on the OpenCL example. I always get the Error in reading profiler output. Only occurring for the Mac Snow Leopard.

Whlle I was able to run the profiling on OpenCL project in ubuntu 9.04. Any suggestion?