OS X - cuda-gdb crashes computer when breaking at breakpoint in kernel

Accelerated Computing CUDA CUDA Setup and Installation
1

Hi all,

I have a MacBook Pro running OS X 10.10.2 and an NVIDIA GeForce GT 750M 2048 MB with CUDA 7.0.

I’m currently having difficulties debugging my CUDA program. I set a breakpoint inside a kernel to debug it, and whenever I run cuda-gdb and try to break in the kernel, my entire laptop freezes and crashes eventually. Before this happens however, I get several errors:

warning: Could not open OSO archive file “/BinaryCache/corecrypto/corecrypto-233.1.2~26/Symbols/BuiltProducts/libcorecrypto_static.a”
warning: `/BinaryCache/coreTLS/coreTLS-35.10.1~5/Objects/coretls.build/coretls.build/Objects-normal/x86_64/system_coretls_vers.o’: can’t open to read symbols: No such file or directory.
warning: Could not open OSO archive file “/BinaryCache/coreTLS/coreTLS-35.10.1~5/Symbols/BuiltProducts/libcoretls_ciphersuites.a”
warning: Could not open OSO archive file “/BinaryCache/coreTLS/coreTLS-35.10.1~5/Symbols/BuiltProducts/libcoretls_handshake.a”
warning: Could not open OSO archive file “/BinaryCache/coreTLS/coreTLS-35.10.1~5/Symbols/BuiltProducts/libcoretls_record.a”
warning: Could not open OSO archive file “/BinaryCache/coreTLS/coreTLS-35.10.1~5/Symbols/BuiltProducts/libcoretls_stream_parser.a”

I get similar results when trying to debug the program in NSight Eclipse Edition. The CUDA program runs fine if I run it without breakpoint, however I get a series of warnings as well, similar to above:

warning: /BinaryCache/cmph/cmph-1~1091/Objects/cmph.build/cmph.build/Objects-normal/x86_64/bdz.o': can't open to read symbols: No such file or directory. warning: /BinaryCache/cmph/cmph-1~1091/Objects/cmph.build/cmph.build/Objects-normal/x86_64/bdz_ph.o’: can’t open to read symbols: No such file or directory.
warning: /BinaryCache/cmph/cmph-1~1091/Objects/cmph.build/cmph.build/Objects-normal/x86_64/bmz.o': can't open to read symbols: No such file or directory. warning: /BinaryCache/cmph/cmph-1~1091/Objects/cmph.build/cmph.build/Objects-normal/x86_64/bmz8.o’: can’t open to read symbols: No such file or directory.
warning: /BinaryCache/cmph/cmph-1~1091/Objects/cmph.build/cmph.build/Objects-normal/x86_64/brz.o': can't open to read symbols: No such file or directory. warning: /BinaryCache/cmph/cmph-1~1091/Objects/cmph.build/cmph.build/Objects-normal/x86_64/buffer_entry.o’: can’t open to read symbols: No such file or directory.
warning: /BinaryCache/cmph/cmph-1~1091/Objects/cmph.build/cmph.build/Objects-normal/x86_64/buffer_manager.o': can't open to read symbols: No such file or directory. warning: /BinaryCache/cmph/cmph-1~1091/Objects/cmph.build/cmph.build/Objects-normal/x86_64/chd.o’: can’t open to read symbols: No such file or directory.
warning: /BinaryCache/cmph/cmph-1~1091/Objects/cmph.build/cmph.build/Objects-normal/x86_64/chd_ph.o': can't open to read symbols: No such file or directory. warning: /BinaryCache/cmph/cmph-1~1091/Objects/cmph.build/cmph.build/Objects-normal/x86_64/chm.o’: can’t open to read symbols: No such file or directory.
warning: /BinaryCache/cmph/cmph-1~1091/Objects/cmph.build/cmph.build/Objects-normal/x86_64/cmph.o': can't open to read symbols: No such file or directory. warning: /BinaryCache/cmph/cmph-1~1091/Objects/cmph.build/cmph.build/Objects-normal/x86_64/cmph_structs.o’: can’t open to read symbols: No such file or directory.
warning: /BinaryCache/cmph/cmph-1~1091/Objects/cmph.build/cmph.build/Objects-normal/x86_64/compressed_rank.o': can't open to read symbols: No such file or directory. warning: /BinaryCache/cmph/cmph-1~1091/Objects/cmph.build/cmph.build/Objects-normal/x86_64/compressed_seq.o’: can’t open to read symbols: No such file or directory.
warning: /BinaryCache/cmph/cmph-1~1091/Objects/cmph.build/cmph.build/Objects-normal/x86_64/fch.o': can't open to read symbols: No such file or directory. warning: /BinaryCache/cmph/cmph-1~1091/Objects/cmph.build/cmph.build/Objects-normal/x86_64/fch_buckets.o’: can’t open to read symbols: No such file or directory.
warning: /BinaryCache/cmph/cmph-1~1091/Objects/cmph.build/cmph.build/Objects-normal/x86_64/graph.o': can't open to read symbols: No such file or directory. warning: /BinaryCache/cmph/cmph-1~1091/Objects/cmph.build/cmph.build/Objects-normal/x86_64/hash.o’: can’t open to read symbols: No such file or directory.
warning: /BinaryCache/cmph/cmph-1~1091/Objects/cmph.build/cmph.build/Objects-normal/x86_64/jenkins_hash.o': can't open to read symbols: No such file or directory. warning: /BinaryCache/cmph/cmph-1~1091/Objects/cmph.build/cmph.build/Objects-normal/x86_64/linear_string_map.o’: can’t open to read symbols: No such file or directory.
warning: /BinaryCache/cmph/cmph-1~1091/Objects/cmph.build/cmph.build/Objects-normal/x86_64/miller_rabin.o': can't open to read symbols: No such file or directory. warning: /BinaryCache/cmph/cmph-1~1091/Objects/cmph.build/cmph.build/Objects-normal/x86_64/select.o’: can’t open to read symbols: No such file or directory.
warning: /BinaryCache/cmph/cmph-1~1091/Objects/cmph.build/cmph.build/Objects-normal/x86_64/vqueue.o': can't open to read symbols: No such file or directory. warning: /BinaryCache/cmph/cmph-1~1091/Objects/cmph.build/cmph.build/Objects-normal/x86_64/vstack.o’: can’t open to read symbols: No such file or directory.
warning: Could not open OSO archive file “/Volumes/Builds/nv/build/rel/gpu_drv/r346/r346_00/drivers/gpgpu/bin/310.41.15_mercury_x86_64_Darwin_release/libNVDAresmanAPI.a”

I can also run the CUDA samples which were prompted during the tutorial of setting up CUDA for Mac without any errors.

Does anybody know what is going on or how I can fix this issue?

Regards,
BananaCode

2

I suspect that preemption is not enabled, and therefore when you set a breakpoint in kernel code, the display driver halts, causing the laptop to appear to freeze, and crash eventually.

from here:

[url]https://docs.nvidia.com/cuda/nsight-eclipse-edition-getting-started-guide/index.html#platform-requirements[/url]

"A GPU that is running X11 (on Linux) or Aqua (on Mac) cannot be used to debug a CUDA application and will be hidden from the application ran in the debugger. Such GPU can still be used for profiling GPU applications. "

On CC 3.5 and higher GPUs (yours seems to be cc3.0) there is a preemption setting that may make a difference, but I’m not sure it’s supported on Mac OS and anyway I don’t think it will work with your cc3.0 GPU.

The recommended approach in such cases is to debug on a separate non-display GPU. I recognize this is probably not possible on a laptop. It’s not a perfect world.

3

This laptop has two GPUs, the second one being Intel Iris Pro, on which Aqua runs - or so I think. How do I enable pre-emption?

4

Preemption is not supported on cc3.0 GPUs. And, although the laptop has 2 GPUs, I suspect that both are involved in running the display.

At least on linux, preemption can be enabled either with an environment variable setting or via a preferences setting within nsight EE or cuda-gdb.

From the cuda-gdb manual:

[url]http://docs.nvidia.com/cuda/cuda-gdb/#single-gpu-debugging-with-desktop-manager-running[/url]