cuda-gdbserver cannot be connected, Unknown register ymm0h requested

Development Tools CUDA Developer Tools CUDA-GDB
#1

How to reproduce:

  1. cuda-gdbserver :2345 some_cuda_app
  2. cuda-gdb
  3. target remote localhost:2345

Then the remote connection will be closed, and the gdbserver will print some error as follows:

Process csr_bfs created; pid = 114385
Listening on port 2345
Remote debugging from host 127.0.0.1
cuda-gdb/7.12/gdb/gdbserver/regcache.c:264: A problem internal to GDBserver has been detected.
Unknown register ymm0h requested

Spec:
OS: Ubuntu 16.04 / Ubuntu 18.04
CUDA toolkits: 9.2
CPU: Intel® Xeon® Gold 6140 CPU / Silver 4108
GPU: Titan V / V100 PCIE 16G

We have several machines with the above specs will face such error.

But, another machine whose specs is:

OS: Ubuntu 16.04
CUDA toolkits: 9.2
CPU: Intel I7 6800k
GPU: 1080ti

will be fine.

So, is this error related to Xeon® Scalable Processors family or volta-based GPU architecture?

Thanks for any comments!

#2

Hi,xiaodai

Sorry we cann’t reproduce this issue in below configuration

OS: Ubuntu 16.04
CUDA toolkits: 9.2
GPU: GV100

I will propose your problem to the dev to see if they have any idea about the error.

#3

Hi, xiaodai

Can you share the source file that can trigger the issue ?

Or if you just want to workaround, may you can “set cuda memcheck off” in cuda-gdb.

#4

Hi veraj,

Thanks for the reply.

When I further explore this issue, I can confirm that it is related to gdb (which means that it is not introduced by cuda-gdb).

As my side, gdb version lower than 8.1 will reproduce this issue on my machine. I highly suspect it is related to the Xeon® Scalable Processors family.

I have tried to build gdb 7.12, 8.0 and the latest 8.11. Only the latest 8.11 do NOT has this issue.

So, please try to reproduce this issue on Xeon® Scalable Processors family. If it can be reproduced, it is appreciated to fix it asap.

#5

I download gdb source file from here: http://ftp.gnu.org/gnu/gdb/

7.12.1: Unknown register ymm0h requested
8.0: Unknown register pkru requested
8.01: Unknown register pkru requested
8.1: good!

It can just be reproduced by a print hello-world c++.

#6

https://sourceware.org/bugzilla/show_bug.cgi?id=22137

The gdb bug report about pkru register is here.

But I am not sure whether ymm0h and pkru is one issue. Since current cuda-gdb bases on gdb 7.12 (and reports error on ymm0h), I am not sure whether this bug is related to what we are discussing.

Thanks.

#7

del

#8

Sorry, I shouldn’t ask you for the source. I’m replying in the wrong topic.

For you problem, I already raised a bug for our developer to ask details

#9

Please also attach my following posts, maybe useful, thanks.

#10

I got the same error:

OS: Ubuntu 16.04
CUDA toolkits: 10.0
GPU: GTX 1080 TI

following the tutorial https://devblogs.nvidia.com/building-cuda-applications-cmake/
and using remote gdb

cuda-gdbserver :5000 test_particles
Process test_particles created; pid = 18774
Listening on port 5000
Remote debugging from host 192.168.10.86
cuda-gdb/7.12/gdb/gdbserver/regcache.c:264: A problem internal to GDBserver has been detected.
Unknown register ymm0h requested
#11

Hi, xiaodai and wumo

We have an internal issue tracking this. Thanks for reporting.
Dev are working on this to fix.

#12

Dear @veraj,

Any good news? Thanks!

#13

Hi, xiaodai

Sorry. The internal issue has not fixed.

#14

Just an update: We are actively working on the underlying issue that is causing this problem and, while not imminent, it a fix is expected in an upcoming release (200439277)

#15

I got the same error, and i have three machines, one is host machines, another two are target, and when i using remote debug, one target machines is fine, but another not, and then console show the same msg as :unknown register ymm0h requested

my host and target machines’ configuration as follows:

OS: ubuntu 16.04
cuda Toolkit; cuda 10.0
GPU : tesla V100
cuda-gdb version: 7.12

if there are any method to avoid this?
Thanks

#16

Unfortunately there isn’t a workaround for this problem on late-model CPUs. The update to resolve this is still pending and will be in an upcoming release.

#17

Hi,
Do you have estimation when the version that resolve this will release?

#18

Hi,

I have the same problem.
Is it already clear when a new release is expected?
Hope to hear from you, thanks in advance.

#19

any good news？

#20

Hello! Could you try the latest cuda-gdb from CUDA toolkit 11.2 (which is based on GDB 8.2+)? This issue should be fixed in that version.