Dr. Watson with CUDA + Xen Desktop + HDX3D-Plugin

Accelerated Computing CUDA CUDA Programming and Performance
1

Dr. Watson with CUDA + Xen Desktop + HDX3D-Plugin

Error by using GPU-based compression

In the past weeks we reported an error to Citrix, because we got problems by using XenDesktop with HDX3D-Plugin on a CAD-Workstation with PNY Quadro FX4000 graphic card

In the end we got the answer, that “DEV Analysis Team” from Citrix analyzed our dump and they told me, that the problem is initiated by nvcuda.dll, because a function
returns a wrong pointer.

They told me, that Citrix does not have the symbol files for CUDa and that they aren’t able to analyze deeper and I should take contact NVidia.

We are using the following environment:

Hardware:

  • Dell Precision T3400 Core 2 Duo E8500
  • PNY Quadro 4000

Software on CAD-Workstation:

  • Windows XP x32 (German) (Full Microsoft patched)
  • NVidia Driver 259.57
  • Citrix Virtual Desktop Agent 4.0.5010 (Xen Desktop 4.0 SP1)
  • Citrix HDX3D for Pro Graphics 1.1.2006 (PlugIn HDX3D_1_1 HDX3D_PROGFX_x86SERVERMSI.msi)

Software on Client:

  • Windows XP x32 (German) SP3 (Full Microsoft patched)
  • 12.0.1.1 client included with HDX3D package (Answer from Citrix: The HDX3D support at the moment is not included in main client branch therefore to leverage HDX3D you would need to use the 12.0.1.1 client included with HDX3D package.)

I got the following error reference infos:

############################################################
#############################################################
##############
Microsoft ® Windows Debugger Version 6.12.0002.633 X86
Copyright © Microsoft Corporation. All rights reserved.

Loading Dump File [C:\MK-Temp\picaDispMgr.exe.hdmp]
User Mini Dump File: Only registers, stack and portions of memory are available
Symbol search path is: SRVC:\SymbolFileshttp://ctxsym.citrix.com/symbols;SRV*C:\SymbolFiles*http://msdl.microsoft.com/download/symbols

Executable search path is:
Windows XP Version 2600 (Service Pack 3) MP (2 procs) Free x86 compatible
Product: WinNt, suite: SingleUserTS
Machine Name:
Debug session time: Fri Oct 8 15:12:05.000 2010 (UTC + 2:00)
System Uptime: not available
Process Uptime: 0 days 1:20:37.000
…
URL requested: [url=“http://watson.microsoft.com/StageOne/picaDispMgr_exe/4_0_0_5010/kernel32_dll/5_1_2600_5781/0000984e.htm?Retriage=1”]http://watson.microsoft.com/StageOne/picaD....htm?Retriage=1[/url]

FAULTING_IP:
kernel32!InterlockedCompareExchange+c
7c80984e f00fb111 lock cmpxchg dword ptr [ecx],edx

EXCEPTION_RECORD: ffffffff – (.exr 0xffffffffffffffff)
ExceptionAddress: 7c80984e (kernel32!InterlockedCompareExchange+0x0000000c)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 00000001
Parameter[1]: 188c0109
Attempt to write to address 188c0109

DEFAULT_BUCKET_ID: INVALID_POINTER_READ

PROCESS_NAME: picaDispMgr.exe

ERROR_CODE: (NTSTATUS) 0xc0000005 - Die Anweisung in 0x%08lx verweist auf Speicher 0x%08lx. Der Vorgang %s konnte nicht im Speicher durchgef hrt werden.

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - Die Anweisung in 0x%08lx verweist auf Speicher 0x%08lx. Der Vorgang %s konnte nicht im Speicher durchgef hrt werden.

EXCEPTION_PARAMETER1: 00000001
EXCEPTION_PARAMETER2: 188c0109
WRITE_ADDRESS: 188c0109
FOLLOWUP_IP:
nvcuda+311495
095e1495 ?? ???
MOD_LIST:
NTGLOBALFLAG: 0
APPLICATION_VERIFIER_FLAGS: 0
FAULTING_THREAD: 00000e84
PRIMARY_PROBLEM_CLASS: INVALID_POINTER_READ
BUGCHECK_STR: APPLICATION_FAULT_INVALID_POINTER_READ_INVALID_POINTER_WRITE
LAST_CONTROL_TRANSFER: from 095e1495 to 7c80984e

…

STACK_COMMAND: ~9s; .ecxr ; kb

SYMBOL_STACK_INDEX: 1

SYMBOL_NAME: nvcuda+311495

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: nvcuda

IMAGE_NAME: nvcuda.dll

DEBUG_FLR_IMAGE_TIMESTAMP: 4c7f0aa4

FAILURE_BUCKET_ID: INVALID_POINTER_READ_c0000005_nvcuda.dll!Unknown

BUCKET_ID: APPLICATION_FAULT_INVALID_POINTER_READ_INVALID_POINTER_WRITE
_nvcuda+311495

WATSON_STAGEONE_URL: [url=“http://watson.microsoft.com/StageOne/picaDispMgr_exe/4_0_0_5010/4c349381/kernel32_dll/5_1_2600_5781/49c4f482/c0000005/0000984e.htm?Retriage=1”]http://watson.microsoft.com/StageOne/picaD....htm?Retriage=1[/url]

Followup: MachineOwner

0:009> lmvm nvcuda
start end module name
092d0000 09775000 nvcuda T (no symbols)
Loaded symbol image file: nvcuda.dll
Image path: C:\WINDOWS\system32\nvcuda.dll
Image name: nvcuda.dll
Timestamp: Thu Sep 02 04:23:32 2010 (4C7F0AA4)
CheckSum: 00000000
ImageSize: 004A5000
File version: 6.14.12.5957
Product version: 5.2.3790.1830
File flags: 8 (Mask 3F) Private
File OS: 40004 NT Win32
File type: 3.4 Driver
File date: 00000000.00000000
Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4
############################################################
#############################################################
##############

I 'm able to upload a “full page heap dump” 55,6MB (2,2MB zipped)

Is there anybody able to help me?

Kind regards

Frank

2

Dr. Watson with CUDA + Xen Desktop + HDX3D-Plugin

Error by using GPU-based compression

In the past weeks we reported an error to Citrix, because we got problems by using XenDesktop with HDX3D-Plugin on a CAD-Workstation with PNY Quadro FX4000 graphic card

In the end we got the answer, that “DEV Analysis Team” from Citrix analyzed our dump and they told me, that the problem is initiated by nvcuda.dll, because a function
returns a wrong pointer.

They told me, that Citrix does not have the symbol files for CUDa and that they aren’t able to analyze deeper and I should take contact NVidia.

We are using the following environment:

Hardware:

  • Dell Precision T3400 Core 2 Duo E8500
  • PNY Quadro 4000

Software on CAD-Workstation:

  • Windows XP x32 (German) (Full Microsoft patched)
  • NVidia Driver 259.57
  • Citrix Virtual Desktop Agent 4.0.5010 (Xen Desktop 4.0 SP1)
  • Citrix HDX3D for Pro Graphics 1.1.2006 (PlugIn HDX3D_1_1 HDX3D_PROGFX_x86SERVERMSI.msi)

Software on Client:

  • Windows XP x32 (German) SP3 (Full Microsoft patched)
  • 12.0.1.1 client included with HDX3D package (Answer from Citrix: The HDX3D support at the moment is not included in main client branch therefore to leverage HDX3D you would need to use the 12.0.1.1 client included with HDX3D package.)

I got the following error reference infos:

############################################################
#############################################################
##############
Microsoft ® Windows Debugger Version 6.12.0002.633 X86
Copyright © Microsoft Corporation. All rights reserved.

Loading Dump File [C:\MK-Temp\picaDispMgr.exe.hdmp]
User Mini Dump File: Only registers, stack and portions of memory are available
Symbol search path is: SRVC:\SymbolFileshttp://ctxsym.citrix.com/symbols;SRV*C:\SymbolFiles*http://msdl.microsoft.com/download/symbols

Executable search path is:
Windows XP Version 2600 (Service Pack 3) MP (2 procs) Free x86 compatible
Product: WinNt, suite: SingleUserTS
Machine Name:
Debug session time: Fri Oct 8 15:12:05.000 2010 (UTC + 2:00)
System Uptime: not available
Process Uptime: 0 days 1:20:37.000
…
URL requested: [url=“http://watson.microsoft.com/StageOne/picaDispMgr_exe/4_0_0_5010/kernel32_dll/5_1_2600_5781/0000984e.htm?Retriage=1”]http://watson.microsoft.com/StageOne/picaD....htm?Retriage=1[/url]

FAULTING_IP:
kernel32!InterlockedCompareExchange+c
7c80984e f00fb111 lock cmpxchg dword ptr [ecx],edx

EXCEPTION_RECORD: ffffffff – (.exr 0xffffffffffffffff)
ExceptionAddress: 7c80984e (kernel32!InterlockedCompareExchange+0x0000000c)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 00000001
Parameter[1]: 188c0109
Attempt to write to address 188c0109

DEFAULT_BUCKET_ID: INVALID_POINTER_READ

PROCESS_NAME: picaDispMgr.exe

ERROR_CODE: (NTSTATUS) 0xc0000005 - Die Anweisung in 0x%08lx verweist auf Speicher 0x%08lx. Der Vorgang %s konnte nicht im Speicher durchgef hrt werden.

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - Die Anweisung in 0x%08lx verweist auf Speicher 0x%08lx. Der Vorgang %s konnte nicht im Speicher durchgef hrt werden.

EXCEPTION_PARAMETER1: 00000001
EXCEPTION_PARAMETER2: 188c0109
WRITE_ADDRESS: 188c0109
FOLLOWUP_IP:
nvcuda+311495
095e1495 ?? ???
MOD_LIST:
NTGLOBALFLAG: 0
APPLICATION_VERIFIER_FLAGS: 0
FAULTING_THREAD: 00000e84
PRIMARY_PROBLEM_CLASS: INVALID_POINTER_READ
BUGCHECK_STR: APPLICATION_FAULT_INVALID_POINTER_READ_INVALID_POINTER_WRITE
LAST_CONTROL_TRANSFER: from 095e1495 to 7c80984e

…

STACK_COMMAND: ~9s; .ecxr ; kb

SYMBOL_STACK_INDEX: 1

SYMBOL_NAME: nvcuda+311495

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: nvcuda

IMAGE_NAME: nvcuda.dll

DEBUG_FLR_IMAGE_TIMESTAMP: 4c7f0aa4

FAILURE_BUCKET_ID: INVALID_POINTER_READ_c0000005_nvcuda.dll!Unknown

BUCKET_ID: APPLICATION_FAULT_INVALID_POINTER_READ_INVALID_POINTER_WRITE
_nvcuda+311495

WATSON_STAGEONE_URL: [url=“http://watson.microsoft.com/StageOne/picaDispMgr_exe/4_0_0_5010/4c349381/kernel32_dll/5_1_2600_5781/49c4f482/c0000005/0000984e.htm?Retriage=1”]http://watson.microsoft.com/StageOne/picaD....htm?Retriage=1[/url]

Followup: MachineOwner

0:009> lmvm nvcuda
start end module name
092d0000 09775000 nvcuda T (no symbols)
Loaded symbol image file: nvcuda.dll
Image path: C:\WINDOWS\system32\nvcuda.dll
Image name: nvcuda.dll
Timestamp: Thu Sep 02 04:23:32 2010 (4C7F0AA4)
CheckSum: 00000000
ImageSize: 004A5000
File version: 6.14.12.5957
Product version: 5.2.3790.1830
File flags: 8 (Mask 3F) Private
File OS: 40004 NT Win32
File type: 3.4 Driver
File date: 00000000.00000000
Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4
############################################################
#############################################################
##############

I 'm able to upload a “full page heap dump” 55,6MB (2,2MB zipped)

Is there anybody able to help me?

Kind regards

Frank

3

Because I got no answer here more details:

Xen Desktop is a product of Citrix and serves a physical
or virtual PC on a corporate network via Citrix established protocols
and processes.
This can be compared with a remote control software. The product
“Xen Desktop” with the add-on “HDX3D” is so effective that even 3D-CAD is usable
on a 2-3MBit line. (So the theory)

Unfortunately a few problems could occur. As a thoughtful condition Citrix list a CUDA graphics card
with at least 96 cores. CUDA functions were used than for calculating screen changes.

We have established a XenDesktop 4.0 Infrastructure (Provisioning Server + WEB server)
It is important to mention that an Enterprise licence of Xen Desktop is necessary to get HDXAddOn from Citrix.

Adaptation of the “default.ica” on the web server (eg “ConnectionBar = 1”)
should also be configured.

For the CAD workstation (CAD server) we use an “Dell Precision T3400 Core 2 Duo E8500”
We replaced the graphic card and integrated a “Quadro 4000” (not Quadro FX4000).

As operating system we use Windows XP SP3 with all MS patches (full patched).
We followed instructions (http://support.citrix.com/article/CTX124443)
to setup driver and Citrix software on this CAD server.

On my client PC “Dell Optiplex 980” (also Windows XP SP3 full patched)
I installed Citrix client software “CitrixOnlinePluginWeb.exe” version “12.0.1.1”
out of the HDX3D-package “HDX3D_1_1 - RTM_with_12.0.1_client_health_Check_Tool” from Citrix
(It’s imported to use this client, not from other sources).

After setting up of provisioning server and web server I’m able to authenticate myself
to this web server. I get my link to the CAD server. After starting
the connection is established and I get the session on this machine (window or full screen size)
After a few actions (eg starting of programs) the connection breaks.
I can see a picture, but it’s nothing more to operate. By installing a
Remote access software (eg Dameware) on the CAD server I can see that an error occures
(message on screen). I can close the Citrix session window and it’s possible to re initiate a
connection to this lost session by double clicking WEB server entry.
But however, a session termination occurs every few seconds, so this product
is not usable yet.
The error message
reads:
“Citrix PortICA Display Manager / Citrix PortICA Display Manager has encountered a problem ,…” (I hope translation from German was OK)

I followed Instructions “http://support.citrix.com/article/CTX104633” for generating a full page heap dump.

I generated this dump and it was analyzed by Citrix, so far it was possible.
According to Citrix the bug is in the file “nvcuda.dll” which returns a wrong pointer.
Due missing symbol files Citrix is not possible to analyze this problem. So I should get in touch with NVidia,
what should be done hereby.

I am aware that Nvidia will now also have problems to get contact to Citrix, because
they have the same customer policy (No direct support to end customers).
Therefore, the problem will occur even in setting up a test environment An
Enterprise license is necessary to get HDX3D-AddOn.

Our Service Partner “PC-Ware” has therefore offered me to be available
to initiate a conference by phone between NVidia and Citrix technicians.
Furthermore, “PC-Ware” offer us to be helpful for Trial licenses and to support NVidia.

For that: German Hotline Number of PC-Ware: 0180-256800.
Our case No: 159878.

Kind regards

Frank

4

Because I got no answer here more details:

Xen Desktop is a product of Citrix and serves a physical
or virtual PC on a corporate network via Citrix established protocols
and processes.
This can be compared with a remote control software. The product
“Xen Desktop” with the add-on “HDX3D” is so effective that even 3D-CAD is usable
on a 2-3MBit line. (So the theory)

Unfortunately a few problems could occur. As a thoughtful condition Citrix list a CUDA graphics card
with at least 96 cores. CUDA functions were used than for calculating screen changes.

We have established a XenDesktop 4.0 Infrastructure (Provisioning Server + WEB server)
It is important to mention that an Enterprise licence of Xen Desktop is necessary to get HDXAddOn from Citrix.

Adaptation of the “default.ica” on the web server (eg “ConnectionBar = 1”)
should also be configured.

For the CAD workstation (CAD server) we use an “Dell Precision T3400 Core 2 Duo E8500”
We replaced the graphic card and integrated a “Quadro 4000” (not Quadro FX4000).

As operating system we use Windows XP SP3 with all MS patches (full patched).
We followed instructions (http://support.citrix.com/article/CTX124443)
to setup driver and Citrix software on this CAD server.

On my client PC “Dell Optiplex 980” (also Windows XP SP3 full patched)
I installed Citrix client software “CitrixOnlinePluginWeb.exe” version “12.0.1.1”
out of the HDX3D-package “HDX3D_1_1 - RTM_with_12.0.1_client_health_Check_Tool” from Citrix
(It’s imported to use this client, not from other sources).

After setting up of provisioning server and web server I’m able to authenticate myself
to this web server. I get my link to the CAD server. After starting
the connection is established and I get the session on this machine (window or full screen size)
After a few actions (eg starting of programs) the connection breaks.
I can see a picture, but it’s nothing more to operate. By installing a
Remote access software (eg Dameware) on the CAD server I can see that an error occures
(message on screen). I can close the Citrix session window and it’s possible to re initiate a
connection to this lost session by double clicking WEB server entry.
But however, a session termination occurs every few seconds, so this product
is not usable yet.
The error message
reads:
“Citrix PortICA Display Manager / Citrix PortICA Display Manager has encountered a problem ,…” (I hope translation from German was OK)

I followed Instructions “http://support.citrix.com/article/CTX104633” for generating a full page heap dump.

I generated this dump and it was analyzed by Citrix, so far it was possible.
According to Citrix the bug is in the file “nvcuda.dll” which returns a wrong pointer.
Due missing symbol files Citrix is not possible to analyze this problem. So I should get in touch with NVidia,
what should be done hereby.

I am aware that Nvidia will now also have problems to get contact to Citrix, because
they have the same customer policy (No direct support to end customers).
Therefore, the problem will occur even in setting up a test environment An
Enterprise license is necessary to get HDX3D-AddOn.

Our Service Partner “PC-Ware” has therefore offered me to be available
to initiate a conference by phone between NVidia and Citrix technicians.
Furthermore, “PC-Ware” offer us to be helpful for Trial licenses and to support NVidia.

For that: German Hotline Number of PC-Ware: 0180-256800.
Our case No: 159878.

Kind regards

Frank

5

I was able to create a contact between NVida (Germany/UK) and Citrix (UK) - Was not easy. Solution:
There exists today no solution. Support of the Fermi-based models of Nvidia card is on the roadmap, and should be addressed in future release of HDX 3D Professional Graphics. But today Xen Desktop is not supported with GPU encoding on this cards (Models: Quadro 2000 , Quadro 4000 , Quadro 5000, Quadro 6000). Only with CPU based encoding. Documentation “HDX 3D for Professional Graphics” will be amended now, so that nobody else will fall into this problem. So purchase an old Quadro FX model. A lot thanks to PNY, NVidia and PC Ware for there work on this case. Good work!!

6

I was able to create a contact between NVida (Germany/UK) and Citrix (UK) - Was not easy. Solution:
There exists today no solution. Support of the Fermi-based models of Nvidia card is on the roadmap, and should be addressed in future release of HDX 3D Professional Graphics. But today Xen Desktop is not supported with GPU encoding on this cards (Models: Quadro 2000 , Quadro 4000 , Quadro 5000, Quadro 6000). Only with CPU based encoding. Documentation “HDX 3D for Professional Graphics” will be amended now, so that nobody else will fall into this problem. So purchase an old Quadro FX model. A lot thanks to PNY, NVidia and PC Ware for there work on this case. Good work!!