A5000 post Display Mode Selector Tool stuck in DataCenter mode

[ilb]

I have A5000 GPUs mounted in a certified Supermicro system, with Ubuntu 20.04 HWE (5.15.0-78-generic); in order to test vGPU drivers I used the Display Mode Selector Tool (v1.6.0, march 2023) to change the gpumode to compute.

Now I’m stuck, I can not change the gpumode back to Graphics.

Specifed GPU Mode "physical_display_enabled_256MB_bar1"

Graphics Device      (10DE,2231,10DE,147E) S:00,B:1B,D:00,F:00

Specified GPU mode not supported on this device 0x2231.

and

Specifed GPU Mode "physical_display_enabled_8GB_bar1"

Graphics Device      (10DE,2231,10DE,147E) S:00,B:1B,D:00,F:00

Specified GPU mode not supported on this device 0x2231.

At the end of listing BIOS versions I received a NvFlash CPU side error message

$ sudo ./displaymodeselector --version

NVIDIA Display Mode Selector Utility (Version 1.60.0)
Copyright (C) 2015-2021, NVIDIA Corporation. All Rights Reserved.

BIOS Versions of NVIDIA display adapters present in system:

<0> Graphics Device      (10DE,2231,10DE,147E) S:00,B:1B,D:00,F:00 Version N/A
<1> Graphics Device      (10DE,2231,10DE,147E) S:00,B:1C,D:00,F:00 Version N/A
<2> Graphics Device      (10DE,2231,10DE,147E) S:00,B:1D,D:00,F:00 Version N/A
<3> Graphics Device      (10DE,2231,10DE,147E) S:00,B:1E,D:00,F:00 Version N/A
<4> Graphics Device      (10DE,2231,10DE,147E) S:00,B:3D,D:00,F:00 Version N/A
<5> Graphics Device      (10DE,2231,10DE,147E) S:00,B:3F,D:00,F:00 Version N/A
<6> Graphics Device      (10DE,2231,10DE,147E) S:00,B:40,D:00,F:00 Version N/A
<7> Graphics Device      (10DE,2231,10DE,147E) S:00,B:41,D:00,F:00 Version N/A
 Nvflash CPU side error Code:2Error Message: Falcon In HALT or STOP state, abort uCode command issuing process.

since than I am not able even to list the current mode without getting this message

 Nvflash CPU side error Code:2Error Message: Falcon In HALT or STOP state, abort uCode command issuing process.

The GPUs seem to work OK, but they are stuck in compute mode; I would like to have the option to go back to workstation mode (graphics).

[ilb]

In case anyone gets into the same state I’ll post the solution I got with the help of NVIDIA Enterprise Support; it seems that the problem was caused by some processes that were still using the devices.

To exit from the stuck state:

1. shutdown system
2. unplug all cables
3. drain all devices by pressing power on button with cables unplugged
4. replug cables
5. turn on system

The safest way to do the switch is then to:

1. ensure nouveau is blacklisted
2. purge all drivers
3. reboot
4. double check that no processes are using the GPUs with
   • sudo ps -ef | grep -i nvidia
   • sudo Isof | grep -i nvidia
   • sudo Isof | grep -i /dev/nvidia
5. switch mode
6. reinstall drivers

[TomNVIDIA]

Thanks for sharing!