390-46 driver broke my K80 GPUs

On a RedHat 6.9 system with two K80 GPUs, we updated from 384.111 to the 390.46 driver. After doing the update, running the “nvidia-smi -q” command crashes the system.

Even worse, the system continue to crash after I revert back to the 384.111 driver.

We did this update on two separate systems. One silently crashes with no messages on the console.
The other reports the following error -

/usr/bin/nvidia-smi -q

{1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 3
{1}[Hardware Error]: event severity: fatal
{1}[Hardware Error]: Error 0, type: fatal
{1}[Hardware Error]: section_type: PCIe error
{1}[Hardware Error]: port_type: 4, root port
{1}[Hardware Error]: version: 1.16
{1}[Hardware Error]: command: 0x0547, status: 0x4010
{1}[Hardware Error]: device_id: 0000:80:03.0
{1}[Hardware Error]: slot: 5
{1}[Hardware Error]: secondary_bus: 0x83
{1}[Hardware Error]: vendor_id: 0x8086, device_id: 0x6f08
{1}[Hardware Error]: class_code: 000406
{1}[Hardware Error]: bridge: secondary_status: 0x2000, control: 0x0003
Kernel panic - not syncing: Fatal hardware error!
Pid: 0, comm: swapper Tainted: P – ------------ 2.6.32-696.20.1.el1
Call Trace:
[] ? panic+0xa7/0x179
[] ? ghes_notify_nmi+0x241/0x250
[] ? notifier_call_chain+0x55/0x80
[] ? atomic_notifier_call_chain+0x1a/0x20
[] ? notify_die+0x2e/0x30
[] ? do_nmi+0x107/0x360
[] ? nmi+0x1f9/0x258
[] ? mwait_idle+0x77/0xd0
<> [] ? cpu_idle+0xb6/0x110
[] ? rest_init+0x7a/0x80
[] ? start_kernel+0x42e/0x43b
[] ? x86_64_start_reservations+0x125/0x129
[] ? x86_64_start_kernel+0x115/0x124

Prior to updating our K80 based systems, we updated our V1000-PCIE based systems without any issues.