Supporting a Gen4 NVIDIA Card in CUDA 10.1 running TF 2.2 when nvidia-smi reports PCIe Gen Max as "Unknown Error"

I am running a PCIe Gen4 NVIDIA card in my platform with a docker image setup to run Tensorflow 2.2 with CUDA 10.1. The problem seems to be that the driver for the Gen4 card works well enough to load and pass to the docker image however when I run nvidia-smi --query the output shows:

    GPU Link Info
        PCIe Generation
            Max                 : Unknown Error
            Current             : 1
        Link Width
            Max                 : 16x
            Current             : 16x

This seems to be ok with CUDA as I can run simple bandwidth tests but I noticed that cuDeviceCanAccessPeer(&can_access_peer, gpuid[i], gpuid[j])) fails which results in

Peer-to-Peer (P2P) access from Graphics Device (GPU0) -> Graphics Device (GPU1) : No
Peer-to-Peer (P2P) access from Graphics Device (GPU1) -> Graphics Device (GPU0) : No

I am not sure if the two items are related but I assumed if nvidia-smi was unable to detect the correct max PCIe generation, it might declare the two cards are either incapable of p2p or not matched.

Can anyone shed some light? I am worried I am in a dependency loop:

  • Newest Driver (440.82) supports this card (not confirmed if it fixes the p2p) as it shows the correct PCIe Max Gen as 4
  • Newest Driver does not support CUDA 10.1
  • CUDA 10.1 is required / works best with Tensorflow 2.2