Error 46: all CUDA-capable devices are busy or unavailable

Hi, we’re running a same from NvPipe: https://github.com/NVIDIA/NvPipe and encountering this error on one of our machines using GRID:

NvPipe/build$ ./nvpExampleMemory
NvPipe example application: Comparison of using host/device memory.

Resolution: 3840 x 2160
Codec: H.264
Bitrate: 32 Mbps @ 90 Hz
Resolution: 3840 x 2160

--- Encode from host memory / Decode to host memory ---
Frame | Encode (ms) | Decode (ms) | Size (KB)
Failed to create encoder: Failed to synchronize device (Error 46: all CUDA-capable devices are busy or unavailable)
Failed to create decoder: Failed to synchronize device (Error 46: all CUDA-capable devices are busy or unavailable)

Context:

$ nvidia-smi  --query | grep 'Compute Mode'
    Compute Mode                    : Default
$ nvidia-smi
Mon Sep 16 19:30:05 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.70       Driver Version: 418.70       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GRID T4-1B          On   | 00000000:00:0B.0 Off |                  N/A |
| N/A   N/A    P8    N/A /  N/A |     80MiB /  1024MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

License info:

$ grep gridd /var/log/syslog
Sep 16 04:14:07 ubuntu2 nvidia-gridd: Calling load_byte_array(tra)
Sep 16 04:14:09 ubuntu2 nvidia-gridd: License renewed successfully. (Info: http://139.128.23.170:7070/request; Quadro-Virtual-DWS,5.0)
Sep 16 07:01:32 ubuntu2 nvidia-gridd: Started (779)
Sep 16 07:01:32 ubuntu2 nvidia-gridd: Ignore service provider licensing
Sep 16 07:01:32 ubuntu2 nvidia-gridd: Unable to open directory /etc/nvidia/license (No such file or directory)
Sep 16 07:01:33 ubuntu2 nvidia-gridd: Service provider detection complete.
Sep 16 07:01:33 ubuntu2 nvidia-gridd: Calling load_byte_array(tra)
Sep 16 07:01:35 ubuntu2 nvidia-gridd: Acquiring license for GRID vGPU Edition.
Sep 16 07:01:35 ubuntu2 nvidia-gridd: Calling load_byte_array(tra)
Sep 16 07:01:36 ubuntu2 nvidia-gridd: Failed to acquire/renew license from license server. (Info: http://139.128.23.170:7070/request; GRID-Virtual-PC,2.0 - Error: [1,7E2,2,0[7000000B,0,702C7]]#012Requested feature was not found.)
Sep 16 07:01:36 ubuntu2 nvidia-gridd: Failed to borrow license
Sep 16 07:01:36 ubuntu2 nvidia-gridd: Calling load_byte_array(tra)
Sep 16 07:01:38 ubuntu2 nvidia-gridd: Failed to acquire/renew license from license server. (Info: http://139.128.23.170:7070/request; GRID-Virtual-WS,2.0 - Error: [1,7E2,2,0[7000000B,0,702C7]]#012Requested feature was not found.)
Sep 16 07:01:38 ubuntu2 nvidia-gridd: Failed to borrow license
Sep 16 07:01:38 ubuntu2 nvidia-gridd: Calling load_byte_array(tra)
Sep 16 07:01:41 ubuntu2 nvidia-gridd: License acquired successfully. (Info: http://139.128.23.170:7070/request; Quadro-Virtual-DWS,5.0)
Sep 16 07:05:41 ubuntu2 nvidia-gridd: Started (729)
Sep 16 07:05:41 ubuntu2 nvidia-gridd: Ignore service provider licensing
Sep 16 07:05:41 ubuntu2 nvidia-gridd: Unable to open directory /etc/nvidia/license (No such file or directory)
Sep 16 07:05:42 ubuntu2 nvidia-gridd: Service provider detection complete.
Sep 16 07:05:42 ubuntu2 nvidia-gridd: Calling load_byte_array(tra)
Sep 16 07:05:43 ubuntu2 nvidia-gridd: Acquiring license for GRID vGPU Edition.
Sep 16 07:05:43 ubuntu2 nvidia-gridd: Calling load_byte_array(tra)
Sep 16 07:05:45 ubuntu2 nvidia-gridd: Failed to acquire/renew license from license server. (Info: http://139.128.23.170:7070/request; GRID-Virtual-PC,2.0 - Error: [1,7E2,2,0[7000000B,0,702C7]]#012Requested feature was not found.)
Sep 16 07:05:45 ubuntu2 nvidia-gridd: Failed to borrow license
Sep 16 07:05:45 ubuntu2 nvidia-gridd: Calling load_byte_array(tra)
Sep 16 07:05:48 ubuntu2 nvidia-gridd: Failed to acquire/renew license from license server. (Info: http://139.128.23.170:7070/request; GRID-Virtual-WS,2.0 - Error: [1,7E2,2,0[7000000B,0,702C7]]#012Requested feature was not found.)
Sep 16 07:05:48 ubuntu2 nvidia-gridd: Failed to borrow license
Sep 16 07:05:48 ubuntu2 nvidia-gridd: Calling load_byte_array(tra)
Sep 16 07:05:50 ubuntu2 nvidia-gridd: License acquired successfully. (Info: http://139.128.23.170:7070/request; Quadro-Virtual-DWS,5.0)
Sep 16 07:13:49 ubuntu2 nvidia-gridd: Started (838)
Sep 16 07:13:49 ubuntu2 nvidia-gridd: Ignore service provider licensing
Sep 16 07:13:49 ubuntu2 nvidia-gridd: Unable to open directory /etc/nvidia/license (No such file or directory)
Sep 16 07:13:50 ubuntu2 nvidia-gridd: Service provider detection complete.
Sep 16 07:13:50 ubuntu2 nvidia-gridd: Calling load_byte_array(tra)
Sep 16 07:13:52 ubuntu2 nvidia-gridd: Acquiring license for GRID vGPU Edition.
Sep 16 07:13:52 ubuntu2 nvidia-gridd: Calling load_byte_array(tra)
Sep 16 07:13:53 ubuntu2 nvidia-gridd: Failed to acquire/renew license from license server. (Info: http://139.128.23.170:7070/request; GRID-Virtual-PC,2.0 - Error: [1,7E2,2,0[7000000B,0,702C7]]#012Requested feature was not found.)
Sep 16 07:13:53 ubuntu2 nvidia-gridd: Failed to borrow license
Sep 16 07:13:53 ubuntu2 nvidia-gridd: Calling load_byte_array(tra)
Sep 16 07:13:56 ubuntu2 nvidia-gridd: Failed to acquire/renew license from license server. (Info: http://139.128.23.170:7070/request; GRID-Virtual-WS,2.0 - Error: [1,7E2,2,0[7000000B,0,702C7]]#012Requested feature was not found.)
Sep 16 07:13:56 ubuntu2 nvidia-gridd: Failed to borrow license
Sep 16 07:13:56 ubuntu2 nvidia-gridd: Calling load_byte_array(tra)
Sep 16 07:13:58 ubuntu2 nvidia-gridd: License acquired successfully. (Info: http://139.128.23.170:7070/request; Quadro-Virtual-DWS,5.0)
Sep 16 12:01:59 ubuntu2 nvidia-gridd: Calling load_byte_array(tra)
Sep 16 12:02:01 ubuntu2 nvidia-gridd: License renewed successfully. (Info: http://139.128.23.170:7070/request; Quadro-Virtual-DWS,5.0)
Sep 16 16:50:02 ubuntu2 nvidia-gridd: Calling load_byte_array(tra)
Sep 16 16:50:04 ubuntu2 nvidia-gridd: License renewed successfully. (Info: http://139.128.23.170:7070/request; Quadro-Virtual-DWS,5.0)

When switching it to passthrough, things worked fine:

$ ./nvpExampleMemory
NvPipe example application: Comparison of using host/device memory.

Resolution: 3840 x 2160
Codec: H.264
Bitrate: 32 Mbps @ 90 Hz
Resolution: 3840 x 2160

--- Encode from host memory / Decode to host memory ---
Frame | Encode (ms) | Decode (ms) | Size (KB)
    0 |        24.7 |        58.3 |     23.1
    1 |        26.2 |       137.0 |      6.8
    2 |        20.1 |        10.6 |      3.9
    3 |        20.2 |        10.4 |      1.6
    4 |        20.0 |        10.5 |      4.0
    5 |        20.1 |        10.5 |      1.2
    6 |        19.9 |         9.3 |      2.0
    7 |        17.2 |         9.2 |      0.5
    8 |        17.2 |         9.1 |      0.5
    9 |        17.1 |         9.1 |      0.4

--- Encode from device memory / Decode to device memory ---
Frame | Encode (ms) | Decode (ms) | Size (KB)
    0 |        10.8 |        44.1 |     23.1
    1 |        12.0 |         5.9 |      6.8
    2 |        12.0 |         5.7 |      3.9
    3 |        12.0 |         5.8 |      1.6
    4 |        12.1 |         5.7 |      4.0
    5 |        12.0 |         5.8 |      1.2
    6 |        12.0 |         5.9 |      2.0
    7 |        12.1 |         5.9 |      0.5
    8 |        12.0 |         5.7 |      0.5
    9 |        11.6 |         5.5 |      0.4

System info:

- Ubuntu 18.04.3 LTS
- Driver Version: 418.70

I believe the issue ended up being that we were using a T4-1B, and a T4-2Q was needed instead