Deepstream Triton Docker container cannot run with MPS

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Hi All,

Here’s the background. I am trying to run deepstream with Triton inference server in separate docker container (Triton & Deepstream).
The Triton inference server will not be specified to the deepstream application since I am planning to use the inference server in separate use cases. The plan is to use gRPC to communicate between

  1. deepstream application and
  2. Triton inference server

following the tutorial in /opt/nvidia/deepstream/deepstream-6.1/samples/configs/deepstream-app-triton-grpc.

While I am able to run both the server (triton) and the client (deepstream) on the same computer, I would like to utilize the GPU resource by using MPS. (I am assuming the triton image and deepstream image share the GPU resource through context switching/time slicing.)

As long as I enable the MPS, the docker image fail to response and fail to enter into the shell. I tried different containers (as described below) and I am able to run nvidia-smi.

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
GTX 3090
• DeepStream Version
nvcr.io/nvidia/deepstream:6.1.1-triton
• NVIDIA GPU Driver Version (valid for GPU only)
515.86.01
• Issue Type( questions, new requirements, bugs)
questions
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)

  1. enable MPS
    export CUDA_VISIBLE_DEVICES=0
    export CUDA_MPS_PIPE_DIRECTORY=/tmp/nvidia-mps
    export CUDA_MPS_LOG_DIRECTORY=/tmp/nvidia-log
    sudo nvidia-smi -i 0 -c EXCLUSIVE_PROCESS
    nvidia-cuda-mps-control -d

  2. Run the container docker run --gpus all --ipc=host -it --rm -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY --net=host --name=deepstream-client nvcr.io/nvidia/deepstream:6.1.1-triton fail

  3. This container works fine docker run --rm --gpus all --ipc=host nvcr.io/nvidia/cuda:11.7.0-devel-ubuntu20.04 nvidia-smi ok

Please kindly advise how to use the images with MPS.

Many Thanks!

3

I will check

4

how many GPUs on your machine? could you share the terminal log of “Run the container failed”?

6

I am using only 1 GPU, which is RTX 3090. The terminal just stuck when I run the docker run command without any response.

7

update:
we tested on local gpu, " docker run --gpus all --ipc=host -it --rm -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY --net=host --name=deepstream-client nvcr.io/nvidia/deepstream:6.1.1-triton" can be started, but there is an error tip ERROR: No supported GPU(s) detected to run this container , and some deepstream samples test failed with error 805, which means the MPS client failed to connect to the MPS control daemon or the MPS server.
we will continue to investigate.