Hi all,
I am trying to use deepstream and Triton inference servers in different computers/Nodes. The plan is to use a dedicated computer to handle inference and manage models, and multiple computers to handle multiple streams. I am able to open 2 containers on the same computer, and successfully run the example. But when I run it in 2 computers, I get
ERROR: infer_grpc_client.cpp:223 Failed to register CUDA shared memory.
Can you advice how to connect deepstream and triton inference server through grpc, as well as how to go through the tutorial?.
Please provide complete information as applicable to your setup.
• Hardware Platform (Jetson / GPU)
Computer 1: RTX3090
Computer 2: RTX3090
• DeepStream Version
both computer: deepstream:6.1.1-triton
• NVIDIA GPU Driver Version (valid for GPU only)
515.86.01
• Issue Type( questions, new requirements, bugs)
bugs
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
Computer 1 (Server):
- Run the docker image
docker run --gpus all -it --rm -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY --net=host --name=triton-server nvcr.io/nvidia/deepstream:6.1.1-triton
Inside the container
-
cd samples
-
./prepare_ds_triton_model_repo.sh
-
tritonserver --model-repository triton_model_repo/
Verify the application can communicate through grpc in different container
-
docker run --gpus all -it --rm -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY --net=host --name=triton-client nvcr.io/nvidia/deepstream:6.1.1-triton
-
cd /opt/nvidia/deepstream/deepstream-6.1/samples/configs/deepstream-app-triton-grpc
-
deepstream-app -c source30_1080p_dec_infer-resnet_tiled_display_int8.txt
Computer 2 (Client):
-
Run the docker image
docker run --gpus all -it --rm -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY --net=host --name=triton-client nvcr.io/nvidia/deepstream:6.1.1-triton
Inside the container -
Change the grpc url to the server ip
cd /opt/nvidia/deepstream/deepstream-6.1/samples/configs/deepstream-app-triton-grpc
-
vim config_infer_plan_engine_primary.txt`
grpc {
url: “192.168.51.13:8001”
# url: “localhost:8001”
enable_cuda_buffer_sharing: true
} -
deepstream-app -c source30_1080p_dec_infer-resnet_tiled_display_int8.txt
Error Messages:
WARNING: infer_proto_utils.cpp:144 auto-update preprocess.network_format to IMAGE_FORMAT_RGB
INFO: infer_grpc_backend.cpp:169 TritonGrpcBackend id:1 initialized for model: Primary_Detector
ERROR: infer_grpc_client.cpp:223 Failed to register CUDA shared memory.
ERROR: infer_grpc_client.cpp:311 Failed to set inference input: failed to register CUDA shared memory region ‘inbuf_0x55558a0e0800’: failed to open CUDA IPC handle: invalid resource handle
ERROR: infer_grpc_backend.cpp:140 gRPC backend run failed to create request for model: Primary_Detector
ERROR: infer_trtis_backend.cpp:350 failed to specify dims when running inference on model:Primary_Detector, nvinfer error:NVDSINFER_TRITON_ERROR
0:00:00.127168214 1531 0x55558ab10920 ERROR nvinferserver gstnvinferserver.cpp:375:gst_nvinfer_server_logger:<primary_gie> nvinferserver[UID 1]: Error in specifyBackendDims() <infer_grpc_context.cpp:154> [UID = 1]: failed to specify input dims triton backend for model:Primary_Detector, nvinfer error:NVDSINFER_TRITON_ERROR
0:00:00.127178191 1531 0x55558ab10920 ERROR nvinferserver gstnvinferserver.cpp:375:gst_nvinfer_server_logger:<primary_gie> nvinferserver[UID 1]: Error in createNNBackend() <infer_grpc_context.cpp:210> [UID = 1]: failed to specify triton backend input dims for model:Primary_Detector, nvinfer error:NVDSINFER_TRITON_ERROR
0:00:00.127188907 1531 0x55558ab10920 ERROR nvinferserver gstnvinferserver.cpp:375:gst_nvinfer_server_logger:<primary_gie> nvinferserver[UID 1]: Error in initialize() <infer_base_context.cpp:79> [UID = 1]: create nn-backend failed, check config file settings, nvinfer error:NVDSINFER_TRITON_ERROR
0:00:00.127192287 1531 0x55558ab10920 WARN nvinferserver gstnvinferserver_impl.cpp:547:start:<primary_gie> error: Failed to initialize InferTrtIsContext
0:00:00.127194044 1531 0x55558ab10920 WARN nvinferserver gstnvinferserver_impl.cpp:547:start:<primary_gie> error: Config file path: /opt/nvidia/deepstream/deepstream-6.1/samples/configs/deepstream-app-triton-grpc/config_infer_plan_engine_primary.txt
0:00:00.127210681 1531 0x55558ab10920 WARN nvinferserver gstnvinferserver.cpp:473:gst_nvinfer_server_start:<primary_gie> error: gstnvinferserver_impl start failed
** ERROR: main:716: Failed to set pipeline to PAUSED
Quitting
ERROR from primary_gie: Failed to initialize InferTrtIsContext
Debug info: gstnvinferserver_impl.cpp(547): start (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInferServer:primary_gie:
Config file path: /opt/nvidia/deepstream/deepstream-6.1/samples/configs/deepstream-app-triton-grpc/config_infer_plan_engine_primary.txt
ERROR from primary_gie: gstnvinferserver_impl start failed
Debug info: gstnvinferserver.cpp(473): gst_nvinfer_server_start (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInferServer:primary_gie
App run failed
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)
Many Thanks!