Deepstream and Triton containers

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) GPU
• DeepStream Version 7.0
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only) 535.183.01
• Issue Type( questions, new requirements, bugs) questions
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

I am running Triton Inference Server in a container serving Yolov8 model. The model has been converted to onnx . I am using rtsp stream from Camera. When using python and triton http api (http://:8000), I am able to get the video feed with inferences.

I am trying to use Deepstream container to connect to Triton Inference Server container and use it for Inference. I have been trying to use deepstream-test3 with following config file. The libnvdsinfer_custom_impl_Yolo.so have been compile from the repo (GitHub - marcoslucianops/DeepStream-Yolo: NVIDIA DeepStream SDK 7.0 / 6.4 / 6.3 / 6.2 / 6.1.1 / 6.1 / 6.0.1 / 6.0 / 5.1 implementation for YOLO models)

After running the it is crashing. Any help will be much appreciated.

python3 deepstream_test_3.py -i rtsp://admin:pass@10.128.4.11:554/Streaming/Channels/101 --pgie nvinferserver -c config_triton.yml
{‘input’: [‘rtsp://admin:pass@10.128.4.11:554/Streaming/Channels/101’], ‘configfile’: ‘config_triton.yml’, ‘pgie’: ‘nvinferserver’, ‘no_display’: False, ‘file_loop’: False, ‘disable_probe’: False, ‘silent’: False}
Creating Pipeline

Creating streamux

Creating source_bin 0

Creating source bin
source-bin-00
Creating Pgie

Creating tiler

Creating nvvidconv

Creating nvosd

Is it Integrated GPU? : 0
Creating EGLSink

At least one of the sources is live
WARNING: Overriding infer-config batch-size 0 with number of sources 1

Adding elements to Pipeline

Linking elements in the Pipeline

Now playing…
0 : rtsp://admin:pass@10.128.4.11:554/Streaming/Channels/101
Starting pipeline

WARNING: infer_proto_utils.cpp:155 auto-update preprocess.normalize.scale_factor to 1.0000
INFO: infer_grpc_backend.cpp:170 TritonGrpcBackend id:1 initialized for model: yolo
Decodebin child added: source

Decodebin child added: decodebin0

Decodebin child added: rtph264depay0

Decodebin child added: h264parse0

Decodebin child added: capsfilter0

Decodebin child added: nvv4l2decoder0

In cb_newpad

gstname= video/x-raw
features= <Gst.CapsFeatures object at 0x73ab7ed1f6a0 (GstCapsFeatures at 0x73aa4c0bc920)>
Segmentation fault (core dumped)


infer_config {
unique_id: 1
gpu_ids: 0
max_batch_size: 1
backend {
triton {
model_name: “yolo”
version: -1
grpc {
url: “0.0.0.0:8001”
enable_cuda_buffer_sharing: true
}

  }
}

preprocess {
network_format: IMAGE_FORMAT_RGB
tensor_order: TENSOR_ORDER_LINEAR
tensor_name: “images”
frame_scaling_hw: FRAME_SCALING_HW_DEFAULT
frame_scaling_filter: 1
symmetric_padding: 1
maintain_aspect_ratio: 1
#normalize {
# scale_factor: 0.0039215697906911373
#channel_offsets: [0.0,0.0,0.0]
#}
}

postprocess {
labelfile_path: "./labels.txt"
detection {
  num_detected_classes: 80
       custom_parse_bbox_func: "NvDsInferParseYolo"
    nms {
    confidence_threshold: 0.25
    iou_threshold: 0.45
    topk: 300
    }
           }
     }

custom_lib {
path: “./libnvdsinfer_custom_impl_Yolo.so”
}
}

input_control {
process_mode: PROCESS_MODE_FULL_FRAME
operate_on_gie_id: -1
interval: 0
}

Could you do a preliminary analysis with gdb tool?

$gdb --args <your command>
$r
$bt

After running the python under gdb with library compile with debug symbols

gdb-log.txt (17.8 KB)

#0  decodeTensorYolo(float const*, float const*, float const*, uint const&, uint const&, uint const&, std::vector<float, std::allocator<float> > const&)
    (boxes=0x7fff2853f950, scores=0x0, classes=0x20221, outputSize=@0x7fff5fdff6c4: 84, netW=@0x5555560a76b4: 640, netH=@0x5555560a76b8: 640, preclusterThreshold=std::vector of length 80, capacity 80 = {...}) at nvdsparsebbox_Yolo.cpp:83
#1  0x00007fffacd8c7c9 in NvDsInferParseCustomYolo(std::vector<NvDsInferLayerInfo, std::allocator<NvDsInferLayerInfo> > const&, NvDsInferNetworkInfo const&, NvDsInferParseDetectionParams const&, std::vector<NvDsInferObjectDetectionInfo, std::allocator<NvDsInferObjectDetectionInfo> >&)
    (outputLayersInfo=std::vector of length 1, capacity 1 = {...}, networkInfo=..., detectionParams=..., objectList=std::vector of length 0, capacity 0) at nvdsparsebbox_Yolo.cpp:150
#2  0x00007fffacd8caef in NvDsInferParseYolo(std::vector<NvDsInferLayerInfo, std::allocator<NvDsInferLayerInfo> > const&, NvDsInferNetworkInfo const&, NvDsInferParseDetectionParams const&, std::vector<NvDsInferObjectDetectionInfo, std::allocator<NvDsInferObjectDetectionInfo> >&)
    (outputLayersInfo=std::vector of length 1, capacity 1 = {...}, networkInfo=..., detectionParams=..., objectList=std::vector of length 0, capacity 0) at nvdsparsebbox_Yolo.cpp:191
#3  0x00007fffea45c3bf in nvdsinferserver::DetectPostprocessor::fillDetectionOutput(std::vector<NvDsInferLayerInfo, std::allocator<NvDsInferLayerInfo> > const&, std::vector<NvDsInferObject, std::allocator<NvDsInferObject> >&) ()
    at ///opt/nvidia/deepstream/deepstream-7.0/lib/libnvds_infer_server.so
#4  0x00007fffea45c668 in nvdsinferserver::DetectPostprocessor::batchParse(std::vector<NvDsInferLayerInfo, std::allocator<NvDsInferLayerInfo> >&, std::vector<std::shared_ptr<nvdsinferserver::BaseBatchBuffer>, std::allocator<std::shared_ptr<nvdsinferserver::BaseBatchBuffer> > >, unsigned int, std::shared_ptr<nvdsinferserver::BaseBatchArray>&) () at ///opt/nvidia/deepstream/deepstream-7.0/lib/libnvds_infer_server.so
#5  0x00007fffea456417 in nvdsinferserver::Postprocessor::postHostImpl(std::shared_ptr<nvdsinferserver::BaseBatchArray>&, std::shared_ptr<nvdsinferserver::BaseBatchArray>&, std::shared_ptr<nvdsinferserver::CudaStream>&) ()
    at ///opt/nvidia/deepstream/deepstream-7.0/lib/libnvds_infer_server.so
#6  0x00007fffea43029f in nvdsinferserver::InferBaseContext::doPostHostProcess(std::shared_ptr<nvdsinferserver::BaseBatchArray>, std::function<void (NvDsInferStatus, std::shared_ptr<nvdsinferserver::BaseBatchArray>)>) ()
    at ///opt/nvidia/deepstream/deepstream-7.0/lib/libnvds_infer_server.so
#7  0x00007fffea4309fe in  () at ///opt/nvidia/deepstream/deepstream-7.0/lib/libnvds_infer_server.so
#8  0x00007fffea434dc3 in nvdsinferserver::BasePostprocessor::postCudaProcess(std::shared_ptr<nvdsinferserver::BaseBatchArray>, std::shared_ptr<nvdsinferserver::CudaStream>, std::function<void (NvDsInferStatus, std::shared_ptr<nvdsinferserver::BaseBatchArray>)>) ()
    at ///opt/nvidia/deepstream/deepstream-7.0/lib/libnvds_infer_server.so
#9  0x00007fffea47116c in std::_Function_handler<bool (std::tuple<std::shared_ptr<nvdsinferserver::BaseBatchArray>, std::shared_ptr<nvdsinferserver::CudaStream>, std::function<void (NvDsInferStatus, std::shared_ptr<nvdsinferserver::BaseBatchArray>)> >), nvdsinferserver::ThreadCudaPostprocessor<nvdsinferserver::DetectPostprocessor>::ThreadCudaPostprocessor<int, nvdsinferserver::config::DetectionParams const&>(int&&, nvdsinferserver::config::DetectionParams const&)::{lambda(std::tuple<std::shared_ptr<nvdsinferserver::BaseBatchArray>, std::shared_ptr<nvdsinferserver::CudaStream>, std::function<void (NvDsInferStatus, std::shared_ptr<nvdsinferserver::BaseBatchArray>)> >)#1}>::_M_invoke(std::_Any_data const&, std::tuple<std::shared_ptr<nvdsinferserver::BaseBatchArray>, std::shared_ptr<nvdsinferserver::CudaStream>, std::function<void (NvDsInferStatus, std::shared_ptr<nvdsinferserver::BaseBatchArray>)> >&&) () at ///opt/nvidia/deepstream/deepstream-7.0/lib/libnvds_infer_server.so
#10 0x00007fffea473a00 in nvdsinferserver::QueueThread<std::vector<std::tuple<std::shared_ptr<nvdsinferserver::BaseBatchArray>, std::shared_ptr--Type <RET> for more, q to quit, c to continue without paging--
<nvdsinferserver::CudaStream>, std::function<void (NvDsInferStatus, std::shared_ptr<nvdsinferserver::BaseBatchArray>)> >, std::allocator<std::tuple<std::shared_ptr<nvdsinferserver::BaseBatchArray>, std::shared_ptr<nvdsinferserver::CudaStream>, std::function<void (NvDsInferStatus, std::shared_ptr<nvdsinferserver::BaseBatchArray>)> > > > >::threadLoop() () at ///opt/nvidia/deepstream/deepstream-7.0/lib/libnvds_infer_server.so
#11 0x00007ffff5606253 in  () at /lib/x86_64-linux-gnu/libstdc++.so.6
#12 0x00007ffff7ce2ac3 in  () at /lib/x86_64-linux-gnu/libc.so.6
#13 0x00007ffff7d74850 in  () at /lib/x86_64-linux-gnu/libc.so.6

According to the log, it was a crash in the libnvdsinfer_custom_impl_Yolo.so. Could you add some log in your decodeTensorYolo API to analyze it yourself?

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.