Segmentation Fault When Accessing Frame Data in Probe Function

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Hello NVIDIA DeepStream Community,

Environment:

  • DeepStream Version: 6.4
  • Platform: DGPU (not Jetson)
  • Docker Image: Yes
  • Sample Application: deepstream-imagedata-multistream (runs successfully)

Issue:

I have developed a DeepStream application for object detection on a DGPU. My pipeline includes queues for better throughput. The application runs well overall till I need to access the frame data through a probe function. When I add the following lines of code to the probe function at the source pad of the filter1 plugin, I encounter a segmentation fault:

# Getting Image data using nvbufsurface
# the input should be address of buffer and batch_id
n_frame = pyds.get_nvds_buf_surface(hash(gst_buffer), frame_meta.batch_id)
# convert python array into numpy array format in the copy mode.
frame_copy = np.array(n_frame, copy=True, order='C')

Error:

Segmentation fault (core dumped)

When I run my script with GDB:

(gdb) run
Starting program: /usr/bin/python3 hdetect.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffeffff640 (LWP 1399)]
[New Thread 0x7fffed7fe640 (LWP 1400)]
[New Thread 0x7fffeaffd640 (LWP 1401)]
[New Thread 0x7fffea7fc640 (LWP 1402)]
[New Thread 0x7fffe7ffb640 (LWP 1403)]
[New Thread 0x7fffe57fa640 (LWP 1404)]
[New Thread 0x7fffe2ff9640 (LWP 1405)]
[New Thread 0x7fffe07f8640 (LWP 1406)]
[New Thread 0x7fffddff7640 (LWP 1407)]
[New Thread 0x7fffdb7f6640 (LWP 1408)]
[New Thread 0x7fffd8ff5640 (LWP 1409)]
[New Thread 0x7fffd47f4640 (LWP 1410)]
[New Thread 0x7fffd3ff3640 (LWP 1411)]
[New Thread 0x7fffcf7f2640 (LWP 1412)]
[New Thread 0x7fffccff1640 (LWP 1413)]
[New Thread 0x7fffca7f0640 (LWP 1414)]
[New Thread 0x7fffc7fef640 (LWP 1415)]
[New Thread 0x7fffc57ee640 (LWP 1416)]
[New Thread 0x7fffc4fed640 (LWP 1417)]

Folder already exists: frames

[New Thread 0x7fffc03e3640 (LWP 1418)]
Updated pre-cluster-threshold to 0.5 and engine file to ../model_b2_gpu0_fp16.engine
Creating streammux

[New Thread 0x7fffb57ff640 (LWP 1419)]
At least one of the sources is live

Create and link source bins to streammux

Creating source_bin 0 

Creating source bin
source-bin-00
Creating source_bin 1 

Creating source bin
source-bin-01
Creating Pgie

[New Thread 0x7fff5ffff640 (LWP 1420)]
WARNING: Overriding infer-config batch-size 1  with number of sources 2 

Creating Tracker

Creating nvvidconv

Creating filter

Creating EGLSink

Creating tiler

Creating nvvidconv

Creating nvosd

Now playing...
0 :  <RTSP_URL1>
1 :   <RTSP_URL2>
Starting pipeline

[New Thread 0x7fff5cbba640 (LWP 1421)]
[New Thread 0x7fff529ff640 (LWP 1422)]
[New Thread 0x7fff51ffe640 (LWP 1423)]
gstnvtracker: Loading low-level lib at libnvds_nvmultiobjecttracker.so
[NvTrackerParams::getConfigRoot()] !!![WARNING] File doesn't exist. Will go ahead with default values
[NvTrackerParams::getConfigRoot()] !!![WARNING] File doesn't exist. Will go ahead with default values
[NvMultiObjectTracker] Initialized
[New Thread 0x7fff5155d640 (LWP 1424)]
[New Thread 0x7fff50d5c640 (LWP 1425)]
0:00:03.729632704  1390 0x7fffb8ab8d00 INFO                 nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:2092> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-6.4/sources/deepstream_python_apps/apps/HDetect_revised_1_NvDCF_Tracker/model_b2_gpu0_fp16.engine
INFO: ../nvdsinfer/nvdsinfer_model_builder.cpp:610 [FullDims Engine Info]: layers num: 4
0   INPUT  kFLOAT input           3x640x640       min: 1x3x640x640     opt: 2x3x640x640     Max: 2x3x640x640     
1   OUTPUT kFLOAT boxes           8400x4          min: 0               opt: 0               Max: 0               
2   OUTPUT kFLOAT scores          8400x1          min: 0               opt: 0               Max: 0               
3   OUTPUT kFLOAT classes         8400x1          min: 0               opt: 0               Max: 0               

0:00:03.823427493  1390 0x7fffb8ab8d00 INFO                 nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2195> [UID = 1]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-6.4/sources/deepstream_python_apps/apps/HDetect_revised_1_NvDCF_Tracker/model_b2_gpu0_fp16.engine
[New Thread 0x7fff1bffd640 (LWP 1426)]
[New Thread 0x7fff1b7fc640 (LWP 1427)]
[New Thread 0x7fff1affb640 (LWP 1428)]
0:00:03.825093882  1390 0x7fffb8ab8d00 INFO                 nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus:<primary-inference> [UID 1]: Load new model:pgie/yolov8_config.txt sucessfully
Decodebin child added: source 

[New Thread 0x7fff1a7fa640 (LWP 1429)]
Decodebin child added: source 

[New Thread 0x7fff19ff9640 (LWP 1430)]
[New Thread 0x7fff197f8640 (LWP 1431)]
[New Thread 0x7fff18ff7640 (LWP 1432)]
[New Thread 0x7ffef3fff640 (LWP 1433)]
[New Thread 0x7ffef37fe640 (LWP 1434)]
[New Thread 0x7ffef2ffd640 (LWP 1435)]
[Thread 0x7fffc4fed640 (LWP 1417) exited]
[Thread 0x7fffc57ee640 (LWP 1416) exited]
[Thread 0x7fffc7fef640 (LWP 1415) exited]
[Thread 0x7fffca7f0640 (LWP 1414) exited]
[Thread 0x7fffccff1640 (LWP 1413) exited]
[Thread 0x7fffcf7f2640 (LWP 1412) exited]
[Thread 0x7fffd3ff3640 (LWP 1411) exited]
[Thread 0x7fffd47f4640 (LWP 1410) exited]
[Thread 0x7fffd8ff5640 (LWP 1409) exited]
[Thread 0x7fffdb7f6640 (LWP 1408) exited]
[Thread 0x7fffddff7640 (LWP 1407) exited]
[Thread 0x7fffe07f8640 (LWP 1406) exited]
[Thread 0x7fffe2ff9640 (LWP 1405) exited]
[Thread 0x7fffe57fa640 (LWP 1404) exited]
[Thread 0x7fffe7ffb640 (LWP 1403) exited]
[Thread 0x7fffea7fc640 (LWP 1402) exited]
[Thread 0x7fffeaffd640 (LWP 1401) exited]
[Thread 0x7fffed7fe640 (LWP 1400) exited]
[Thread 0x7fffeffff640 (LWP 1399) exited]
[Detaching after fork from child process 1436]
[New Thread 0x7fffc4fed640 (LWP 1437)]
[New Thread 0x7fffc57ee640 (LWP 1438)]
[New Thread 0x7fffc7fef640 (LWP 1439)]
[New Thread 0x7fffca7f0640 (LWP 1440)]
[New Thread 0x7fffed7fe640 (LWP 1441)]
[New Thread 0x7fffeaffd640 (LWP 1442)]
[New Thread 0x7fffea7fc640 (LWP 1443)]
[New Thread 0x7fffe7ffb640 (LWP 1444)]
[New Thread 0x7fffe57fa640 (LWP 1445)]
[New Thread 0x7fffe2ff9640 (LWP 1446)]
[New Thread 0x7fffe07f8640 (LWP 1447)]
[New Thread 0x7fffddff7640 (LWP 1448)]
[New Thread 0x7fffdb7f6640 (LWP 1449)]
Decodebin child added: decodebin0 

Decodebin child added: decodebin1 

Decodebin child added: rtppcmadepay0 

Decodebin child added: rtph264depay0 

Decodebin child added: alawdec0 

Decodebin child added: h264parse0 

Decodebin child added: capsfilter0 

In cb_newpad

Error: Decodebin did not pick nvidia decoder plugin.
Decodebin child added: nvv4l2decoder0 

[New Thread 0x7fffcf7f2640 (LWP 1450)]
Warning: Color primaries 5 not present and will be treated BT.601
In cb_newpad

[New Thread 0x7fffccff1640 (LWP 1451)]
[New Thread 0x7ffef27fc640 (LWP 1452)]
[New Thread 0x7ffef1ffb640 (LWP 1453)]
[New Thread 0x7ffef17fa640 (LWP 1454)]
[New Thread 0x7ffef0ff9640 (LWP 1455)]
[Thread 0x7ffef2ffd640 (LWP 1435) exited]
[New Thread 0x7ffef2ffd640 (LWP 1456)]
[New Thread 0x7ffeb519a640 (LWP 1457)]
[New Thread 0x7ffeb4999640 (LWP 1458)]

Thread 28 "python3" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fff50d5c640 (LWP 1425)]
0x00007ffff7dee9cd in ?? () from /usr/lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0  0x00007ffff7dee9cd in  () at /usr/lib/x86_64-linux-gnu/libc.so.6
#1  0x00007ffff2da0cc2 in _contig_to_contig ()
    at /usr/local/lib/python3.10/dist-packages/numpy/core/_multiarray_umath.cpython-310-x86_64-linux-gnu.so
#2  0x00007ffff2d102f6 in raw_array_assign_array ()
    at /usr/local/lib/python3.10/dist-packages/numpy/core/_multiarray_umath.cpython-310-x86_64-linux-gnu.so
#3  0x00007ffff2d10fb7 in PyArray_AssignArray ()
    at /usr/local/lib/python3.10/dist-packages/numpy/core/_multiarray_umath.cpython-310-x86_64-linux-gnu.so
#4  0x00007ffff2d1ef25 in PyArray_NewCopy ()
    at /usr/local/lib/python3.10/dist-packages/numpy/core/_multiarray_umath.cpython-310-x86_64-linux-gnu.so
#5  0x00007ffff2dccb51 in array_array ()
    at /usr/local/lib/python3.10/dist-packages/numpy/core/_multiarray_umath.cpython-310-x86_64-linux-gnu.so
#6  0x00005555556a199b in  ()
#7  0x000055555569853c in _PyEval_EvalFrameDefault ()
#8  0x00005555556bc93e in  ()
#9  0x00007ffff783d54f in  ()
    at /usr/lib/python3/dist-packages/gi/_gi.cpython-310-x86_64-linux-gnu.so
#10 0x00007ffff76427ec in  () at /usr/lib/x86_64-linux-gnu/libffi.so.8
#11 0x00007ffff7643050 in  () at /usr/lib/x86_64-linux-gnu/libffi.so.8
#12 0x00007ffff6f334a6 in  ()
    at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#13 0x00007ffff7705626 in g_hook_list_marshal ()
    at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#14 0x00007ffff6f33a55 in  ()
    at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#15 0x00007ffff6f38c32 in  ()
    at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#16 0x00007ffff6f3918e in gst_pad_push ()
    at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#17 0x00007ffff532d21f in  () at /usr/lib/x86_64-linux-gnu/libgstbase-1.0.so.0
#18 0x00007ffff6f357cd in  ()
    at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#19 0x00007ffff6f38d69 in  ()
    at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#20 0x00007ffff6f3918e in gst_pad_push ()
    at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#21 0x00007ffff532d21f in  () at /usr/lib/x86_64-linux-gnu/libgstbase-1.0.so.0
#22 0x00007ffff6f357cd in  ()
    at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#23 0x00007ffff6f38d69 in  ()
    at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#24 0x00007ffff6f3918e in gst_pad_push ()
--Type <RET> for more, q to quit, c to continue without paging--

Context:

  • I am working with DeepStream 6.4 on a Docker image.
  • I can run the deepstream-imagedata-multistream sample, which accesses image data on the pipeline through a probe function without any problem.
  • My application includes queues in the pipeline for better throughput, and I add the probe to the source pad of the filter1 plugin.
  • When I add the lines of code to access the image data, I receive a segmentation fault.

Request:

Could you please provide insights into:

  1. Probable reasons for this segmentation fault when accessing nvbufsurface and converting it to a NumPy array.
  2. Any known issues or limitations related to the usage of pyds.get_nvds_buf_surface and NumPy array conversion in DeepStream 6.4.

Any help or pointers would be greatly appreciated. Thank you!

3
  1. to narrow down this issue, could you share the whole code modifications?
  2. tiler_sink_pad_buffer_probe is added on tiler’s sink. if using tiler_sink_pad_buffer_probe instead of adding new function, will the custom code crash? since the origin code runs well, could you simplify the code to check which modification will cause the error?
4

Thank you for your response.

I simplified my project as you suggested and identified the issue. It was due to forgetting to set CUDA unified memory for one of the nvvidconv elements. After correcting this, the segmentation fault no longer occurs.

Additionally, I can confirm that placing the probe at the source pad of the filter1 plugin to access image data works correctly.

Thank you again for your help.

5

Thanks for the update! Is this still an DeepStream issue to support? Thanks!

6

Thank you for your continued support. Regarding the initial issue, I don’t have any other problems.

However, I want to utilize Triton for the first time in my project. Currently, my pipeline works properly using nvinfer, but I would like to benefit from Triton. I am using the 6.4-triton-multiarch Docker image of DeepStream 6.4 on a DGPU (not Jetson).

Could you guide me on the initial preparation for Triton before using it with this Docker image? Additionally, is there a test application that can help me get started with Triton?

Thank you again for all your help. If I need to bring this up in a separate topic, please let me know.

7

first 6.4-triton-multiarch already has triton environment. DeepStream uses nvinferserver plugin to do interaction with triton. please refer to deepstream-test1 for nvinferserver sample. especially you need to disable nvinfer and enable nvinferserver in dstest1_config.yml.
please open a new topic if you encounter a new DS issue. Thanks!

8

Sure,
Once again thanks a lot for your support.

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.