Deepstream_test3_app on DS 4.0 shows undefined behaviour

I have compiled and run the deepstream_test3_app successfully two or three times, but out of nothing, I am facing the following error:

0:00:02.541635659  2228 0x7f4d28004890 WARN                 nvinfer gstnvinfer.cpp:515:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]:log(): TensorRT was compiled against cuBLAS 10.1.0 but is linked against cuBLAS 10.2.0. This mismatch may potentially cause undefined behavior.
Decodebin child added: source
Decodebin child added: decodebin0
Running...
Decodebin child added: h264parse0
Decodebin child added: capsfilter0
Decodebin child added: nvv4l2decoder0
Seting bufapi_version

(videotest:2228): GLib-GObject-WARNING **: 08:06:32.687: g_object_set_is_valid_property: object class 'nvv4l2decoder' has no property named 'bufapi-version'

(videotest:2228): GStreamer-CRITICAL **: 08:06:32.688: gst_value_set_fraction: assertion 'denominator != 0' failed
In cb_newpad
in videoconvert caps = video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, framerate=(fraction)30/1, batch-size=(int)1, num-surfaces-per-frame=(int)1, format=(string)RGBA
Creating LL OSD context new
0:00:02.942199078  2228 0x7f4d203eb230 WARN                 nvinfer gstnvinfer.cpp:1815:gst_nvinfer_output_loop:<primary-nvinference-engine> error: Internal data stream error.
0:00:02.942221803  2228 0x7f4d203eb230 WARN                 nvinfer gstnvinfer.cpp:1815:gst_nvinfer_output_loop:<primary-nvinference-engine> error: streaming stopped, reason error (-5)

Is that because of the TensorRT linkage? I wonder why ‘nvv4l2decoder’ would not have ‘bufapi-version’

I also have that linkage warning, but the result is correct.

“bufapi-version” is the propoerty for tegra platform. It will also not affect result.

What’s your platfrom?
Can you check your setup pre-requisites:

  • Ubuntu 18.04
  • Gstreamer 1.14.1
  • NVIDIA driver 418+
  • CUDA 10.1
  • TensorRT 5.1.2 RC

For me on Ubuntu on AMDx64 platform it works correctly, but on Jetson Nano it crashes like this:

Now playing: file:///opt/nvidia/deepstream/deepstream-4.0/samples/streams/sample_1080p_h264.mp4,

Using winsys: x11 
Creating LL OSD context new
0:00:01.463463651  2149   0x55a2abe440 INFO                 nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]:initialize(): Trying to create engine from model files
0:00:01.463730063  2149   0x55a2abe440 WARN                 nvinfer gstnvinfer.cpp:515:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]:generateTRTModel(): INT8 not supported by platform. Trying FP16 mode.
0:02:16.055636805  2149   0x55a2abe440 INFO                 nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]:generateTRTModel(): Storing the serialized cuda engine to file at /opt/nvidia/deepstream/deepstream-4.0/samples/models/Primary_Detector/resnet10.caffemodel_b1_fp16.engine
Decodebin child added: source
Decodebin child added: decodebin0
Running...
Decodebin child added: qtdemux0
Decodebin child added: multiqueue0
Decodebin child added: h264parse0
Decodebin child added: capsfilter0
Decodebin child added: aacparse0
Decodebin child added: avdec_aac0
Decodebin child added: nvv4l2decoder0
Seting bufapi_version
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
In cb_newpad
In cb_newpad
Creating LL OSD context new
Frame Number = 0 Number of objects = 5 Vehicle Count = 3 Person Count = 2
0:02:16.788362444  2149   0x55a2660f20 WARN                 nvinfer gstnvinfer.cpp:1830:gst_nvinfer_output_loop:<primary-nvinference-engine> error: Internal data stream error.
0:02:16.788426560  2149   0x55a2660f20 WARN                 nvinfer gstnvinfer.cpp:1830:gst_nvinfer_output_loop:<primary-nvinference-engine> error: streaming stopped, reason error (-5)
ERROR from element primary-nvinference-engine: Internal data stream error.
Error details: /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvinfer/gstnvinfer.cpp(1830): gst_nvinfer_output_loop (): /GstPipeline:dstest3-pipeline/GstNvInfer:primary-nvinference-engine:
streaming stopped, reason error (-5)
Returned, stopping playback
Frame Number = 1 Number of objects = 5 Vehicle Count = 3 Person Count = 2
Frame Number = 2 Number of objects = 5 Vehicle Count = 3 Person Count = 2
Frame Number = 3 Number of objects = 6 Vehicle Count = 4 Person Count = 2
Deleting pipeline

Hi s.tivyakovenko,
Could you try to re-flash the system and install SDK components, DS4.0 through sdkmanager? We have verified the samples in each release. If you don’t make any modification, it should work well.

I am observing same problem again, now not on Jetson, but also on x86 pc also. Here is some more details:

I am running sample deepstream-test3 from deepstream-4.0_4.0.1-1_amd64.deb on ubuntu x64 machine with GTX 1080Ti videocard. Sometimes sample runs correctly, sometimes it crashes with SEGFAULT, sometimes it gives this error before finishing video shortly after starting:

cuCtxCreate failed with error(2) gst_eglglessink_cuda_init

I’ve launched the sample with valgrind and it has shown some suspicious errors like

  1. Warning: noted but unhandled ioctl 0x30000001 with no size/direction hints.
  2. Warning: set address range perms: large range [0xae000000, 0xcc000000) (noaccess)
  3. Conditional jump or move depends on uninitialised value(s)
  4. Address 0x64ebc900 is 96 bytes inside a block of size 120 free’d

Here is list of packages, installed to build this sample:

apt list --installed | grep cud

cuda-cudart-10-1/unknown,now 10.1.243-1 amd64 [installed,automatic]
cuda-cudart-dev-10-1/unknown,now 10.1.243-1 amd64 [installed,automatic]
cuda-driver-dev-10-1/unknown,now 10.1.243-1 amd64 [installed,automatic]
cuda-license-10-1/unknown,now 10.1.243-1 amd64 [installed,automatic]
cuda-npp-10-1/unknown,now 10.1.243-1 amd64 [installed,automatic]
cuda-npp-dev-10-1/unknown,now 10.1.243-1 amd64 [installed,automatic]
cuda-repo-ubuntu1804-10-1-local-10.1.243-418.87.00/now 1.0-1 amd64 [installed,local]
libcudnn7/now 7.6.4.38-1+cuda10.1 amd64 [installed,local]
libcudnn7-dev/now 7.6.4.38-1+cuda10.1 amd64 [installed,local]
libnvinfer-dev/unknown,now 5.1.5-1+cuda10.1 amd64 [installed,automatic]
libnvinfer5/unknown,now 5.1.5-1+cuda10.1 amd64 [installed,automatic]
nv-tensorrt-repo-ubuntu1804-cuda10.1-trt5.1.5.0-ga-20190427/now 1-1 amd64 [installed,local]

Can you please investigate this somehow?
I also attach output of valgrind: https://gist.github.com/stiv-yakovenko/0ecf9ebc11a080277b24c145f89d78ca

Hi,
For more information, please share the command you run on x86 PC.

I run it like this:

./deepstream-test3-app file:///opt/nvidia/deepstream/deepstream-4.0/samples/streams/sample_720p.mp4

I add valgrind before this command line to get log.

Any progress with this bug?

Hi,
We don’t observe the issue. Maybe you can try DS4.0.1

Did you run it with valgrind? Did you build sample or used prebuilt one? I have retested it recently on freshly flashed jetson nano.

Hi,
For more information, does it work without valgrind? Would like to know if it works in simply running deepstream-test3.

I am already using DS 4.0.1, just tested recently. If I use prebuilt deepstream-test3, it works. If I build it myself with make command, it crashes. I use the same version of gcc as it was used to build it in deepstream. It may also depend on type of input video file. This is a memory bug, it doesn’t reproduce itself 100% reliably, the best way to fix it is first of all fix errors reported by valgrind.

Hi,
For Jetson Nano, do you use sdcard module(A02) or emmc module(B01)? On A02 module, the storage can be limited to ~14GB, triggering certain memory issue. There is steps in FAQ to re-size SD card image.

Hi,
I’m using DS 4.0.1 on Ubuntu 18.04.3, and I had the same problem. I run deepstream-test1 and deepstream-test3, getting the same error: internal data stream error.

The error message(deepstream-test1):

ERROR from element primary-nvinference-engine: Internal data stream error.
Error details: gstnvinfer.cpp(1830): gst_nvinfer_output_loop (): /GstPipeline:dstest1-pipeline/GstNvInfer:primary-nvinference-engine:
streaming stopped, reason not-negotiated (-4)

Then I deleted the element primary-nvinference-engine, but the error occured on element h264-parser again.
I debugged with gst-launch-1.0 for it, and I found the way.

This command is ok:
gst-launch-1.0 filesrc location=channel0.H264 ! h264parse ! nvv4l2decoder ! nvvideoconvert ! fakesink

But this one throws internal data stream error:
gst-launch-1.0 filesrc location=channel0.H264 ! h264parse ! nvv4l2decoder ! nvvideoconvert ! nveglglessink

So I changed nveglglessink to fakesink, and then everything was ok.

I don’t know why. Anybody can explain for it? A lot of thanks!

SD card has nothing to do with these memory problems. I have 32G SSD.

Hi,
We have verified the following case on Nano, r32.2.1, DS4.0.1
Server pipeline:

$ ./test-launch "filesrc location=/opt/nvidia/deepstream/deepstream-4.0/samples/streams/sample_720p.mp4 ! qtdemux ! rtph264pay name=pay0 pt=96"

Client pipeline:

$ ./deepstream-test3-app rtsp://127.0.0.1:8554/test

Could you check if you can also run it? It should be good if we can align in running a same case.

Do not use ssh to debug the app.I used to have the same problem.