TX2 sigsegv with patched libnvjpeg.so

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

Running R28.2.1

Downloaded patch from https://devtalk.nvidia.com/default/topic/1035699/jetson-tx1/premature-end-of-jpeg-file-tegra_multimedia-master-samples-06_jpeg_decode-/post/5263099/#5263099

A simple test fails with a SIGSEGV and sometimes a malloc- memory corruption error

gst-launch-1.0 videotestsrc ! nvjpegenc ! nvjpegdec ! fakesink

I renamed the the downloaded file to libnvjpeg.so and replaced the file found in: /usr/lib/aarch64-linux-gnu/tegra

The above command is merely a simple replication of the problem.
The real implementation uses jpeg to stream video across the network.
In my particular use case, that code is 3x faster than the equivalent h.264 implementation!
A functional nvjpdec would thus provide a significant improvement in our product.

2

Hi,
Please try the following pipeline to check if encoding and decoding are running.

$ gst-launch-1.0 videotestsrc ! nvjpegenc ! nvjpegdec ! video/x-raw ! xvimagesink
3

That also fails

gst-launch-1.0 videotestsrc ! nvjpegenc ! nvjpegdec ! video/x-raw ! xvimagesink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Caught SIGSEGV
4

A second test generated some more useful data

New clock: GstSystemClock
Caught SIGSEGV
#0  0x0000007f78083030 in __GI___poll (fds=0x669ec0, nfds=2,
#1  0x0000007f781843ac in ?? () from /lib/aarch64-linux-gnu/libglib-2.0.so.0
#2  0x00000000004fa300 in ?? ()
Spinning.  Please run 'gdb gst-launch-1.0 3065' to continue debugging, Ctrl-C to quit, or Ctrl-\ to dump core.

The XWindow is created and populated with the expected test image.
It obviously does not update

5

Hi,
We can observe this on r28.2.1 + the prebuilt lib
Will update once there is further finding on the issue.

6

Hi,
Please try attachment. It is valid for r28.2.1 and r28.3.
r28_3_TEST_libnvjpeg.zip (141 KB)

7

Thank you for the update

This test now works

gst-launch-1.0 videotestsrc ! nvjpegenc ! nvjpegdec ! video/x-raw ! xvimagesink

However, my original pipeline still fails with a SIGSEGV

gst-launch-1.0  v4l2src device=/dev/video0 ! 'video/x-raw, format=UYVY, width=1920, height=1080, framerate=60/1' ! nvvidconv ! 'video/x-raw(memory:NVMM), format=I420' ! nvjpegenc !  nvjpegdec ! video/x-raw ! nvoverlaysink

The pipeline does not fail if nvvidconv is replaced by videoconvert.
I can even remove the conversion step altogether

gst-launch-1.0  v4l2src device=/dev/video0 ! 'video/x-raw, format=I420, width=1920, height=1080, framerate=60/1' ! nvjpegenc !  nvjpegdec ! video/x-raw ! nvoverlaysink

However, the non nvvidconv implementation latency is about 50% higher.

8

I am still seeing a slow memory leak with this pipeline

gst-launch-1.0 videotestsrc ! nvjpegenc ! nvjpegdec ! video/x-raw ! nvoverlaysink

There is no leak with

gst-launch-1.0 videotestsrc ! nvjpegenc ! nvjpegdec ! video/x-raw ! fakesink

It seems there some issue with the integration of nvjpegdec with the other NVIDIA specific plugins.

My product requires nvoverlaysink so the behavior of xvimagesink is irrelevant.

9

Hi,
For continuous encoding, we would suggest use h264/h265 encoding. nvjpegenc supports one-shot encoding for ‘video/x-raw(memory:NVMM)’ buffer, but it hits issue while feeding NVMM buffers continuously.

Here is a similar post:
https://devtalk.nvidia.com/default/topic/1058439/deepstream-sdk/cannot-get-nvjpegenc-to-work-with-nvdsosd/post/5387623/#5387623

10

That is disappointing.

The jpeg stream works very well for one of my use cases, providing better latency and quality than h264.

Is there a mechanism to request that NVIDIA resolve this deficiency?

11

Hi,
Please try attachment for r28.2.1.
r28_2_1_TEST_libnvjpeg.zip (141 KB)

12

Thanks for the update, but that still leaks memory

13

Hi,
On r28.2.1 + r28_2_1_TEST_libnvjpeg.zip, we don’t observe the memory leak in running:

$ gst-launch-1.0 videotestsrc ! nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12' ! nvjpegenc ! nvjpegdec ! video/x-raw ! xvimagesink

Do you run another pipeline?

14

The pipeline leaks when xvimagesink is replaced by nvoverlaysink

The r28_2_1_TEST_libnvjpeg.zip does improve the leak.
The leak rate is only 58kbyte/minute.
The original library leaks hundreds of MBytes per minute.

My product requires nvoverlaysink.

15

This pipeline with r28.2.1 + r28_2_1_TEST_libnvjpeg.zip does not leak.

Note the second nvvidconv before the nvoverlaysink

gst-launch-1.0 videotestsrc ! nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12' ! nvjpegenc ! nvjpegdec ! 
video/x-raw ! nvvidconv ! nvoverlaysink

The pipeline still leaks very rapidly with the original library.

This is sufficient for me to proceed, thank you for the assistance.
When will the patched library be officially released?

16

My full test pipeline uses v4l2src and still leaks memory

gst-launch-1.0  v4l2src device=/dev/video0 ! 'video/x-raw, format=UYVY, width=1920, height=1080, framerate=60/1' ! nvvidconv ! 'video/x-raw(memory:NVMM), format=I420' ! nvjpegenc !  nvjpegdec ! video/x-raw ! nvvidconv ! nvoverlaysink sync=false

The rate of leakage can be changed by adjusting parts of the pipeline.
So far, this pipeline has the slowest leakage

gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw, format=I420, width=1920, height=1080, framerate=60/1 ! nvjpegenc ! nvjpegdec ! nvvidconv ! video/x-raw ! nvoverlaysink sync=false

But it still leaks and the latency is no better than h.264.
(The first pipeline is almost 3x faster than h.264, which makes the jpeg approach worthwhile)

17

Hi,
The leak is not seen with fakesink and xvimagesink, so the attached prebuilt libs for r28.2.1 and r28.3 are good.

We don’t observe leak in nvoverlaysink. You may try

# sysctl vm.drop_caches=3
18

Unfortunately that makes no difference to the memory leak.
I tried both the prebuilt and patched libs.

19

Hi,
If nvjpegdec does not bring better performance, please execute ‘$ sudo jetson_clocks’ and utilize jpegdec.

20

nvjpegdec provides significantly better performance (latency and power) , which is why I was testing it.

Your suggestion will increase both power and latency, probably beyond that of h264.
That would completely negate the utility of streaming jpeg images.