Image Artifacts on Jetson NX with JetPack 5.1.2 When Running GStreamer Pipeline

[alex318]

Hello,

I’m encountering an issue with my Jetson NX device running JetPack 5.1.2 when executing a GStreamer pipeline. The pipeline is designed to capture video from two sources, decode MJPEG, and save the frames to JPEG images, but I’m experiencing image artifacts (broken lines in the images).

The pipeline is as follows:

gst-launch-1.0 -vvv \
v4l2src do-timestamp=true device=/dev/video2 ! \
image/jpeg,framerate=60/1,width=1280,height=960 ! \
videorate ! \
image/jpeg,framerate=10/1 ! \
nvv4l2decoder mjpeg=1 ! \
nvvidconv flip-method=2 ! \
'video/x-raw(memory:NVMM),width=1280,height=960' ! \
nvjpegenc ! \
multifilesink location="C_2/frame_%05d.jpg" post-messages=true \
v4l2src do-timestamp=true device=/dev/video0 ! \
image/jpeg,framerate=60/1,width=1280,height=960 ! \
videorate ! \
image/jpeg,framerate=10/1 ! \
nvv4l2decoder mjpeg=1 ! \
nvvidconv flip-method=2 ! \
'video/x-raw(memory:NVMM),width=1280,height=960' ! \
nvjpegenc ! \
multifilesink location="C_1/frame_%05d.jpg" post-messages=true

When I run this pipeline on JetPack 5.1.2, I notice occasional image artifacts such as broken lines appearing in the images. However, when running the exact same pipeline on the same device but with JetPack 4.4.1, the issue doesn’t occur, and the images are saved without any artifacts.

I have also noted the following GStreamer version differences between the two JetPack versions:

• JetPack 4.4.1: GStreamer 1.14.5

• JetPack 5.1.2: GStreamer 1.16.3

The image example

I would appreciate any insights on the following:

1. Potential reasons for this behavior when using JetPack 5.1.2 and GStreamer 1.16.3.
2. Whether there are any known compatibility issues with this pipeline and JetPack 5.1.2.
3. Any possible pipeline modifications or updates that could resolve the artifact issue.

Thank you in advance for your assistance! I’m also happy to provide sample images showing the artifacts if that would help.

[Honey_Patouceul]

Hi,
I’m no longer publicly helping here, I also saw your post on gstreamer, I might help if possible, though you may better tell your case:

1. Why are you decoding MJPG with NVENC for re-encoding with NVJPG ? If you just want to save JPG files you wouldn’t need this.
2. It may also be a USB BW issue. Why do you use video0 and video4 ? Are there other video devices 1,2,3 ? If yes, there may be some USB BW allocation/or else issues, but I wouldn’t be able to further advice on this. If not, better have contiguous video0 and video1.
3. There are a lot of changes between JP4 and JP5, not only the gstreamer version.
4. You may try a single recording from one camera only and say if works.
5. multifilesink property post-messages might only be relevant for a gst app. With gst-launch this might not be relevant. Does disabling this help ?
6. You may try launching 2 different gst-launch processes for each camera and say if it gets better or worse.
7. You may say how frequent is the issue. (less or more than 65536 frames encoded frames ?)
8. You may try adding a queue after source.
9. You may try adding jpegparse after that queue or after videorate.
10. You may try converting into I420 format while rotating between nvv4l2decoder (that ouputs NV12) and nvjpegenc.
11. If the above advice (10) fails, try outputting I420 into system memory (video/x-raw) instead of NVMM memory.
12. You may also say what gives for both cameras (0 and 4):

v4l2-ctl -d0 --list-formats-ext
v4l2-ctl -d4 --list-formats-ext

Have fun and let me know!

[alex318]

First of all, I want to thank you for your answer.
I will try to respond to your points one by one.

1. Why are you decoding MJPG with NVENC for re-encoding with NVJPG ? If you just want to save JPG files you wouldn’t need this.

1. For my application, I need the output of the line 'video/x-raw(memory:NVMM),width=1280,height=960' !, where I detected the problem. I save it back just to visualize the issue.

2. It may also be a USB BW issue.

2. In my system, I have two cameras connected to USB under /dev/video*. I see four devices listed. After investigation, I found that two are virtual devices for internal use, so I utilize the real devices, which dynamically change from restart to restart and are not always video0 and video2. However, it may still be a USB issue, but only for JP5.1.2.

4. You may try a single recording from one camera only and say if works.

3. I tried this, and the issue persists, although the number of broken images is significantly lower.

5. multifilesink property post-messages might only be relevant for a gst app.

4. multifilesink is irrelevant in my case because I’m only using it to visualize the issue.

6. 1. You may try launching 2 different gst-launch processes for each camera and say if it gets better or worse.

5. I tried this, and the result was the same as running both cameras in a single process.

7. You may say how frequent is the issue. (less or more than 65536 frames encoded frames ?)

6. It occurs once every 100–300 frames at 10 FPS. It can be a single broken image with one broken line or a sequence of 5–10 broken images in a row, like in the example.

8. You may try adding a queue after source.
9. You may try adding jpegparse after that queue or after videorate.

7. Adding queue did not help. :(

10-11. You may try converting into I420 format

8. Changing the format to I420 had no effect on the outcome.

Additional:
I ran the system qv4l2 application and observed the same issue, so I think the root of the problem may lie in a v4l2 API that is shared between processes.

Also, I’d like to share some debug output I see after slightly adjusting the debugger log level:

0:00:00.881343834 806447 0xaaaadcd9f2a0 WARN            v4l2videodec gstv4l2videodec.c:2305:gst_v4l2_video_dec_decide_allocation:<nvv4l2decoder0> Duration invalid, not setting latency
0:00:00.893342282 806447 0xaaaadcd9f2a0 WARN          v4l2bufferpool gstv4l2bufferpool.c:1114:gst_v4l2_buffer_pool_start:<nvv4l2decoder0:pool:src> Uncertain or not enough buffers, enabling copy threshold
0:00:00.901646273 806447 0xaaaadcd9f980 WARN          v4l2bufferpool gstv4l2bufferpool.c:1565:gst_v4l2_buffer_pool_dqbuf:<nvv4l2decoder0:pool:src> Driver should never set v4l2_buffer.field to ANY
NvMMLiteBlockCreate : Block : BlockType = 1 
0:00:00.918878068 806447 0xaaaadcd9f240 WARN                    v4l2 gstv4l2object.c:4561:gst_v4l2_object_probe_caps:<nvv4l2decoder1:src> Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Unknown error -1
0:00:00.919118006 806447 0xaaaadcd9f240 WARN                    v4l2 gstv4l2object.c:2420:gst_v4l2_object_add_interlace_mode:0xaaaadcf16260 Failed to determine interlace mode
.
.
.
0:00:00.933118899 806447 0xaaaadcd9f240 WARN          v4l2bufferpool gstv4l2bufferpool.c:1114:gst_v4l2_buffer_pool_start:<nvv4l2decoder1:pool:src> Uncertain or not enough buffers, enabling copy threshold
0:00:00.940239715 806447 0xffff8807b060 WARN          v4l2bufferpool gstv4l2bufferpool.c:1565:gst_v4l2_buffer_pool_dqbuf:<nvv4l2decoder1:pool:src> Driver should never set v4l2_buffer.field to ANY
.
.
.
0:00:04.027783039 806447 0xaaaadcd9f860 WARN                 v4l2src gstv4l2src.c:978:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 3 - ts: 0:00:03.576535803
0:00:19.021223030 806447 0xaaaadcd9f860 WARN                 v4l2src gstv4l2src.c:978:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:18.570921027

[DaneLLL]

Hi,
We would suggest upgrade to Jetpack 5.1.4 and try again. Certain issues are discovered and fixed in later releases.

[alex318]

Hello,
Thank you for the response.
For now, I may not be able to immediately upgrade my device to 5.1.4, but I plan to do so in the near future and will update you with the results.