Latency/lag with multiple nvdrmvideosink outputs

• Hardware Platform (Jetson / GPU)
Xavier NX
• DeepStream Version
6.0
• JetPack Version (valid for Jetson only)
4.6
• TensorRT Version
8.0.1.6
• Issue Type( questions, new requirements, bugs)
Maybe a bug?

I am seeing an issue when I run a Gstreamer/Deepstream pipeline with two nvdrmvideosink outputs as opposed to just one, specifically the frames displayed to screen are laggy/have higher latency than expected. The following pipeline works as it should:

gst-launch-1.0 videotestsrc ! nvvideoconvert ! 'video/x-raw(memory:NVMM),width=1920,height=1080,format=NV12' ! tee name=disp_t disp_t. ! queue ! nvvideoconvert ! nvvidconv ! nvdrmvideosink conn-id=0 sync=false

with a single nvdrmvideosink output. However if I had another such output like so:

gst-launch-1.0 videotestsrc ! nvvideoconvert ! 'video/x-raw(memory:NVMM),width=1920,height=1080,format=NV12' ! tee name=disp_t disp_t. ! queue ! nvvideoconvert ! nvvidconv ! nvdrmvideosink conn-id=0 sync=false disp_t. ! queue ! nvvideoconvert ! nvvidconv ! nvdrmvideosink conn-id=1 sync=false

then there is a noticeable lag/latency issue and I get the following console messages repeating semi-regularly:

Pipeline is PREROLLING ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
cannot close gem buffer
Cannot close bo
cannot close gem buffer
Cannot close bo
cannot close gem buffer
Cannot close bo
...

When I use a different input to videotestsrc such as nvarguscamerasrc like in my real pipeline, this lag is very noticeably and a problem. I can sort of solve the issue by setting sync=true on the output sinks which will just drop frames slower than the clock and smooth out the video, but I’d like to solve the underlying issue if possible.

Does anyone have any idea what’s going on here? I suspect there is some buffer issue of some kind, though I thought with such a simple pipeline, the buffer should just be shared on both branches and displayed quickly, but that does not appear to be the case.

Small edit: This also leads to a seg fault after a little while (sometimes as short as a minute) with this double nvdrmvideosink output and the videotestsrc input. However I don’t see any seg faults when using nvarguscamerasrc which is a weird difference.

Can you attach the recorded video when the issue happens?

I tried different ways to capture video by adding a branch to the display tee to save to mp4, however the mp4 simply saves all frames to the video and the lag/latency which is present in the live view doesn’t show up in the saved video. I tried different parameters to try and capture it but couldn’t figure out if there was a way to do it. I’m also unsure if the problem which appears in the nvdrmvideosink branches occurs at all in the mp4 encoding branch since the buffers are treated differently.

If I had a way to capture my screen directly I might be able to produce a video but unfortunately I’m not set up for that at the moment.

Hi,
We don’t support dual display on Xavier NX developer kit with Jetpack 5 release. You are using Jetpack 4.6 and it would be great if you can try later Jetpack 4.6.4.

Not sure I fully follow - are you saying that both HDMI’s don’t work on Jetpack 5 or that the plugin nvdrmvideosink doesn’t work on Jetpack 5? Dual display works with Jetpack 4.6, it’s just that there is some kind of lag/latency introduced from adding the second nvdrmsink in the pipeline, I’m guessing because there is some buffer copy going on in the background slowing things down.

I will see if I can get set up to try 4.6.4.

This issue appears to be solved by setting nvdrmvideosink set-mode=1 rather than using the default of set-mode=0. Not sure exactly what’s going on, but from my understanding this parameter changes the behavior from automatically determining video stream resolution to pass on to the output to just passing on whatever resolution is entering the sink. So something about that automatic determination is creating lag when using two nvdrmvideosink elements vs just one. I tried with different monitors that accept 1920x1080 and still saw the issue, so I don’t think it’s dependent on the type of connected outputs, but I could be wrong.

This is with Jetpack 4.6, I never tried 4.6.4 though I expect the behavior to be the same.

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.