Is nvdrmvideosink improved on JetPack 5.1.3? / Is it possible to build nvoverlaysink on JetPack 5.1.3?

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

• Hardware Platform (Jetson / GPU)
Xavier NX
• DeepStream Version
6.0
• JetPack Version (valid for Jetson only)
4.6.4

Hi there, I have a had a great deal of trouble trying to use the plugin nvdrmvideosink for display in my company’s robotic surgery deepstream application. New issues keep arising, and whenever I fix one a new issue pops up.

My general question for anyone who has explicit knowledge of the plugin nvdrmvideosink and/or the commits to said plugin, is this plugin’s performance improved from JetPack 4.6.4 to JetPack 5.1.3?

If not, is it possible to build and use the deprecated plugin nvoverlaysink on JetPack 5.1.3? I understand it is deprecated, but this plugin did not cause us any issues before, and now nvdrmvideosink does.

To be clear about the issues we are facing in case they are helpful, here is some background information:

We are on JetPack 4.6.4, using Xavier NX, on a custom carrier board using a custom Yocto/BusyBox build. Our carrier board has 2 HDMI outputs which we need to display video to in RGBA format. We are going to move to JetPack 5.1.3 in the near future, but for now it is not possible, and so I am unable to test whether nvdrmvideosink is improved or not. A simplified version of our pipeline regarding the sink pieces looks something like:

            nvarguscamerasrc ee-mode=1 ee-strength=.1 tnr-mode=1 tnr-strength=0.1 wbmode=fluorescent ispdigitalgainrange='1 1' bufapi-version=1 
                ! video/x-raw(memory:NVMM),width=1920,height=1080,format=NV12 ! tee name=input_t
            input_t. ! other branch...
            input_t. ! other branch...
            input_t. ! queue ! nvvideoconvert ! video/x-raw,width=1920,height=1080,format=RGBA ! queue max-size-buffers=1 leaky=downstream ! custom plugin ! tee name=disp_t
                disp_t. ! queue ! nvdrmvideosink conn-id=0 plane-id=1 sync=false set-mode=0
                disp_t. ! queue ! nvdrmvideosink conn-id=1 plane-id=1 sync=false set-mode=0

The specific order of issues with nvdrmvideosink is as follows:

  1. Single nvdrmvideosink has no issues
  2. Adding a second nvdrmvideosink in parallel, with set-mode=0 causes increased latency of 20-40 ms, which is not acceptable for our live surgery video feed application (nvoverlaysink does not do this)
  3. Changing set-mode=1 improves the latency, but causes “tearing” along parts of the image
  4. Adding nvvideoconvert element upstream of nvdrmvideosink in the pipeline fixes the tearing, but then the program crashes randomly after 20-30 minutes (strange that it takes so long, and that it’s so random)

If anyone has knowledge of whether there is a way to improve the behavior of nvdrmvideosink I would love to know. If anyone has explicit knowledge of the nvdrmvideosink plugin and it’s code changes I would also love to know. Since it is closed source I cannot yet determine whether this is fixed in JetPack 5.1.3 or not, especially since we are on a custom carrier board and build. Alternatively, is it possible to build nvoverlaysink for JetPack 5.1.3, and if so, how can I do that?

3

I am checking!

4

Thank for the sharing! nvoverlaysink is deprecated.

  1. is there any abnormal printing? can you use gdb to get the the crash stack?
  2. there are many custom code in the pipeline. can you simplify the code to narrow down this issue?
5

Thanks for looking into it. Was there any updates to the plugin after JetPack 4.6.4?

  1. No abnormal printing, just crashes. Unfortunately I cannot use gdb at the moment since we are on this Linux BusyBox system with many missing utilities. I do not know if the crash is related to nvdrmvideosink directly or this is some issue with this plugin and our specific system (hardware or software).
  2. I will try to produce a more minimal example to see if you can reproduce on your end. Only thing is you will need to use 2 output monitors, both hdmi, in order for it to be comparable to our system. Since the dev kits don’t have 2 hdmis I am not sure whether you can test or not.
6

From 4.6.4 to 5.1.3 , there are many JetPack versions. please refer to this compatibility table. there are some bug fixes in the latter versions, which may affect the performance.

7

So there are bug fixes to the nvdrmvideosink code? Is it possible to know what those bug fixes are? Commit messages or anything similar could potentially be very helpful.

9

Is it possible to know what the bugs were that were fixed, even if not the details for the fix? Even just knowing that information could potentially be helpful. I’ve looked through various Jetpack and DeepStream release notes, but I haven’t been able to find anything specific to nvdrmvideosink.

Here is a pipeline which consistently crashes after 10-30 minutes:

gst-launch-1.0 nvarguscamerasrc ee-mode=1 ee-strength=.1 tnr-mode=1 tnr-strength=0.1 wbmode=fluorescent ispdigitalgainrange='1 1' bufapi-version=1 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,format=NV12' ! tee name=input_t input_t. ! nvvideoconvert ! 'video/x-raw,width=1920,height=1080,format=RGBA' ! nvvideoconvert ! tee name=disp_t  disp_t. ! queue ! nvdrmvideosink conn-id=0 sync=false set-mode=1 disp_t. ! queue ! nvdrmvideosink conn-id=1 sync=false set-mode=1

Not sure if you can test it out or not since you’ll have to have a board with 2 HDMI outputs and a camera to work with nvarguscamerasrc, but if you do it would be great to see whether you see the same crash.

I’ve also had the following pipeline crash on occasion, but it appears more resilient in the latest JP 4.6 build I’m on. This would let you test without nvarguscamerasrc:

gst-launch-1.0 videotestsrc ! tee name=input_t input_t. ! nvvideoconvert ! 'video/x-raw,width=1920,height=1080,format=RGBA' ! nvvideoconvert ! tee name=disp_t disp_t. ! queue ! nvdrmvideosink conn-id=0 sync=false set-mode=1 disp_t. ! queue ! nvdrmvideosink conn-id=1 sync=false set-mode=1

12

Hi,
We don’t support dual display on Jetpack 5 release. If you would like to use dual display, please stay on Jetpack 4 release. The nvdrmovideosink plugin on Jetpack 5 should work with single display.

13

Are you saying that dual display with nvdrmvideosink doesn’t work at all with Jetpack 5? Or just that the issues I’m seeing with dual display on Jetpack 4 just haven’t been fixed in Jetpack 5?

The motivation for this question was that there are issues with dual display on Jetpack 4, which I was hoping a higher version of Jetpack would fix. We’d like to move up versions, both to gain access to newer features and because one of our models has an issue in Jetpack 4 which is resolved in Jetpack 5. What about Jetpack 6, any ideas if there is improvement with dual display with nvdrmvideosink there?

Alternatively, is it possible for us to build/use nvoverlaysink in Jetpack 5? I understand it’s deprecated and will not receive further updates, but it works for us in Jetpack 4 and if we could just use it in higher versions that might work too.

I appreciate the help.

14

Hi,
The nvoverlaysink plugin is deprecated on Jetpack 5.

In DeepStream SDK, we use nvmultistreamtiler to composite the sources into single video plane and render out in single display. This is the setup well tested in SQA coverage and we would suggest run in this setup.

There’s a patch for nvdrmvideosink of Jetpack 4. Please apply it and give it a try:
Issue with nvdrmvideosink NULL state - #8 by DaneLLL

15

We’re unable to use a single display at the moment due to our use case. We have a monitor which we display to during surgery for the surgeon to view, and then we have a capture card which we connect to with a 2nd HDMI to record video. In the future we are planning on getting rid of the capture card and storing video a different way, but for now the hardware is unchangeable.

And sorry to ask again but I would love answers to my previous questions if possible (even if the answer is no):

  • I understand nvoverlaysink is deprecated in Jetpack 5 - does this mean it is impossible to use on Jetpack 5 entirely? Or is there a way for me to build manually and it could still work, just without future updates?
  • Is nvdrmvideosink potentially improved in Jetpack 5 or 6? It’s still not clear to me whether there are any changes at all, and I’m unable to test for the time being, and I’m unable to check myself since it’s all closed info. Such as the patch you mention - is that fixed in Jetpack 5 or 6?
16

Hi,

No, it is not possible to use nvoverlaysink on Jetpack 5

Yes, it is fixed on Jetpack 5. You can download source code of nvdrmvideosink and build it. The source code is in the package:

Jetson Linux | NVIDIA Developer
Driver Package (BSP) Sources

17

Got it on both accounts, thanks for letting me know.

18

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.