Question about nvv4l2decoder element

Hi, I have done some test on jetson-utils video-viewer with Nsight System.
And I found that status of plugin nvv4l2decoder is quite different:
When it is 4k, there is long time block ‘ppoll’ in every period.


But when it is 8k, instead of block ‘ppoll’, there is block ‘ioctl’.

I wonder why this difference occurs.

And we can see from the 8k screenshot, that when process one frame, the next frame is arrived.

Here are the data files:

video-viewer-4k.nsys-rep (3.3 MB)

video-viewer-8k-slow.nsys-rep (8.0 MB)

Meanwhile, I have done test with gst-launch-1.0 with 4k & 8k.
Both of them can not get a smooth video.
I can not understand the timeline of this program, it is more complicated.
nvv4l2decoder in gst-launch-1.0 is not similar with that in video-viewer .

The parameter I used is like:
rtspsrc protocols=tcp location=rtsp://10.98.32.1/live_stream latency=0 ! rtph265depay ! h265parse ! nvv4l2decoder ! nvvidconv ! video/x-raw,format=BGRx ! nv3dsink

Here are the data files:

gst-4k.nsys-rep (3.5 MB)
gst-8k.nsys-rep (3.7 MB)

I wonder why nvv4l2decoder element has different behavior, and how to get high resolution image correctly with gstreamer?
Thanks.

Hi,
For decoding 8K resolution, it takes significant usage of decoder, so it may take more time in decoding the frames. By default the decoder runs in blocking mode so it is blocked in queuing buffers in output plane.

@DaneLLL Hi, thanks for your reply.
Does this means that 8k29.97 5Mbps H265 stream can not be decoded real time?

I see the document of Xavier NX Series says support for H265 is up to 8K30 240Mbps. But I can not find the specific parameter for Orin, just 8K30 on the web page.
https://docs.nvidia.com/jetson/archives/r34.1/DeveloperGuide/text/SO/JetsonXavierNxSeries.html#multimedia

Hi,
For decoding 8K content on Orin 64GB, it should be able to achieve 30fps by running the command:

$ gst-launch-1.0 -v rtspsrc protocols=tcp location=rtsp://10.98.32.1/live_stream latency=0 ! rtph265depay ! h265parse ! nvv4l2decoder enable-max-performance=1 ! fpsdisplaysink text-overlay=0 video-sink=fakesink sync=0

But the following command may not achieve 30fps:

$ gst-launch-1.0 -v rtspsrc protocols=tcp location=rtsp://10.98.32.1/live_stream latency=0 ! rtph265depay ! h265parse ! nvv4l2decoder enable-max-performance=1 ! nvvidconv ! video/x-raw,format=BGRx ! fpsdisplaysink text-overlay=0 video-sink=fakesink sync=0

Since there is additional memory copy in

... ! nvvidconv ! video/x-raw,format=BGRx

In the step, it converts decoded NVMM buffer(in I420) to BGRx, and then copy to CPU buffer. It occupies significant CPU loading and can be performance bottleneck.

Hi, @DaneLLL , Thanks for you reply.
I added ! nvvidconv ! video/x-raw,format=BGRx because image could not be shown with out this node.

In the document: Accelerated GStreamer — Jetson Linux<br/>Developer Guide 34.1 documentation
$ gst-launch-1.0 filesrc location=<filename_h265.mp4> ! \ qtdemux ! queue ! h265parse ! nvv4l2decoder ! nv3dsink -e
Node nvvidconv is not needed when using nv3dsink to display image.

But the result I tested is failed.

Have you met this problem?

Hi,
There are sample files in DeepStream SDK. Please try and see if you can run this command:

$ gst-launch-1.0 filesrc location= /opt/nvidia/deepstream/deepstream-6.1/samples/streams/sample_1080p_h265.mp4 ! qtdemux ! queue ! h265parse ! nvv4l2decoder ! nv3dsink -e

Hi, @DaneLLL
This 2K file can be smoothly played.

And I changed camera resolution to 2k, using
gst-launch-1.0 -v rtspsrc protocols=tcp location=rtsp://10.98.32.1/live_stream latency=0 ! rtph265depay ! h265parse ! nvv4l2decoder enable-max-performance=1 ! nv3dsink
still there was no image showed.
And here is the log.
test_Gst_2k_0922.txt (72.3 KB)

Then I add ! nvvidconv ! 'video/x-raw(memory:NVMM), format=(string)NV12' before nv3dsink, 2k image can be shown correctly .

Hi,
Please check if this works:

gst-launch-1.0 -v rtspsrc protocols=tcp location=rtsp://10.98.32.1/live_stream latency=0 ! rtph265depay ! h265parse ! nvv4l2decoder enable-max-performance=1 ! 'video/x-raw(memory:NVMM)' ! nv3dsink

gst-launch-1.0 -v rtspsrc protocols=tcp location=rtsp://10.98.32.1/live_stream latency=0 ! rtph265depay ! h265parse ! nvv4l2decoder enable-max-performance=1 ! 'video/x-raw(memory:NVMM)' ! nv3dsink
could not show image.
Here is log:
test_Gst_8k_0922.txt (58.9 KB)

gst-launch-1.0 -v rtspsrc protocols=tcp location=rtsp://10.98.32.1/live_stream latency=0 ! rtph265depay ! h265parse ! nvv4l2decoder enable-max-performance=1 ! nvvidconv ! 'video/x-raw(memory:NVMM), format=(string)NV12' ! nv3dsink
could show image.

And when testing using /opt/nvidia/deepstream/deepstream-6.1/samples/streams/sample_1080p_h265.mp4, the format is NV12.

It seems that format P010_10LE is not supported by nv3dsink?
I don’t know why nvv4l2decoder sometimes outputs NV12, but sometimes outputs P010_10LE.
Maybe reason is the bit width of the source?
The 2k sample file is 8bit, and rtsp stream I have is 10bit.


And I found out by using
gst-launch-1.0 -v rtspsrc protocols=tcp location=rtsp://10.98.32.1/live_stream latency=0 ! rtph265depay ! h265parse ! nvv4l2decoder enable-max-performance=1 ! nvvidconv ! 'video/x-raw(memory:NVMM), format=(string)NV12' ! nv3dsink
sometimes 8k video could be shown smoothly after about 10 seconds, but most of the time, it went stuck like this:

Hi,
Did not notice the h265 stream can be 10-bit YUV. You are correct the nv3dsink does not support 10-bit YUV and has to convert the frame data to 8-bit YUV for rendering.

Please execute sudo nvpmodel -m 0, sudo jetson_clocks, and apply this:
Nvvideoconvert issue, nvvideoconvert in DS4 is better than Ds5? - #3 by DaneLLL

And run

gst-launch-1.0 -v rtspsrc protocols=tcp location=rtsp://10.98.32.1/live_stream latency=0 ! rtph265depay ! h265parse ! nvv4l2decoder enable-max-performance=1 ! nvvidconv ! 'video/x-raw(memory:NVMM), format=(string)NV12' ! fpsdisplaysink text-overlay=0 video-sink=nv3dsink sync=0

See if you can achieve target frame rate.

Hi, @DaneLLL

I have tried to do like this:

  1. Run sudo nvpmodel -m 0

  2. Run sudo jetson_clocks

  3. Try to find VIC in /sys/device, but I cound not find it in the place in your link (/sys/devices/13e10000.host1x/15340000.vic/power/control).

    I found VIC in other place:
    /sys/devices/platform/13e40000.host1x/15340000.vic
    (I am not sure if this is the right one)
    So I run
    echo on > /sys/devices/platform/13e40000.host1x/15340000.vic/power/control
    The result was permission denied.
    Then I trid to use sudo, the result was the same.


    And if try to open this folder, system froze.

  4. I found another post of yours about VIC:
    Camera's frame rate unstable - #24 by DaneLLL
    And then edit /etc/systemd/nv.sh (change VIC folder path)

After these steps, I am not sure that DFS is disabled correctly or not.

Hi,
Please execute $ sudo su to be root user and then set the nodes.

Thanks for the reply.

So I run:

  1. be root user

    sudo su

  2. sudo nvpmodel -m 0

  3. sudo jetson_clocks

  4. Set userspace governor

    echo on > /sys/devices/platform/13e40000.host1x/15340000.vic/power/control

  5. Set max_freq

    cat /sys/devices/13e10000.host1x/15340000.vic/devfreq/15340000.vic/available_frequencies

    echo 729600000 > /sys/devices/13e10000.host1x/15340000.vic/devfreq/15340000.vic/max_freq

  6. Set target frequency

    echo 729600000 > /sys/devices/13e10000.host1x/15340000.vic/devfreq/15340000.vic/userspace/set_freq

  7. exit root user

    exit

This time all settings run OK.
Is this setting only need once or everytime after boot?


And I am not sure if this nv.sh editing is needed?

Hi,
For making it a persistent setting, please refer to the link to modify nv.sh to enable it at maximum clock:
Camera's frame rate unstable - #24 by DaneLLL

By default VIC engine is set to dynamic frequency scaling.

I am not sure how to edit this nv.sh file.
I just added the text you provided in the link into line126 ~ 162.

I feel that the edited .sh is not worked.

nv.sh (15.6 KB)

Here is the file I edited.

And I noticed that you have add sync = 0 at last.
If I add sync = 0 after nv3dsink, 8K images can be shown smoothly even without setting VIC parameter.
I could not find parameter of nv3dsink from the document.
Could please introduce why add this parameter?

Hi,
It is a mechanism in gstreamer frameworks. Please take a look at
GstBaseSink

Thanks for the reply.
I will check the link.

Btw, is nv.sh I edited right?

Hi, @DaneLLL ,
I use sudo tegrastats to check the frequency of VIC.
The result shows that commands for VIC actually work (115->729).
But nv.sh is not work (still 115).

normal boot:

after run commands:

boot with edited nv.sh:

Hi,
It should work if nv.sh is revised correctly. Please check it further if you are able to spend some time.

Another solution is to keep nv.sh untouched and run a late script to overwrite the setting in nv.sh. Please refer to Jetson Nano FAQ

Q: How to autorun an app or commands at startup?