TX2 Camera convert/encode using Multimedia API issue

Hi,
I’m a newbie for Jetson products.

Recently, I’m trying to capture/pixel-format-converting/encoding on frame units. (To meet the conditions for finally streaming)
First, the camera interface I use is PCIe. (V4L2 support)

So, I’m implementing code through samples(12, etc… ) and API documents.

  • Capture: using ioctl (MMAP)
  • Converting: ouput plane (MMAP), capture plane (MMAP)
  • Encoding: ouput plane (MMAP), capture plane (MMAP)

However, I am blocked in the part of encoding after converting.

How can I connect the “render_dmabuf_fd” obtained through NvBufferTransform() to “NvBuffer” that NVVideoEncoder() can handle?

I tried to find similar content in this forum, but I still have failed.
I uploaded the code I am implementing.
The part that encoding is required to be TODO on the 620 line.
camera_v4l2_cuda.cpp (22.3 KB)

Can anyone please help me resolve this issue?
Thank you.

Hi,
There is a patch for 12_camera_v4l2_cuda + NvVideoEncoder. Please take a look at
tegra multimedia samples not working properly - #11 by DaneLLL

Thanks for your answer.
I will try to implement it with the code and share the results with you.

Hi, @DaneLLL

I tried to implement the code by referring to patch file and original sample.12 of 32.2. (I’m using 32.5 version.)
Because of the current hardware specification, I must use MMAP in the capture plane. (Not available DMA)
Therefore, I wrote the code as a condition that does not use DMA. (request_camera_buff_mmap() in use)
And I used Raw2NVBuffer() instead of NvBufferMemSyncForDevice() in start_capture(). (Because it is MMAP)

First of all, if I use Raw2NVBuffer() and NvBufferMemSyncForDevice(), all the following events are generated, blocked, and backtrace information is as follows.

#0 0x0000007fb7f732a4 in futex_wait_cancelable (private=, expected=0, futex_word=0x555561e308) at …/sysdeps/unix/sysv/linux/futex-internal.h:88
#1 0x0000007fb7f732a4 in __pthread_cond_wait_common (abstime=0x0, mutex=0x555561e2b0, cond=0x555561e2e0) at pthread_cond_wait.c:502
#2 0x0000007fb7f732a4 in __pthread_cond_wait (cond=0x555561e2e0, mutex=0x555561e2b0)
at pthread_cond_wait.c:655
#3 0x0000007fb75b7fdc in () at /usr/lib/aarch64-linux-gnu/tegra/libnvos.so
#4 0x0000007fb4d24b54 in TegraV4L2_Poll_OPlane ()
at /usr/lib/aarch64-linux-gnu/tegra/libtegrav4l2.so
#5 0x0000007fb4d8a088 in plugin_ioctl ()
at /usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvvideocodec.so
#6 0x0000007fb7e555c0 in v4l2_ioctl () at /usr/lib/aarch64-linux-gnu/libv4l2.so.0
#7 0x000000555558f3d8 in NvV4l2ElementPlane::dqBuffer(v4l2_buffer&, NvBuffer**, NvBuffer**, unsigned int) ()
#8 0x000000555555ccd8 in start_capture(context_t*) ()
#9 0x000000555555b0e0 in main ()

Also, if I use NvBufferMemSyncForDevice(), the following error is displayed.

NvMapMemCacheMaint:1075334668 failed [99]
Got EOS, exiting…
[Thread 0x7fa58211d0 (LWP 9665) exited]
NvMapMemCacheMaint:1075334668 failed [99]
NvMapMemCacheMaint:1075334668 failed [99]
NvMapMemCacheMaint:1075334668 failed [99]
NvMapMemCacheMaint:1075334668 failed [99]
NvMapMemCacheMaint:1075334668 failed [99]
NvMapMemCacheMaint:1075334668 failed [99]

Below is the code I modified.
camera_v4l2_cuda.cpp (24.8 KB)
If the use according to MMAP/DMA is wrong, please let me know.
Thank you.

Hi,
Should be something wrong in the app. Please check if you can capturing frames and encoding dummy NvBuffer separately. May do

  1. Capture frames through v4l2 but do not copy to NvBuffer
  2. Feed dummy NvBuffer(empty buffer; no valid data) to NvVideoEncoder.

If capturing frames and encoding dummy NvBuffer work, may try to call Raw2NVBuffer().

As comparison please also try gstreamer commands. Please try the command:

$ gst-launch-1.0 v4l2src ! video/x-raw,width=640,height=480,format=NV12,framerate=30/1 ! fakesink

If it works, please try

$ gst-launch-1.0 v4l2src ! video/x-raw,width=640,height=480,format=NV12,framerate=30/1 ! nvvidconv ! nvv4l2h264enc ! h264parse ! matroskamux ! filesink location=a.mkv
1 Like

Hi,
The first command returns errors… (Note, it works with capture and converting with MMAP without encoding to output to EGLDISPLAY. - with sample.12)

$ gst-launch-1.0 v4l2src ! video/x-raw,width=1280,height=720,format=NV12,framerate=30/1 ! fakesink
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
Additional debug info:
gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason not-negotiated (-4)
ERROR: pipeline doesn't want to preroll.
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

Below is the current video format.

$ v4l2-ctl --get-fmt-video
Format Video Capture:
        Width/Height      : 1280/720
        Pixel Format      : 'NV12'
        Field             : None
        Bytes per Line    : 1280
        Size Image        : 921600
        Colorspace        : SMPTE 170M
        Transfer Function : Default (maps to Rec. 709)
        YCbCr/HSV Encoding: Default (maps to ITU-R 601)
        Quantization      : Default (maps to Limited Range)

In fact, I first tested Gstreamer, but it did not work. So I was implementing directly with Multimedia API.
Fortunately, like notes on the above, I confirmed that “Capture → Converting → EgLDisplay” through Sample.12 modifications, and were in implementing parts for encoding.

Note, below is the entire format of the capture card I use. (PCIe interface) .

> $ v4l2-ctl --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
        Index       : 0
        Type        : Video Capture
        Pixel Format: 'MPEG' (compressed)
        Name        : H264 Encoded Stream
                Size: Discrete 640x480
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 720x480
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 720x576
                        Interval: Discrete 0.020s (50.000 fps)
                Size: Discrete 800x600
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1024x768
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1280x720
                        Interval: Discrete 0.020s (50.000 fps)
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1280x800
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1280x1024
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1360x768
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1440x900
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1920x1080
                        Interval: Discrete 0.067s (15.000 fps)
                        Interval: Discrete 0.042s (24.000 fps)
                        Interval: Discrete 0.040s (25.000 fps)
                        Interval: Discrete 0.033s (30.000 fps)
                        Interval: Discrete 0.020s (50.000 fps)
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1680x1050
                        Interval: Discrete 0.067s (15.000 fps)
                        Interval: Discrete 0.042s (24.000 fps)
                        Interval: Discrete 0.040s (25.000 fps)
                        Interval: Discrete 0.033s (30.000 fps)
                        Interval: Discrete 0.020s (50.000 fps)
                        Interval: Discrete 0.017s (60.000 fps)

        Index       : 1
        Type        : Video Capture
        Pixel Format: 'YV12'
        Name        : YVU 4:2:0 12bpp
                Size: Discrete 640x480
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 720x480
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 720x576
                        Interval: Discrete 0.020s (50.000 fps)
                Size: Discrete 800x600
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1024x768
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1280x720
                        Interval: Discrete 0.020s (50.000 fps)
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1280x800
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1280x1024
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1360x768
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1440x900
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1920x1080
                        Interval: Discrete 0.067s (15.000 fps)
                        Interval: Discrete 0.042s (24.000 fps)
                        Interval: Discrete 0.040s (25.000 fps)
                        Interval: Discrete 0.033s (30.000 fps)
                        Interval: Discrete 0.020s (50.000 fps)
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1680x1050
                        Interval: Discrete 0.067s (15.000 fps)
                        Interval: Discrete 0.042s (24.000 fps)
                        Interval: Discrete 0.040s (25.000 fps)
                        Interval: Discrete 0.033s (30.000 fps)
                        Interval: Discrete 0.020s (50.000 fps)
                        Interval: Discrete 0.017s (60.000 fps)

        Index       : 2
        Type        : Video Capture
        Pixel Format: 'YU12'
        Name        : YUV 4:2:0 12bpp
                Size: Discrete 640x480
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 720x480
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 720x576
                        Interval: Discrete 0.020s (50.000 fps)
                Size: Discrete 800x600
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1024x768
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1280x720
                        Interval: Discrete 0.020s (50.000 fps)
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1280x800
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1280x1024
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1360x768
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1440x900
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1920x1080
                        Interval: Discrete 0.067s (15.000 fps)
                        Interval: Discrete 0.042s (24.000 fps)
                        Interval: Discrete 0.040s (25.000 fps)
                        Interval: Discrete 0.033s (30.000 fps)
                        Interval: Discrete 0.020s (50.000 fps)
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1680x1050
                        Interval: Discrete 0.067s (15.000 fps)
                        Interval: Discrete 0.042s (24.000 fps)
                        Interval: Discrete 0.040s (25.000 fps)
                        Interval: Discrete 0.033s (30.000 fps)
                        Interval: Discrete 0.020s (50.000 fps)
                        Interval: Discrete 0.017s (60.000 fps)

        Index       : 3
        Type        : Video Capture
        Pixel Format: 'NV12'
        Name        : NV12 4:2:0 12bpp
                Size: Discrete 640x480
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 720x480
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 720x576
                        Interval: Discrete 0.020s (50.000 fps)
                Size: Discrete 800x600
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1024x768
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1280x720
                        Interval: Discrete 0.020s (50.000 fps)
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1280x800
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1280x1024
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1360x768
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1440x900
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1920x1080
                        Interval: Discrete 0.067s (15.000 fps)
                        Interval: Discrete 0.042s (24.000 fps)
                        Interval: Discrete 0.040s (25.000 fps)
                        Interval: Discrete 0.033s (30.000 fps)
                        Interval: Discrete 0.020s (50.000 fps)
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1680x1050
                        Interval: Discrete 0.067s (15.000 fps)
                        Interval: Discrete 0.042s (24.000 fps)
                        Interval: Discrete 0.040s (25.000 fps)
                        Interval: Discrete 0.033s (30.000 fps)
                        Interval: Discrete 0.020s (50.000 fps)
                        Interval: Discrete 0.017s (60.000 fps)

And, as you advise me, I will more check for the following items.

  • v4l2 to NvBuffer
  • Dummy NvBuffer to NVVideoEncorder

Hi,
It supports 720p50 and 720p60. You may try:

$ gst-launch-1.0 v4l2src ! video/x-raw,width=1280,height=720,format=NV12,framerate=60/1 ! fakesink

And may try io-mode=mmap:

$ gst-launch-1.0 v4l2src io-mode=2 ! video/x-raw,width=1280,height=720,format=NV12,framerate=60/1 ! fakesink
1 Like

It supports 720p50 and 720p60.

Oh, thank you. It was my fault…

And, after added “io-mode = 2” option, it works well. Also, encoding works well :)

gst-launch-1.0 v4l2src io-mode=2 ! video/x-raw,width=1280,height=720,format=NV12,framerate=50/1 ! nvvidconv ! nvv4l2h264enc ! h264parse ! matroskamux ! filesink location=a.mkv

Implementing the same process as the GStreamer command above using the “Multimedia API”, it seems to be able to solve this issue.
As you say, I think I’m using Multimedia API incorrectly.
Could you recommend samples or documents related to this?

Thank you!

Please refer to Jetson Linux API Reference: Main Page (nvidia.com)