How to overlay two picture image on the TX2

Hi,

I have a question.

I want to know how to overlay two picture image(jpeg or mp4) from images taken EO/IR camera to TX2 board display throughput the HDMItoCSI.

kernel source version is R28.1 and, gstreamer ver. is 1.12.3

Would you let me know solving way this issue?

Thanks.

You may refer to https://devtalk.nvidia.com/default/topic/1014789/jetson-tx1/-the-cpu-usage-cannot-down-use-cuda-decode-/post/5188538/#5188538

It runs four rtsp streaming playback simultaneously.

It sounds like you want to overlay the IR image over the HDMI input image.

The brute force way of doing this is to push each image into a separate texture, and write some OpenGL/ES code to draw the one on top of the other, using whatever blend mode and geometry positioning you choose.

The slightly-less brute-force way of doing this seems to be to use the “overlay” support of the NVIDIA MultimediaAPI buffers, and send the different images into the different overlays. However, you’d need to resize the smaller image to be the size you want manually this way.

You may also be able to put together a gstreamer pipeline to scale and compose the images, or draw them on top of each other. The “nvidia gstreamer plugins” manual shows you what the various components are:
http://developer2.download.nvidia.com/embedded/L4T/r24_Release_v2.1/Docs/Accelerated_GStreamer_User_Guide_Release_24.2.1.pdf?rKtXjjjzuz0jABPCY7whGroCrFjTCAdOEKFkIZIZz0NtGN2HNu6HhAKXUg9OjNjAjBJe7HyhfFletmOaCxNy1vt4MHPdAS8GKIiv2xJ06RgE_Amgfkpx9-PdAOhc9Vq4RAW7QAjzwxemgQoecxOC7k83wyx7j3G18eP4v1FpawB0AsjRfkIXSQkl5b3KaSTLgkTdeqo
Hopefully the “mix” image sink (non-nvidia) would work to blend two images after processing.
(I find that the NVIDIA documentation isn’t that great – there is no section that documents the arguments for nvoverlaysink, and most examples pass “-e” to it – what does that mean? Who knows?)

Gstreamer plugins doc is inside plugins. Use gst-inspect-1.0 and you’ll get the plugin caps and options with descriptions.

[EDIT: @Snarky was right it’s a bit short. I’ve asked the question there https://devtalk.nvidia.com/default/topic/1026818/jetson-tx2/nvoverlaysink-parameters-/post/5222530/#5222530]

-e, --eos-on-shutdown
Force an EOS event on sources before shutting the pipeline down. This is useful to make sure muxers create readable files when a muxing pipeline is shut down forcefully via Control-C.

Hello Dane,

Thank you for your support.

But, First of all, I’d like to check output for the overlay image throughput display on monitor connected to TX2.

So, Could you let me know the command about the gstreamer?

Thanks & BR,

Hi,
You can run the command with onboard camera to see preview on HDMI out(TV)

gst-launch-1.0 nvcamerasrc ! nvoverlaysink

My HDMItoCSI driver is tc358840.

That is based on v4l2src not nvcamerasrc.

Therefore, Would you let me know gst command for v4l2src?

But, Is no set of the resolution for EO/IR cam in the gst command?

Thanks,

For tc358840, please refer to below topics:
https://devtalk.nvidia.com/default/topic/938200/
https://devtalk.nvidia.com/default/topic/952733/
https://devtalk.nvidia.com/default/topic/1023423/
https://devtalk.nvidia.com/default/topic/1024528/

I have seen your link topics. but, I don’t know command for nvoverlaysinc exactly.

Could you let me know gst command working well directly?

Thanks & Best Regards,

You may try:

gst-inspect-1.0 nvoverlaysink

to get capabilities and options.

You can also have a look to https://devtalk.nvidia.com/default/topic/1025938/jetson-tx2/imx274-drivers-on-tx1-vs-tx2/post/5220725/#5220725 for a working example with a single overlay.

Right, but the question is how to overlay two pictures.
I imagine you can run two separate pipelines entirely, but then you get into the question of which picture is on top of which.
Is there a way to overlay two different pictures in the same pipeline using gstreamer? Similar to how Multimedia API supports up to three planes?

I see the problem…but not the solution.
Indeed, if I launch a pipeline

gst-launch-1.0 -ev nvcamerasrc ! 'video/x-raw(memory:NVMM),width=1920,height=1080' ! nvvidconv ! 'video/x-raw(memory:NVMM), format=I420, width=320, height=190' ! nvoverlaysink overlay-x=1000 overlay-y=100 overlay-w=320 overlay-h=190 overlay=1

or alternately (I only have onboard camera)

gst-launch-1.0 -ev videotestsrc ! nvvidconv ! 'video/x-raw(memory:NVMM), format=I420' ! nvoverlaysink overlay-x=400 overlay-y=600 overlay-w=320 overlay-h=190 overlay=0

It works fine. But once one has been started, launching a second pipeline fails, even using a different overlay with errors:

NvxBaseWorkerFunction[2575] comp OMX.Nvidia.std.iv_renderer.overlay.yuv420 Error -2147479552 
NvxBaseWorkerFunction[2575] comp OMX.Nvidia.std.iv_renderer.overlay.yuv420 Error -2147479552 
NvxBaseWorkerFunction[2575] comp OMX.Nvidia.std.iv_renderer.overlay.yuv420 Error -2147479552 
NvxBaseWorkerFunction[2575] comp OMX.Nvidia.std.iv_renderer.overlay.yuv420 Error -2147479552 
NvxBaseWorkerFunction[2575] comp OMX.Nvidia.std.iv_renderer.overlay.yuv420 Error -2147479552 
NvxBaseWorkerFunction[2575] comp OMX.Nvidia.std.iv_renderer.overlay.yuv420 Error -2147479552 
NvxBaseWorkerFunction[2575] comp OMX.Nvidia.std.iv_renderer.overlay.yuv420 Error -2147479552 
NvxBaseWorkerFunction[2575] comp OMX.Nvidia.std.iv_renderer.overlay.yuv420 Error -2147479552 
NvxBaseWorkerFunction[2575] comp OMX.Nvidia.std.iv_renderer.overlay.yuv420 Error -2147479552 
NvxBaseWorkerFunction[2575] comp OMX.Nvidia.std.iv_renderer.overlay.yuv420 Error -2147479552 
NvxBaseWorkerFunction[2575] comp OMX.Nvidia.std.iv_renderer.overlay.yuv420 Error -2147479552 
NvxBaseWorkerFunction[2575] comp OMX.Nvidia.std.iv_renderer.overlay.yuv420 Error -2147479552

Still no idea what happens, sorry.

[EDIT: overlay=0 is wrong. overlay indexes are supposed to be in range [1 - 2]. I guess 0 was defaulted to index 1. overlay 2 would be correct, but you may need to see https://devtalk.nvidia.com/default/topic/1024472/jetson-tx2/using-overlay-2-in-l4t-28-1-with-tx2/post/5213583/#5213583 for using overlay 2.]

Note that using

gst-launch-1.0 -ev videotestsrc ! nvvidconv ! 'video/x-raw(memory:NVMM), format=I420, height=320, width=190' ! nvoverlaysink overlay-x=400 overlay-y=600 overlay-w=320 overlay-h=190  overlay=1

you’ll see the stream height and width seem to take priority over overlay parameters overlay-w and overlay-h.

hi nwlee,
Have you checked your source via v4l2-ctl?
If your source is /dev/video1, you can check formats:

nvidia@tegra-ubuntu:~$ v4l2-ctl -d /dev/video1 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
        Index       : 0
        Type        : Video Capture
        Pixel Format: 'YUYV'
        Name        : YUYV 4:2:2
                Size: Discrete 640x480
                        Interval: Discrete 0.033s (30.000 fps)
                        Interval: Discrete 0.050s (20.000 fps)
                        Interval: Discrete 0.067s (15.000 fps)
                        Interval: Discrete 0.100s (10.000 fps)
                        Interval: Discrete 0.200s (5.000 fps)
                        Interval: Discrete 1.000s (1.000 fps)
                Size: Discrete 352x288
                        Interval: Discrete 0.033s (30.000 fps)
                        Interval: Discrete 0.050s (20.000 fps)
                        Interval: Discrete 0.067s (15.000 fps)
                        Interval: Discrete 0.100s (10.000 fps)
                        Interval: Discrete 0.200s (5.000 fps)
                        Interval: Discrete 1.000s (1.000 fps)
                Size: Discrete 320x240
                        Interval: Discrete 0.033s (30.000 fps)
                        Interval: Discrete 0.050s (20.000 fps)
                        Interval: Discrete 0.067s (15.000 fps)
                        Interval: Discrete 0.100s (10.000 fps)
                        Interval: Discrete 0.200s (5.000 fps)
                        Interval: Discrete 1.000s (1.000 fps)
                Size: Discrete 176x144
                        Interval: Discrete 0.033s (30.000 fps)
                        Interval: Discrete 0.050s (20.000 fps)
                        Interval: Discrete 0.067s (15.000 fps)
                        Interval: Discrete 0.100s (10.000 fps)
                        Interval: Discrete 0.200s (5.000 fps)
                        Interval: Discrete 1.000s (1.000 fps)
                Size: Discrete 160x120
                        Interval: Discrete 0.033s (30.000 fps)
                        Interval: Discrete 0.050s (20.000 fps)
                        Interval: Discrete 0.067s (15.000 fps)
                        Interval: Discrete 0.100s (10.000 fps)
                        Interval: Discrete 0.200s (5.000 fps)
                        Interval: Discrete 1.000s (1.000 fps)

and do capture:

nvidia@tegra-ubuntu:~$ v4l2-ctl --set-fmt-video=width=640,height=480,pixelformat="YUYV" --device=/dev/video1 --stream-to=/tmp/a.yuv --stream-count=45 --stream-mmap
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.00 fps
<<<<<<<<<<<<<<

Here is an example. You may see different pixel format, width, height, framerate in your source.

Hello Honey,

Thank you for your support.

The command that you known is working well.

I’d like to know working structure for gstreamer.

So, could you let me know related source position and document?

Thanks & BR,

Hi nwlee,

For handy gstreamer commands about nv elements, you may take a look at https://developer.nvidia.com/embedded/dlc/l4t-tx2-accelerated-gstreamer-guide-28-1 first.

You may also check this https://devtalk.nvidia.com/default/topic/1024472/jetson-tx2/using-overlay-2-in-l4t-28-1-with-tx2/post/5213583/#5213583.

Hi,

I don’t know for nvoverlaysinc command exactly yet.

Is it possible working two resolution in one pipeline?

Maybe, is it patch for R28.1 source?

Thaks & BR,

Hi Dane,

I have tried patching that attached two patch files “0001-dt-quill-dc-update-plugin-manager-assignments.patch”, “0001-dt-t18x-quill-nvdisplay-update-window-mappings.patch”.

“0001-dt-quill-dc-update-plugin-manager-assignments.patch” file is no problem.

But, “0001-dt-t18x-quill-nvdisplay-update-window-mappings.patch” is not patch.

I have check that I have not found *.dts files in the “0001-dt-t18x-quill-nvdisplay-update-window-mappings.patch”.

So, Could you please check for that? My source is R28.1

Thank you & Best Regards,

Hi Dane,

I have test for frame-rate as below.

===============================================================================================

nvidia@tegra-ubuntu:~$ v4l2-ctl -d /dev/video0 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
Index : 0
Type : Video Capture
Pixel Format: ‘AR24’
Name : 32-bit BGRA 8-8-8-8
Size: Discrete 3840x2160
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.040s (25.000 fps)
Interval: Discrete 0.033s (30.000 fps)
Size: Discrete 1920x1080
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.040s (25.000 fps)
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.017s (60.000 fps)
Size: Discrete 1280x720
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.040s (25.000 fps)
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.017s (60.000 fps)
Size: Discrete 640x480
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.040s (25.000 fps)
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.017s (60.000 fps)

    Index       : 1
    Type        : Video Capture
    Pixel Format: 'UYVY'
    Name        : UYVY 4:2:2
            Size: Discrete 3840x2160
                    Interval: Discrete 0.042s (24.000 fps)
                    Interval: Discrete 0.040s (25.000 fps)
                    Interval: Discrete 0.033s (30.000 fps)
            Size: Discrete 1920x1080
                    Interval: Discrete 0.042s (24.000 fps)
                    Interval: Discrete 0.040s (25.000 fps)
                    Interval: Discrete 0.033s (30.000 fps)
                    Interval: Discrete 0.017s (60.000 fps)
            Size: Discrete 1280x720
                    Interval: Discrete 0.042s (24.000 fps)
                    Interval: Discrete 0.040s (25.000 fps)
                    Interval: Discrete 0.033s (30.000 fps)
                    Interval: Discrete 0.017s (60.000 fps)
            Size: Discrete 640x480
                    Interval: Discrete 0.042s (24.000 fps)
                    Interval: Discrete 0.040s (25.000 fps)
                    Interval: Discrete 0.033s (30.000 fps)
                    Interval: Discrete 0.017s (60.000 fps)

    Index       : 2
    Type        : Video Capture
    Pixel Format: 'NV16'
    Name        : Y/CbCr 4:2:2

nvidia@tegra-ubuntu:~ nvidia@tegra-ubuntu:~ v4l2-ctl -d /dev/video1 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
Index : 0
Type : Video Capture
Pixel Format: ‘AR24’
Name : 32-bit BGRA 8-8-8-8
Size: Discrete 3840x2160
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.040s (25.000 fps)
Interval: Discrete 0.033s (30.000 fps)
Size: Discrete 1920x1080
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.040s (25.000 fps)
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.017s (60.000 fps)
Size: Discrete 1280x720
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.040s (25.000 fps)
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.017s (60.000 fps)
Size: Discrete 640x480
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.040s (25.000 fps)
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.017s (60.000 fps)

    Index       : 1
    Type        : Video Capture
    Pixel Format: 'UYVY'
    Name        : UYVY 4:2:2
            Size: Discrete 3840x2160
                    Interval: Discrete 0.042s (24.000 fps)
                    Interval: Discrete 0.040s (25.000 fps)
                    Interval: Discrete 0.033s (30.000 fps)
            Size: Discrete 1920x1080
                    Interval: Discrete 0.042s (24.000 fps)
                    Interval: Discrete 0.040s (25.000 fps)
                    Interval: Discrete 0.033s (30.000 fps)
                    Interval: Discrete 0.017s (60.000 fps)
            Size: Discrete 1280x720
                    Interval: Discrete 0.042s (24.000 fps)
                    Interval: Discrete 0.040s (25.000 fps)
                    Interval: Discrete 0.033s (30.000 fps)
                    Interval: Discrete 0.017s (60.000 fps)
            Size: Discrete 640x480
                    Interval: Discrete 0.042s (24.000 fps)
                    Interval: Discrete 0.040s (25.000 fps)
                    Interval: Discrete 0.033s (30.000 fps)
                    Interval: Discrete 0.017s (60.000 fps)

    Index       : 2
    Type        : Video Capture
    Pixel Format: 'NV16'
    Name        : Y/CbCr 4:2:2

nvidia@tegra-ubuntu:~ nvidia@tegra-ubuntu:~
nvidia@tegra-ubuntu:~ nvidia@tegra-ubuntu:~ v4l2-ctl --set-fmt-video=width=1920,height=1080,pixelforma
at=“UYVY” --device=/dev/video0 --stream-to=/tmp/a.yuv --stream-count=45 --stream-
-mmap
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
nvidia@tegra-ubuntu:~ nvidia@tegra-ubuntu:~ v4l2-ctl --set-fmt-video=width=1280,height=720,pixelforma
t=“UYVY” --device=/dev/video1 --stream-to=/tmp/a.yuv --stream-count=45 --stream-
mmap
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
nvidia@tegra-ubuntu:~$

nvidia@tegra-ubuntu:~$ v4l2-ctl --set-fmt-video=width=640,height=480,pixelformat
=“AR24” --device=/dev/video1 --stream-to=/tmp/a.yuv --stream-count=45 --stream-m
map
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 32.67 fps
<<<<<<<<<<<

===============================================================================================

/dev/video0 is EO camera, and /dev/video1 is IR camera.

So, could you please let me know how to test nvoverlaysinc for EO and IR camera two playback?

Thanks & BR,