Unable to obtain the real-time preview video of the camera

Robotics & Edge Computing Jetson & Embedded Systems Jetson Orin NX
1

During the development of the imx415 camera on Jetson version 6.1, I can capture frame data normally, but cannot perform real-time video preview.

查看摄像头支持的图像格式和分辨率

v4l2-ctl -d /dev/video0 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture

        [0]: 'GB12' (12-bit Bayer GBGB/RGRG)
                Size: Discrete 3864x2192
                        Interval: Discrete 0.067s (15.000 fps)

查看设备信息

v4l2-ctl --list-devices
NVIDIA Tegra Video Input Device (platform:tegra-camrtc-ca):
        /dev/media0

vi-output, iimx415 10-0037 (platform:tegra-capture-vi:0):
        /dev/video1

vi-output, iimx415 9-0037 (platform:tegra-capture-vi:2):
        /dev/video0

查看设备格式

v4l2-ctl -d /dev/video0 --get-fmt-video
Format Video Capture:
        Width/Height      : 3864/2192
        Pixel Format      : 'GB12' (12-bit Bayer GBGB/RGRG)
        Field             : None
        Bytes per Line    : 7728
        Size Image        : 16939776
        Colorspace        : sRGB
        Transfer Function : Default (maps to sRGB)
        YCbCr/HSV Encoding: Default (maps to ITU-R 601)
        Quantization      : Default (maps to Full Range)
        Flags             :

查看 vi、csi、imx415 端口绑定结果

sudo media-ctl -p -d /dev/media0
Media controller API version 5.15.148

Media device information
------------------------
driver          tegra-camrtc-ca
model           NVIDIA Tegra Video Input Device
serial
bus info
hw revision     0x3
driver version  5.15.148

Device topology
- entity 1: 13e00000.host1x:nvcsi@15a00000- (2 pads, 2 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev0
        pad0: Sink
                <- "iimx415 10-0037":0 [ENABLED]
        pad1: Source
                -> "vi-output, iimx415 10-0037":0 [ENABLED]

- entity 4: 13e00000.host1x:nvcsi@15a00000- (2 pads, 2 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev1
        pad0: Sink
                <- "iimx415 9-0037":0 [ENABLED]
        pad1: Source
                -> "vi-output, iimx415 9-0037":0 [ENABLED]

- entity 7: iimx415 9-0037 (1 pad, 1 link)
            type V4L2 subdev subtype Sensor flags 0
            device node name /dev/v4l-subdev2
        pad0: Source
                [fmt:SGBRG12_1X12/3864x2192 field:none colorspace:srgb]
                -> "13e00000.host1x:nvcsi@15a00000-":0 [ENABLED]

- entity 9: vi-output, iimx415 9-0037 (1 pad, 1 link)
            type Node subtype V4L flags 0
            device node name /dev/video0
        pad0: Sink
                <- "13e00000.host1x:nvcsi@15a00000-":1 [ENABLED]

- entity 23: iimx415 10-0037 (1 pad, 1 link)
             type V4L2 subdev subtype Sensor flags 0
             device node name /dev/v4l-subdev3
        pad0: Source
                [fmt:SGBRG12_1X12/3864x2192 field:none colorspace:srgb]
                -> "13e00000.host1x:nvcsi@15a00000-":0 [ENABLED]

- entity 25: vi-output, iimx415 10-0037 (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video1
        pad0: Sink
                <- "13e00000.host1x:nvcsi@15a00000-":1 [ENABLED]

When I enter the following real-time preview commands, the displayed screens are all black. And accompanied by these log information

gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! nvvidconv ! nv3dsink

Setting pipeline to PAUSED ...

Pipeline is live and does not need PREROLL ...

Pipeline is PREROLLED ...

Setting pipeline to PLAYING ...

New clock: GstSystemClock

GST_ARGUS: Creating output stream

CONSUMER: Waiting until producer is connected...

GST_ARGUS: Available Sensor modes :

GST_ARGUS: 3864 x 2192 FR = 16.000000 fps Duration = 62500000 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;



ARGUS_ERROR: Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute: 946 Frame Rate specified is greater than supported

GST_ARGUS: Running with following settings:

   Camera index = 0 

   Camera mode  = 0 

   Output Stream W = 3864 H = 2192 

   seconds to Run    = 0 

   Frame Rate = 16.000000 

GST_ARGUS: Setup Complete, Starting captures for 0 seconds

GST_ARGUS: Starting repeat capture requests.

CONSUMER: Producer has connected; continuing.

ARGUS_ERROR: Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute: 1208 InvalidState.

GST_ARGUS: Cleaning up

ERROR: from element /GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0: Internal data stream error.

Additional debug info:

../libs/gst/base/gstbasesrc.c(3127): gst_base_src_loop (): /GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0:

streaming stopped, reason error (-5)

Execution ended after 0:00:01.018388005

Setting pipeline to NULL ...

nvbuf_utils: dmabuf_fd -1 mapped entry NOT found

Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, threadExecute:734 NvBufSurfaceFromFd Failed.

Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, threadFunction:245 (propagating)

Freeing pipeline ...

st-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=3864,height=2192,format=NV12,framerate=15/1' ! nvvidconv ! nv3dsink

Setting pipeline to PAUSED ...

Pipeline is live and does not need PREROLL ...

Pipeline is PREROLLED ...

Setting pipeline to PLAYING ...

New clock: GstSystemClock

GST_ARGUS: Creating output stream

CONSUMER: Waiting until producer is connected...

GST_ARGUS: Available Sensor modes :

GST_ARGUS: 3864 x 2192 FR = 16.000000 fps Duration = 62500000 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;



GST_ARGUS: Running with following settings:

   Camera index = 0 

   Camera mode  = 0 

   Output Stream W = 3864 H = 2192 

   seconds to Run    = 0 

   Frame Rate = 16.000000 

GST_ARGUS: Setup Complete, Starting captures for 0 seconds

GST_ARGUS: Starting repeat capture requests.

CONSUMER: Producer has connected; continuing.

nvbuf_utils: dmabuf_fd -1 mapped entry NOT found

Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, threadExecute:734 NvBufSurfaceFromFd Failed.

Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, threadFunction:245 (propagating)

Redistribute latency...

WARNING: from element /GstPipeline:pipeline0/GstNv3dSink:nv3dsink0: Pipeline construction is invalid, please add queues.

Additional debug info:

../libs/gst/base/gstbasesink.c(1249): gst_base_sink_query_latency (): /GstPipeline:pipeline0/GstNv3dSink:nv3dsink0:

Not enough buffering available for  the processing deadline of 0:00:00.015000000, add enough queues to buffer  0:00:00.015000000 additional data. Shortening processing latency to 0:00:00.000000000.

WARNING: from element /GstPipeline:pipeline0/GstNv3dSink:nv3dsink0: Pipeline construction is invalid, please add queues.

Additional debug info:

../libs/gst/base/gstbasesink.c(1249): gst_base_sink_query_latency (): /GstPipeline:pipeline0/GstNv3dSink:nv3dsink0:

Not enough buffering available for  the processing deadline of 0:00:00.015000000, add enough queues to buffer  0:00:00.015000000 additional data. Shortening processing latency to 0:00:00.000000000.

Got EOS from element "pipeline0".

Execution ended after 0:00:04.017392542

Setting pipeline to NULL ...

GST_ARGUS: Cleaning up

GST_ARGUS: Done Success

Freeing pipeline ...

gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=3864,height=2192,format=NV12,framerate=15/1' ! nvvidconv ! nv3dsink

Setting pipeline to PAUSED ...

Pipeline is live and does not need PREROLL ...

Pipeline is PREROLLED ...

Setting pipeline to PLAYING ...

New clock: GstSystemClock

GST_ARGUS: Creating output stream

CONSUMER: Waiting until producer is connected...

GST_ARGUS: Available Sensor modes :

GST_ARGUS: 3864 x 2192 FR = 16.000000 fps Duration = 62500000 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;



GST_ARGUS: Running with following settings:

   Camera index = 0 

   Camera mode  = 0 

   Output Stream W = 3864 H = 2192 

   seconds to Run    = 0 

   Frame Rate = 16.000000 

GST_ARGUS: Setup Complete, Starting captures for 0 seconds

GST_ARGUS: Starting repeat capture requests.

CONSUMER: Producer has connected; continuing.

nvbuf_utils: dmabuf_fd -1 mapped entry NOT found

Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, threadExecute:734 NvBufSurfaceFromFd Failed.

Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, threadFunction:245 (propagating)

Redistribute latency...

WARNING: from element /GstPipeline:pipeline0/GstNv3dSink:nv3dsink0: Pipeline construction is invalid, please add queues.

Additional debug info:

../libs/gst/base/gstbasesink.c(1249): gst_base_sink_query_latency (): /GstPipeline:pipeline0/GstNv3dSink:nv3dsink0:

Not enough buffering available for  the processing deadline of 0:00:00.015000000, add enough queues to buffer  0:00:00.015000000 additional data. Shortening processing latency to 0:00:00.000000000.

WARNING: from element /GstPipeline:pipeline0/GstNv3dSink:nv3dsink0: Pipeline construction is invalid, please add queues.

Additional debug info:

../libs/gst/base/gstbasesink.c(1249): gst_base_sink_query_latency (): /GstPipeline:pipeline0/GstNv3dSink:nv3dsink0:

Not enough buffering available for  the processing deadline of 0:00:00.015000000, add enough queues to buffer  0:00:00.015000000 additional data. Shortening processing latency to 0:00:00.000000000.

Got EOS from element "pipeline0".

Execution ended after 0:00:04.012720925

Setting pipeline to NULL ...

GST_ARGUS: Cleaning up

GST_ARGUS: Done Success

Freeing pipeline ...

However, when using frame data capture, the data can be obtained normally.And it can also save pictures in raw format, which open normally.

root@pcai-desktop:/home/pcai/pkg_orin_nx_jp6.1_v7.4.1# v4l2-ctl -d /dev/video0 -V --set-ctrl bypass_mode=0 --stream-mmap
Format Video Capture:
        Width/Height      : 3864/2192
        Pixel Format      : 'GB12' (12-bit Bayer GBGB/RGRG)
        Field             : None
        Bytes per Line    : 7728
        Size Image        : 16939776
        Colorspace        : sRGB
        Transfer Function : Default (maps to sRGB)
        YCbCr/HSV Encoding: Default (maps to ITU-R 601)
        Quantization      : Default (maps to Full Range)
        Flags             :
<<<<<<<<<<<<<<<<< 15.00 fps
<<<<<<<<<<<<<<< 15.00 fps
<<<<<<<<<<<<<<< 15.00 fps
<<<<<<<<<<<<<<< 15.00 fps
<<<<<<<<<<<<<<< 15.00 fps

image
image1920×1101 76.5 KB

This is the LOG information captured when the screen is black.
dmesg.txt (66.3 KB)
trace.txt (21.7 KB)

source file:
imx415.c.txt (17.9 KB)
imx415-dtsi.txt (11.8 KB)
imx415-overlay.txt (2.9 KB)

2

Hello @2450732921,

Can you try the following pipeline ?

gst-launch-1.0 nvarguscamerasrc sensor-id=0 !  'video/x-raw(memory:NVMM), width=3864, height=2192, format=NV12, framerate=15/1' ! nvvidconv ! xvimagesink sync=false

best regards,
Andrew
Embedded Software Engineer at ProventusNova

3

Hi proventusnova,

This command also fails to run.
May I ask what is causing this? Are there any other feasible solutions?

pcai@pcai-desktop:~$ gst-launch-1.0 nvarguscamerasrc sensor-id=0 !  'video/x-raw(memory:NVMM), width=3864, height=2192, format=NV12, framerate=15/1' ! nvvidconv ! xvimagesink sync=false

Setting pipeline to PAUSED ...

Pipeline is live and does not need PREROLL ...

Pipeline is PREROLLED ...

Setting pipeline to PLAYING ...

New clock: GstSystemClock

GST_ARGUS: Creating output stream

CONSUMER: Waiting until producer is connected...

GST_ARGUS: Available Sensor modes :

GST_ARGUS: 3864 x 2192 FR = 16.000000 fps Duration = 62500000 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;



GST_ARGUS: Running with following settings:

   Camera index = 0 

   Camera mode  = 0 

   Output Stream W = 3864 H = 2192 

   seconds to Run    = 0 

   Frame Rate = 16.000000 

GST_ARGUS: Setup Complete, Starting captures for 0 seconds

GST_ARGUS: Starting repeat capture requests.

CONSUMER: Producer has connected; continuing.

nvbuf_utils: dmabuf_fd -1 mapped entry NOT found

Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, threadExecute:734 NvBufSurfaceFromFd Failed.

Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, threadFunction:245 (propagating)

Redistribute latency...

Got EOS from element "pipeline0".

Execution ended after 0:00:04.041460128

Setting pipeline to NULL ...

GST_ARGUS: Cleaning up



(gst-launch-1.0:3971): GStreamer-Video-CRITICAL **: 09:07:07.156: gst_video_center_rect: assertion 'src->h != 0' failed



(gst-launch-1.0:3971): GStreamer-Video-CRITICAL **: 09:07:07.207: gst_video_center_rect: assertion 'src->h != 0' failed



(gst-launch-1.0:3971): GStreamer-Video-CRITICAL **: 09:07:07.257: gst_video_center_rect: assertion 'src->h != 0' failed



(gst-launch-1.0:3971): GStreamer-Video-CRITICAL **: 09:07:07.508: gst_video_center_rect: assertion 'src->h != 0' failed



(gst-launch-1.0:3971): GStreamer-Video-CRITICAL **: 09:07:07.558: gst_video_center_rect: assertion 'src->h != 0' failed



(gst-launch-1.0:3971): GStreamer-Video-CRITICAL **: 09:07:07.608: gst_video_center_rect: assertion 'src->h != 0' failed



(gst-launch-1.0:3971): GStreamer-Video-CRITICAL **: 09:07:07.658: gst_video_center_rect: assertion 'src->h != 0' failed
4

Hello @2450732921,

Thanks for running the test.

That is very interesting.

What happens if you try capturing straight into a fake sink? Does the pipeline work:?

gst-launch-1.0 nvarguscamerasrc ! fake sink

best regards,
Andrew
Embedded Software Engineer at ProventusNova

6

Hi proventusnova,

Nothing happened; only these logs were printed.

warning: erroneous pipeline: no element "fake"
7

Both the same problem form cam0 and cam1?
BTW it’s fakesink element.

8

Hi ShaneCCC,

Yes, both cam0 and cam1 are reporting the same error. Could this be related to my driver configuration? What should I check?

9

The port-index should be 1 and tegra_sinterface should be serial_b for CAM0, but both cameras are the same problem then it could be the sensor driver CID function have incorrect setting cause the capture failed. Maybe modify the exposure/frame_rate control function to dummy function to verify.

Thanks

10

Hi ShaneCCC,

The exposure/frame_rate functions are already empty; no settings are made in them, only print statements are included.

image
image1295×961 80.6 KB

11

Replace the libnvscf.so by attached lib and enable the infinite mode to try.

sudo service nvargus-daemon stop
sudo enableCamInfiniteTimeout=1 nvargus-daemon

libnvscf.so.36.4.3.infinit (8.5 MB)

12

Hi ShaneCCC,

This is my operation process, and it seems that it also doesn’t work.

image

gst-launch-1.0 nvarguscamerasrc ! fake sink
warning: erroneous pipeline: no element "fake"

image
image3136×561 76.9 KB

gst-launch-1.0 nvarguscamerasrc sensor-id=0 !  'video/x-raw(memory:NVMM), width=3864, height=2192, format=NV12, framerate=15/1' ! nvvidconv ! xvimagesink sync=false

image
image2449×887 223 KB

13

What should I do?

14

Need set the display if remote to the Jetson.

export DISPLAY=:0
gst-launch-1.0 nvarguscamerasrc ! fakesink
or 
gst-launch-1.0 nvarguscamerasrc sensor-id=0 !  'video/x-raw(memory:NVMM), width=3864, height=2192, format=NV12, framerate=15/1' ! nvvidconv ! xvimagesink sync=false
15

Hi ShaneCCC,

I used SSH remote access when overwriting the so file. However, when entering the command to obtain the data stream, I did not use remote access; instead, I operated directly on the JETSON main controller.

16

The log show the EGL initial failed. It could be the GPU driver have probelm.

17

Hello @2450732921,

Sorry, the pipe I sent had a typo.

You can try with:

gst-launch-1.0 nvarguscamerasrc ! fakesink

best regards,
Andrew
Embedded Software Engineer at ProventusNova

18

Hi proventusnova,

This command doesn’t work either.

pcai@pcai-desktop:~$ gst-launch-1.0 nvarguscamerasrc ! fakesink

Setting pipeline to PAUSED ...

Pipeline is live and does not need PREROLL ...

Pipeline is PREROLLED ...

Setting pipeline to PLAYING ...

New clock: GstSystemClock

GST_ARGUS: Creating output stream

CONSUMER: Waiting until producer is connected...

GST_ARGUS: Available Sensor modes :

GST_ARGUS: 3864 x 2192 FR = 16.000000 fps Duration = 62500000 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;



ARGUS_ERROR: Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute: 946 Frame Rate specified is greater than supported

GST_ARGUS: Running with following settings:

   Camera index = 0 

   Camera mode  = 0 

   Output Stream W = 3864 H = 2192 

   seconds to Run    = 0 

   Frame Rate = 16.000000 

GST_ARGUS: Setup Complete, Starting captures for 0 seconds

GST_ARGUS: Starting repeat capture requests.

CONSUMER: Producer has connected; continuing.

ARGUS_ERROR: Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute: 1208 InvalidState.

GST_ARGUS: Cleaning up

ERROR: from element /GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0: CANCELLED

Additional debug info:

Argus Error Status

nvbuf_utils: dmabuf_fd -1 mapped entry NOT found

Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, threadExecute:734 NvBufSurfaceFromFd Failed.Execution ended after 0:00:00.141831058



Setting pipeline to NULL ...

Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, threadFunction:245 (propagating)

Freeing pipeline ...

pcai@pcai-desktop:~$
19

Hi @2450732921,

Thanks for running the test.

Yeah, that confirms that the issue is not video preview, but video capture through the Argus subsystem.

Allow me to explain.
When you capture using v4l2-ctl successfully you are capturing RAW data bypassing the Argus capture subsystem. Which means that the NVIDIA Jetson board is not passing buffers through the ISP nor the media processing libraries.

This sounds like a possible device tree misconfiguration at nvcsi or agus camera level.
Or, this could also be an issue with the camera driver.

Where did you get the drivers from?

best regards,
Andrew
Embedded Software Engineer at ProventusNova

20

Hi proventusnova,

This is a driver program written by myself according to the IMX415 sensor manual. The device tree code and driver program code are included in the problem description. Could you please help check where the specific error is?

image
image994×609 61.9 KB

21

Hello?