Unable to stream video from IMX296 sensor on Orin NX

san1998 · December 4, 2023, 11:07am

Hi,

I have a Vision Components MIPI driver installed for IMX296 on Orin NX board. (p3768).
I am using the pipeline

gst-launch-1.0 -e nvarguscamerasrc sensor-id=0 sensor-mode=0 ! 'video/x-raw(memory:NVMM),width=1440,height=1080,framerate=30/1' ! nvvidconv ! nvv4l2h265enc ! h265parse ! mp4mux ! filesink location=out.mp4

The verbose for the above:

libEGL warning: DRI3: failed to query the version
Setting pipeline to PAUSED ...
Opening in BLOCKING MODE 
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Redistribute latency...
NvMMLiteOpen : Block : BlockType = 8 
===== NvVideo: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 8 
GST_ARGUS: Creating output stream
(Argus) Error BadParameter:  (propagating from src/eglstream/FrameConsumerImpl.cpp, function initialize(), line 89)
(Argus) Error BadParameter:  (propagating from src/eglstream/FrameConsumerImpl.cpp, function create(), line 44)
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, threadInitialize:320 Failed to create FrameConsumer
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, threadFunction:241 (propagating)
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, waitRunning:203 Invalid thread state 3
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute:806 (propagating)
Got EOS from element "pipeline0".
Execution ended after 0:00:00.120006916
Setting pipeline to NULL ...
Freeing pipeline ...

on
$ vlc out.mp4

I do not get any recorded video output.

$ cat /dev/video0 gives the frame dump.

The sensor is connected to CSI-2 on the dev board.

The dmesg for the sensor is as follows:

$ sudo dmesg | grep vc_mipi
[    0.002552] DTS File Name: /home/vc_mipi_nvidia/build/Nano_35.3.1/Linux_for_Tegra/source/public/kernel/kernel-5.10/arch/arm64/boot/dts/../../../../../../hardware/nvidia/platform/t23x/p3768/kernel-dts/tegra234-p3767-0000-p3768-0000-a0.dts
[    7.426324] vc_mipi 9-0036: vc_probe(): Probing UNIVERSAL VC MIPI Driver (v0.15.0)
[    9.836345] vc_mipi 9-0036: vc_mod_setup(): Unable to get module I2C client for address 0x10
[    9.845025] vc_mipi 9-0036: vc_probe(): Error in vc_core_init!
[    9.904728] vc_mipi 10-001a: vc_probe(): Probing UNIVERSAL VC MIPI Driver (v0.15.0)
[   10.278330] vc_mipi 10-001a: vc_init_io(): Init trigger and flash mode
[   10.299007] vc_mipi 10-001a: vc_init_frmfmt(): Init frame (width: 1440, height: 1080, fps: 0)
[   10.307982] vc_mipi 10-001a: tegracam sensor driver:vc_mipi_v2.0.6
[   10.314470] tegra-camrtc-capture-vi tegra-capture-vi: subdev vc_mipi 10-001a bound
[   10.323104] vc_mipi 10-001a: vc_core_set_num_lanes(): Set number of lanes 1
[   10.330268] vc_mipi 10-001a: vc_init_lanes(): Init lanes (num_lanes: 1)
[   10.337077] vc_mipi 10-001a: vc_init_tegra_controls(): Read control gain (min: 0, max: 48000, default: 0)
[   10.346912] vc_mipi 10-001a: vc_init_tegra_controls(): Overwrite control exposure (min: 1, max: 15533515, default: 10000)
[   10.358170] vc_mipi 10-001a: vc_init_tegra_controls(): Overwrite control framerate (min: 0, max: 60816, default: 0)
[   47.031643] vc_mipi 10-001a: vc_core_set_format(): Set format 0x300f (RG10)
[   47.042413] vc_mipi 10-001a: vc_core_try_format(): Try format 0x300f (RG10, format: 0x2b)
[   47.042418] vc_mipi 10-001a: vc_update_image_size_from_mode(): Update image size from mode0 (l: 0, t: 0, w: 1440, h: 1080)
[   47.464153] vc_mipi 10-001a: vc_sen_set_roi(): Set sensor roi: (left: 0, top: 0, width: 1440, height: 1080)
[   47.475525] vc_mipi 10-001a: vc_sen_set_exposure(): Set sensor exposure: 10000 us
[   47.484115] vc_mipi 10-001a: vc_sen_set_gain(): Set sensor gain: 0
[   47.490785] vc_mipi 10-001a: vc_sen_set_blacklevel(): Set sensor black level: 60
[   47.498820] vc_mipi 10-001a: vc_sen_start_stream(): Start streaming
[   54.778365] vc_mipi 10-001a: vc_sen_stop_stream(): Stop streaming

While running the below pipeline with autovideosink:

gst-launch-1.0 -e nvarguscamerasrc sensor-id=0 sensor-mode=0 ! 'video/x-raw(memory:NVMM),width=1440,height=1080,framerate=30/1' ! nvvidconv ! autovideosink

The gst launcher opens and closes.
Verbose for the above:


libEGL warning: DRI3: failed to query the version
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
GST_ARGUS: Creating output stream
(Argus) Error BadParameter:  (propagating from src/eglstream/FrameConsumerImpl.cpp, function initialize(), line 89)
(Argus) Error BadParameter:  (propagating from src/eglstream/FrameConsumerImpl.cpp, function create(), line 44)
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, threadInitialize:320 Failed to create FrameConsumer
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, threadFunction:241 (propagating)
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, waitRunning:203 Invalid thread state 3
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute:806 (propagating)
WARNING: from element /GstPipeline:pipeline0/GstAutoVideoSink:autovideosink0/GstXvImageSink:autovideosink0-actual-sink-xvimage: Pipeline construction is invalid, please add queues.
Additional debug info:
gstbasesink.c(1209): gst_base_sink_query_latency (): /GstPipeline:pipeline0/GstAutoVideoSink:autovideosink0/GstXvImageSink:autovideosink0-actual-sink-xvimage:
Not enough buffering available for  the processing deadline of 0:00:00.015000000, add enough queues to buffer  0:00:00.015000000 additional data. Shortening processing latency to 0:00:00.000000000.
Got EOS from element "pipeline0".
Execution ended after 0:00:00.116393343
Setting pipeline to NULL ...
Freeing pipeline ...

Please advice, thanks.

ShaneCCC · December 5, 2023, 5:30am

Could be the same like below topic.

https://forums.developer.nvidia.com/t/libegl-warning-dri3-failed-to-query-the-version-libegl-warning-dri2-failed-to-authenticate/140607/2

san1998 · December 6, 2023, 4:18am

@ShaneCCC

Hi
Is the register address correct for CSI - 2?

cat /dev/video0 gives frame dump.

Could you tell us if the pipeline has to be modified

`gst-launch-1.0 -e nvarguscamerasrc sensor-id=0 sensor-mode=0 ! 
'video/x-raw(memory:NVMM),width=1440,height=1080,framerate=30/1' ! 
nvvidconv ! nvv4l2h265enc ! h265parse ! mp4mux ! filesink 
location=out.mp4`

On VLC out.mp4 i get this:

$ vlc output.mp4 
VLC media player 3.0.9.2 Vetinari (revision 3.0.9.2-0-gd4c1aefe4d)
[0000aaaad7447c40] vlcpulse audio output error: PulseAudio server connection failure: Connection refused
[0000aaaad73835c0] main libvlc: Running vlc with the default interface. Use 'cvlc' to use vlc without interface.
[0000ffff7c001970] cache_read stream error: cannot pre fill buffer
[0000ffff7c001190] mjpeg demux error: cannot peek
qt.qpa.xcb: QXcbConnection: XCB error: 3 (BadWindow), sequence: 2423, resource id: 8414820, major code: 40 (TranslateCoords), minor code: 0
Opening in BLOCKING MODE
Available Sensor modes :
Resolution: 1440 x 1080 ; Framerate = 59.999999; Analog Gain Range Min 0.000000, Max 48.000000, Exposure Range Min 1000, Max 1000000000

[0000ffffa8001950] v4l2 demux error: not a video capture device
Opening in BLOCKING MODE
Available Sensor modes :
Resolution: 1440 x 1080 ; Framerate = 59.999999; Analog Gain Range Min 0.000000, Max 48.000000, Exposure Range Min 1000, Max 1000000000

[0000ffffa8001950] v4l2 demux error: not a radio tuner device
Opening in BLOCKING MODE
Available Sensor modes :
Resolution: 1440 x 1080 ; Framerate = 59.999999; Analog Gain Range Min 0.000000, Max 48.000000, Exposure Range Min 1000, Max 1000000000

[0000ffffa806ddd0] v4l2 stream error: not a video capture device
Segmentation fault (core dumped)

The device tree configuration for CSI - 2 is as follows:


#if VC_MIPI_CAM_1 == 1
                i2c@1 {
#if VC_MIPI_MANUFACTURER == 1
                        vc_mipi_cam1: vc_mipi@1a {
                                reg                     = <0x1a>;
#else
                        vc_mipi_cam1: vc_mipi@60 {
                                reg                     = <0x60>;
#endif
                                compatible              = "nvidia,vc_mipi";
                                devnode                 = "video1";
                                use_sensor_mode_id      = "false";
                                sensor_model            = "vc_mipi";

                                num_lanes               = NUM_LANES;
                                trigger_mode            = "0";
                                io_mode                 = "0";

                                // ----------------------------------------------------
                                // If you want to use GStreamer with nvarguscamerasrc
                                // you have to adjust this settings
                                physical_w              = "4.968";
                                physical_h              = "3.726";
                                // ----------------------------------------------------

                                // This node is needed by the Tegra framework.
                                // You don't have to change any settings if just want to use the V4L API.
                                mode0 {
                                        num_lanes                = NUM_LANES;
                                        tegra_sinterface         = "serial_c";
                                        embedded_metadata_height = VC_MIPI_METADATA_H;
                                        readout_orientation      = "0";

Thanks.

ShaneCCC · December 6, 2023, 4:37am

Don’t understand your question.

You can copy the out.mp4 to host to check the context.

Thanks

san1998 · December 6, 2023, 5:24am

@ShaneCCC

Checked on the host, the mp4 file is blank. Nothing is captured.
Could you check if the device tree configuration for CSI-2 is correct.
The IMX296 sensor on hardware end is configured for 1440x1080 resolution.

ShaneCCC · December 6, 2023, 5:27am

What’s the port-index? CSI2 should be port-index should be 2

san1998 · December 6, 2023, 5:33am

@ShaneCCC

Yes the port-index is set as 2.

ports {
                                        #address-cells = <1>;
                                        #size-cells = <0>;
                                        port@0 {
                                                reg = <0>;
                                                vc_mipi_out1: endpoint {
                                                        port-index = <2>;
                                                        bus-width = <BUS_WIDTH>;
                                                        remote-endpoint = <&vc_csi_in1>;
                                                };

Also,


$ v4l2-ctl --list-devices
NVIDIA Tegra Video Input Device (platform:tegra-camrtc-ca):
	/dev/media0

vi-output, vc_mipi 10-001a (platform:tegra-capture-vi:2):
	/dev/video0

ShaneCCC · December 6, 2023, 5:41am

Did you verify the sensor driver HW by v4l2-ctl.

v4l2-ctl --set-fmt-video=width=1440,height=1080 --set-ctrl bypass_mode=0 --stream-mmap

san1998 · December 6, 2023, 5:43am

@ShaneCCC

yes


 v4l2-ctl --set-fmt-video=width=1440,height=1080 --set-ctrl bypass_mode=0 --stream-mmap
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 53.76 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 56.75 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 57.74 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 58.24 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 59.75 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 59.71 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 59.76 fps

After running unset DISPLAY

I was able to capture the video
But on vlc output , the captured video is vague.

ShaneCCC · December 6, 2023, 5:51am

Did you dump the raw to check the image.

v4l2-ctl --set-fmt-video=width=1449,height=1080 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=1 --stream-to=t.raw -d /dev/video0

san1998 · December 6, 2023, 5:54am

@ShaneCCC

I checked it now,

$ v4l2-ctl --set-fmt-video=width=1440,height=1080 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=1 --stream-to=t.raw -d /dev/video0
<

this is what i get.

ShaneCCC · December 6, 2023, 6:13am

Check the t.raw by 7yuv

san1998 · December 6, 2023, 6:28am

@ShaneCCC

Sorry missed on attaching the screenshots. I see a lot of difference while changing the height and width.

ShaneCCC · December 6, 2023, 6:35am

What’s the pixel format? Suppose shouldn’t RGB 8

san1998 · December 6, 2023, 6:44am

@ShaneCCC

pixel format is RGB and pixel bit depth is set to 10.

ShaneCCC · December 6, 2023, 6:56am

Attached the file here to check.
Why do you check the preview?

gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1,format=NV12' ! nvvidconv ! fpsdisplaysink video-sink=xvimagesink sync=false -v

san1998 · December 6, 2023, 7:00am

@ShaneCCC

ON running
gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1,format=NV12' ! nvvidconv ! fpsdisplaysink video-sink=xvimagesink sync=false -v
the app opens and closes
raw.zip (163.6 KB)

ShaneCCC · December 6, 2023, 7:26am

What’s the error message?

export DISPLAY=:0
gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1,format=NV12' ! nvvidconv ! fpsdisplaysink video-sink=xvimagesink sync=false -v

san1998 · December 6, 2023, 7:29am

@ShaneCCC

Without export DISPLAY=:0

$ gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1,format=NV12' ! nvvidconv ! fpsdisplaysink video-sink=xvimagesink sync=false -v
libEGL warning: DRI3: failed to query the version
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstXvImageSink:xvimagesink0: sync = false
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:src: caps = video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)I420
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)I420
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay.GstPad:src: caps = video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)I420
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstXvImageSink:xvimagesink0.GstPad:sink: caps = video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)I420
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay.GstPad:video_sink: caps = video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)I420
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink: caps = video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)I420
GST_ARGUS: Creating output stream
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)NV12, framerate=(fraction)30/1
(Argus) Error BadParameter:  (propagating from src/eglstream/FrameConsumerImpl.cpp, function initialize(), line 89)
(Argus) Error BadParameter:  (propagating from src/eglstream/FrameConsumerImpl.cpp, function create(), line 44)
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, threadInitialize:320 Failed to create FrameConsumer
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, threadFunction:241 (propagating)
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, waitRunning:203 Invalid thread state 3
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute:806 (propagating)
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstXvImageSink:xvimagesink0: sync = false
Got EOS from element "pipeline0".
Execution ended after 0:00:00.110382378
Setting pipeline to NULL ...
Freeing pipeline ...

With export DISPLAY=:0

$ gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1,format=NV12' ! nvvidconv ! fpsdisplaysink video-sink=xvimagesink sync=false -v
No protocol specified
No protocol specified
No protocol specified
No protocol specified
nvbufsurftransform: Could not get EGL display connection
Setting pipeline to PAUSED ...
No protocol specified
ERROR: Pipeline doesn't want to pause.
ERROR: from element /GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstXvImageSink:xvimagesink0: Could not initialise Xv output
Additional debug info:
xvimagesink.c(1773): gst_xv_image_sink_open (): /GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstXvImageSink:xvimagesink0:
Could not open display (null)
Setting pipeline to NULL ...
Freeing pipeline ...

ShaneCCC · December 6, 2023, 7:36am

Do you have HDMI display connect to Orin NX?

xinit&
export DISPLAY=:0
gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=1440,height=1080,framerate=30/1,format=NV12' ! nvvidconv ! fpsdisplaysink video-sink=xvimagesink sync=false -v

Topic		Replies	Views
Kernel panic occurs while running gstreamer pipeline Jetson Orin NX kernel , gstreamer , video	10	530	February 6, 2024
Jetson AGX Orin nvargus_nvraw doesn't show sensors Jetson AGX Orin camera	18	69	October 11, 2024
Orin Nano error with IMX219 Camera Jetson Orin Nano camera , gstreamer	47	2830	June 21, 2023
TX2 VI mode and ISP mode bugs when using CSI2 virtual channels Jetson TX2 camera , nvbugs	15	1313	March 9, 2022
CSI camera change RGB888 format to bayer format issue Jetson AGX Xavier camera	39	2381	April 13, 2022
[R36.3] Can't capture images successfully with ISX031, Need some help Jetson AGX Orin camera , board-design	14	66	December 3, 2024
Problems with nvarguscamerasrc using the last imx565 mode Jetson AGX Xavier camera , gstreamer	43	344	July 11, 2024
Video node available but data not streaming Jetson Orin NX camera , gstreamer	43	1861	August 11, 2023
Failing to get input from CSI camera (IMX219-83) Jetson Nano camera , ubuntu , gstreamer	29	1427	September 25, 2023
Gstreamer reports Raspberry Pi camera streaming at 120fps when in reality it is only 60fps Jetson Nano camera , gstreamer	53	3775	July 19, 2022

Unable to stream video from IMX296 sensor on Orin NX

Related topics