IMX477 Corrupted Captures/Stream on Jetson Orin

I’ve configured an IMX477 camera on Jetson Orin JetPack 6.2.1 using jetson-io.py, and while the camera is detected and GStreamer can capture images, the nvargus-daemon is failing to initialize the V4L2 sensor driver properly. results in Corrupted/half-frame images from nvarguscamerasrc.

v4l2-ctl info:

$ v4l2-ctl --device /dev/video0 --all
Driver Info:
        Driver name      : tegra-video
        Card type        : vi-output, imx477 9-001a
        Bus info         : platform:tegra-capture-vi:2
        Driver version   : 5.15.148
        Capabilities     : 0x84200001
                Video Capture
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps      : 0x04200001
                Video Capture
                Streaming
                Extended Pix Format
Media Driver Info:
        Driver name      : tegra-camrtc-ca
        Model            : NVIDIA Tegra Video Input Device
        Serial           : 
        Bus info         : 
        Media version    : 5.15.148
        Hardware revision: 0x00000003 (3)
        Driver version   : 5.15.148
Interface Info:
        ID               : 0x0300000b
        Type             : V4L Video
Entity Info:
        ID               : 0x00000009 (9)
        Name             : vi-output, imx477 9-001a
        Function         : V4L2 I/O
        Pad 0x0100000a   : 0: Sink
          Link 0x0200000f: from remote pad 0x1000006 of entity '13e00000.host1x:nvcsi@15a00000-' (Unknown sub-device (0002000a)): Data, Enabled
Priority: 2
Video input : 0 (Camera 2: ok)
Format Video Capture:
        Width/Height      : 4032/3040
        Pixel Format      : 'RG10' (10-bit Bayer RGRG/GBGB)
        Field             : None
        Bytes per Line    : 8064
        Size Image        : 24514560
        Colorspace        : sRGB
        Transfer Function : Default (maps to sRGB)
        YCbCr/HSV Encoding: Default (maps to ITU-R 601)
        Quantization      : Default (maps to Full Range)
        Flags             : 
Streaming Parameters Video Capture:
        Capabilities     : timeperframe
        Frames per second: 30.000 (30/1)
        Read buffers     : 0

Camera Controls

                     group_hold 0x009a2003 (bool)   : default=0 value=0 flags=execute-on-write
                    sensor_mode 0x009a2008 (int64)  : min=0 max=2 step=1 default=0 value=1 flags=slider
                           gain 0x009a2009 (int64)  : min=16 max=357 step=1 default=16 value=16 flags=slider
                       exposure 0x009a200a (int64)  : min=13 max=683710 step=1 default=2495 value=33330 flags=slider
                     frame_rate 0x009a200b (int64)  : min=2000000 max=60000000 step=1 default=60000000 value=30000001 flags=slider
           sensor_configuration 0x009a2032 (u32)    : min=0 max=4294967295 step=1 default=0 dims=[22] flags=read-only, volatile, has-payload
         sensor_mode_i2c_packet 0x009a2033 (u32)    : min=0 max=4294967295 step=1 default=0 dims=[1026] flags=read-only, volatile, has-payload
      sensor_control_i2c_packet 0x009a2034 (u32)    : min=0 max=4294967295 step=1 default=0 dims=[1026] flags=read-only, volatile, has-payload
                    bypass_mode 0x009a2064 (intmenu): min=0 max=1 default=0 value=1 (1 0x1)
                                0: 0 (0x0)
                                1: 1 (0x1)
                override_enable 0x009a2065 (intmenu): min=0 max=1 default=0 value=1 (1 0x1)
                                0: 0 (0x0)
                                1: 1 (0x1)
                   height_align 0x009a2066 (int)    : min=1 max=16 step=1 default=1 value=1
                     size_align 0x009a2067 (intmenu): min=0 max=2 default=0 value=0 (1 0x1)
                                0: 1 (0x1)
                                1: 65536 (0x10000)
                                2: 131072 (0x20000)
               write_isp_format 0x009a2068 (int)    : min=1 max=1 step=1 default=1 value=1
       sensor_signal_properties 0x009a2069 (u32)    : min=0 max=4294967295 step=1 default=0 dims=[30][18] flags=read-only, has-payload
        sensor_image_properties 0x009a206a (u32)    : min=0 max=4294967295 step=1 default=0 dims=[30][16] flags=read-only, has-payload
      sensor_control_properties 0x009a206b (u32)    : min=0 max=4294967295 step=1 default=0 dims=[30][36] flags=read-only, has-payload
              sensor_dv_timings 0x009a206c (u32)    : min=0 max=4294967295 step=1 default=0 dims=[30][16] flags=read-only, has-payload
               low_latency_mode 0x009a206d (bool)   : default=0 value=0
               preferred_stride 0x009a206e (int)    : min=0 max=65535 step=1 default=0 value=0
    override_capture_timeout_ms 0x009a206f (int)    : min=-1 max=2147483647 step=1 default=2500 value=2500
                   sensor_modes 0x009a2082 (int)    : min=0 max=30 step=1 default=30 value=2 flags=read-only

GStreamer Test Results

Command:

gst-launch-1.0 -e nvarguscamerasrc sensor-id=0 num-buffers=1 ! \
  'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1' ! \
  nvvidconv flip-method=0 ! \
  'video/x-raw,format=NV12' ! \
  videoconvert ! 'video/x-raw,format=BGR' ! \
  jpegenc quality=95 ! \
  filesink location=/tmp/test.jpg

Output: Creates JPEG successfully (~200KB) but image shows corrupted/interlaced frames
Sample Image:

Hi,

In order to know if the problem is with nvarguscamerasrc only, can you try to capture using v4l2-ctl?

v4l2-ctl -d /dev/video0 --stream-mmap --stream-count=1 --stream-to=image_1920x1080.raw —set-fmt-video=width=1920,height=1080

You can attach the image. Or try to see the image with vooya as single channel image.

Manuel Leiva
Embedded SW Engineer at RidgeRun
Contact us: support@ridgerun.com
Developers wiki: https://developer.ridgerun.com
Website: www.ridgerun.com

hello mshey904,

is this only reproduced with 1920x1080 sensor mode?
could you please test with 3840x2160 sensor mode for confirmation.

besides.. is this only seen with 1st captured frame?
please also give it a try to encode several frames for checking.
for instance,
$ gst-launch-1.0 -e nvarguscamerasrc sensor-id=0 num-buffers=10 ! 'video/x-raw(memory:NVMM),width=1920, height=1080, framerate=30/1, format=NV12' ! nvvidconv ! jpegenc ! multifilesink location=~/Desktop/tmp/capture%d.jpeg

@ManuelLeiva

this is the sensor info:

orin@ubuntu:~$ v4l2-ctl --list-devices
NVIDIA Tegra Video Input Device (platform:tegra-camrtc-ca):
        /dev/media0

vi-output, imx477 10-001a (platform:tegra-capture-vi:1):
        /dev/video0

---------------------------------------
orin@ubuntu:~$ v4l2-ctl -d /dev/video0 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture

        [0]: 'RG10' (10-bit Bayer RGRG/GBGB)
                Size: Discrete 4032x3040
                        Interval: Discrete 0.048s (21.000 fps)
                Size: Discrete 3840x2160
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 1920x1080
                        Interval: Discrete 0.017s (60.000 fps)
---------------------------------------
orin@ubuntu:~$ v4l2-ctl -d /dev/video0 --get-fmt-video
Format Video Capture:
        Width/Height      : 4032/3040
        Pixel Format      : 'RG10' (10-bit Bayer RGRG/GBGB)
        Field             : None
        Bytes per Line    : 8064
        Size Image        : 24514560
        Colorspace        : sRGB
        Transfer Function : Default (maps to sRGB)
        YCbCr/HSV Encoding: Default (maps to ITU-R 601)
        Quantization      : Default (maps to Full Range)
        Flags             : 

this is the raw image for FullHD res:
image_1920x1080.raw.gz (23.4 MB)

4k with pixelformat=RG10:
image_3840x2160.raw.gz (23.4 MB)

I added a .gz at the end of the file name to be able to upload the file, which should be removed after the download.

Thanks in advance.

Hi @JerryChang

It’s in both sensor mode.

Full HD:
First frame:

10th frame:

4K:
First frame:

10th frame:

hello mshey904,

may I know which camera module you’re using?
I cannot reproduce this locally with IMX477. we’re testing on Orin NX/JP-6.2.1/Raspberry PI HQ (IMX477).

Hi @JerryChang,

The camera module is imx477 from waveshare.

Is it possible that the ribbon cable is the cause of this issue too?

hello mshey904,

may I know how you dump the raw images.
that Raw file looks incorrect. could you please double check the raw file dumps.
I’ve tried using 3rdparty tools to open your 4K raw file, I don’t see validate image by setting the same formats.

Hi @JerryChang

For fullHD:

v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080 --stream-mmap --stream-count=1 --stream-to=image_1920x1080.raw

and for 4K:

v4l2-ctl -d /dev/video0 --set-fmt-video=width=3840,height=2160,pixelformat=RG10 --stream-mmap --stream-count=1 --stream-to=image_3840x2160.raw

But both files have the same file size, which seems to not follow the flags.

This is sensor info:

orin@ubuntu:~$ v4l2-ctl -d /dev/video0 --get-fmt-video
Format Video Capture:
        Width/Height      : 4032/3040
        Pixel Format      : 'RG10' (10-bit Bayer RGRG/GBGB)
        Field             : None
        Bytes per Line    : 8064
        Size Image        : 24514560
        Colorspace        : sRGB
        Transfer Function : Default (maps to sRGB)
        YCbCr/HSV Encoding: Default (maps to ITU-R 601)
        Quantization      : Default (maps to Full Range)
        Flags             : 

and when i check the raw image:

wc -c < image_3840x2160.raw
 24514560

I got 24514560, which when divided by 2, then divided by width 4032, I get the height of the sensor info.

hello mshey904,

is it driver side always select 4032*3040 sensor mode for streaming?
you may try adding use_sensor_mode_id = "true"; to the sensor device tree,
then, you’ll be able to use v4l2 options, --set-ctrl sensor_mode=<ID> for mode selection.
for instance,
$ v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --set-ctrl bypass_mode=0 --set-ctrl sensor_mode=2 --stream-mmap --stream-count=100

Yes.

How can I view the raw file results?

hello mshey904,

you may use the 3rdparty tool, such as 7yuv to open/view the raw files.