Failed to switch between sensor existing formats using v4l2 and captured raw image is wrong as well

Robotics & Edge Computing Jetson & Embedded Systems Jetson Xavier NX
1

I’m working on Jetson Xavier NX with imx219 sensor. My sensor supports 4 modes listed below:

v4l2-ctl -d /dev/video0 --list-formats-ext
  ioctl: VIDIOC_ENUM_FMT
  Type: Video Capture
  [0]: 'RG10' (10-bit Bayer RGRG/GBGB)
	Size: Discrete 3280x2464
		Interval: Discrete 0.048s (21.000 fps)
	Size: Discrete 3280x1848
		Interval: Discrete 0.036s (28.000 fps)
	Size: Discrete 1920x1080
		Interval: Discrete 0.033s (30.000 fps)
	Size: Discrete 1640x1232
		Interval: Discrete 0.033s (30.000 fps)
	Size: Discrete 1280x720
		Interval: Discrete 0.017s (60.000 fps)

Problem 1:

When I try to change the current format of my sensor using v4l2, it fails and remains on the default format i.e., 3280x2464 (terminal log below)

v4l2-ctl --set-fmt-video=width=1280,height=720,pixelformat=RG10
talha@talha-desktop:~/Desktop/gstreamer/gstreamer-custom-plugin/gstreamer-myplugin-10$ v4l2-ctl -d /dev/video0 --get-fmt-video
Format Video Capture:
	Width/Height      : 3280/2464
	Pixel Format      : 'RG10' (10-bit Bayer RGRG/GBGB)
	Field             : None
	Bytes per Line    : 6560
	Size Image        : 16163840
	Colorspace        : sRGB
	Transfer Function : Default (maps to sRGB)
	YCbCr/HSV Encoding: Default (maps to ITU-R 601)
	Quantization      : Default (maps to Full Range)
	Flags             :

The set-fmt-video command runs without any error but still the format is not set to the expected format.

And I’m also unable to capture raw bayer image for other dimensions apart from the default 3280x2464. Even when I try to capture the smaller size image, it captures 3280x2464 image

v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --set-ctrl=bypass_mode=0 --stream-mmap --stream-count=1 --stream-to=file_from_v4l2_1920x1080.raw

The above command captures a raw file of size 16,163,840 bytes which is equal to 3280 x 2464 x 2 = 16163840 (multiplied by 2 as 10 bit image is stored in 16 bits)

I’ve confirmed with nvargus to see if I’m able to extract 1080p raw image from my sensor and yes I am able to extract it so the sensor does support 1080p format as well as shown by the v4l2 list-formats-ext command.

nvargus_nvraw --lps
nvargus_nvraw version 1.15.0
Number of supported sensor entries 5
Entry  Source Mode      Uniquename             Resolution   FR  BitDepth  Mode
Index  Index  Index                                             CSI Dyn   Type
0      0      0           jakku_rear_RBP194   3280x2464   20  10  10    Bayer
1      0      1           jakku_rear_RBP194   3280x1848   28  10  10    Bayer
2      0      2           jakku_rear_RBP194   1920x1080   29  10  10    Bayer
3      0      3           jakku_rear_RBP194   1640x1232   29  10  10    Bayer
4      0      4           jakku_rear_RBP194   1280x 720   59  10  10    Bayer
talha@talha-desktop:~/Desktop/gstreamer/gstreamer-custom-plugin/gstreamer-myplugin-11/test_isp/in_frames$ nvargus_nvraw --c 0 --mode 2 --file “file_from_nvargus_1920x1080.raw” --format “raw”
nvargus_nvraw version 1.15.0
capture: Total captured frame count 90
±----- Exposure info results -----------+
| Exposure | Exposure Time | Sensor Gain |
|        0 |      0.040000 |     8.64462 |
±---------------------------------------+
writeToFile: files file_from_nvargus_1920x1080.raw and file_from_nvargus_1920x1080.txt are successfully
created

The above 2 commands shows my different sensor modes along with successful 1080p raw image capture whose size is 4,154,880 bytes which is equal to 1920 x 1080 x 2.

Problem 2

I’ve passed the bayer image (file_from_v4l2_1920x1080.raw) through my custom written software ISP code and saved it as png file. I’ve attached it here and you can clearly see image captured from v4l2 is giving me ghosty hallows and it looks like that 2 images placed on top of each other and 1 of them is shifted towards left from the right side:

file_from_v4l2_1920x1080
file_from_v4l2_1920x10801920×1442 81.6 KB

The images captured from nvargus looks fine after I pass them from my software ISP.

I’ve attached both raw bayer images as well here (as the discussion forum isn’t allowing me to upload .raw files)

Note: file_from_v4l2_1920x1080.raw is actually 3280x2464 size image. Named it 1920x1080 as I was trying to capture that resolution

3

hello talhatahir01022001,

>> Problem 1
the default mode selection logic, which selects a mode based on resolution, color format, and frame rate
you may add use_sensor_mode_id= "true"; property into sensor device tree, then you’ll be able to specify the mode index for capturing.
for instance,
$ v4l2-ctl -d /dev/video0 --set-fmt-video=width=1280,height=720,pixelformat=RG10 --set-ctrl bypass_mode=0 --set-ctrl sensor_mode=4 --stream-mmap --stream-count=100

>> Problem 2
let’s narrow down the issue, please try Argus NvRaw Tool to capture NvRaw and JPG file for checking.
you’re able to capture multiple format types via nvargus_nvraw simultaneously.
for instance,
$ sudo nvargus_nvraw --c 0 --mode 0 --format "nvraw, jpg" --file /home/nvidia/output

4

Thanks! Problem 1 solved with –set-ctrl sensor_mode=x ✅

Problem 2

Yes I was able to capture both nvraw and jpg. Both raw and JPG from nvargus tool are fine (no hallows like captured from v4l2). Here’s my terminal log:

nvargus_nvraw --c 0 --mode 0 --format “nvraw, jpg” --file /home/talha/Desktop/v4l2_issue/file_from_nvargus_3280x2464
nvargus_nvraw version 1.15.0
capture: Total captured frame count 90
±----- Exposure info results -----------+
| Exposure | Exposure Time | Sensor Gain |
|        0 |      0.044622 |     8.00000 |
±---------------------------------------+
writeToFile: file /home/talha/Desktop/v4l2_issue/file_from_nvargus_3280x2464.nvraw is successfully c
reated
writeToFile_Jpeg: file /home/talha/Desktop/v4l2_issue/file_from_nvargus_3280x2464.jpg is successfully
created

I’ve also attached both files (raw and jpg) here

file_from_nvargus_3280x2464
file_from_nvargus_3280x24641920×1442 666 KB

5

hello talhatahir01022001,

FYI,
we should have follow VI’s 64 byte aligned to set the correct stride, by setting the width alignment to 64. for example, for your 3280x2464 size image, the width should be padding to 3328 to fulfill the alignment.
or.. you may try again with use_sensor_mode_id= "true"; for capturing 1920x1080 sensor mode, this is the mode follow VI’s alignment.

6

Thanks! Yes for dimensions following VI’s alignment, v4l2 is giving me correct output (1080p, 720p etc.). But how can I get correct data for resolutions not following VI’s alignment (3280x2464)? I can’t capture 3328x2496 as it’s not supported by the sensor. Is there a way to extract correct data for non VI’s cases?

My usecase is that I’ve developed a cuda based software ISP and have added it’s support inside gstreamer using custom plugins. I can’t use nvargus as the src plugin of nvargus (nvarguscamerasrc) doesn’t allow us to bypass ISP and extract bayer data in gstreamer, so I’ve used v4l2src as my src for gstreamer:

gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-bayer,format=rggb10le,width=1280,height=720 ! cudaisp ! videoconvert ! autovideosink

7

hello talhatahir01022001,

there’s set control option, --set-ctrl preferred_stride=<value> to configure the surface stride.
you may try the following for testing.
$ v4l2-ctl -d /dev/video0 ... --set-ctrl preferred_stride=3328 ...

8

It captures the image but the issue is still same (no errors but still facing hallows)

v4l2-ctl -d /dev/video0 --set-fmt-video=width=3280,height=2464,pixelformat=RG10 --set-ctrl=bypass_mode=0,sensor_mode=0,preferred_stride=3328 --stream-mmap --stream-count=1 --stream-to=file_from_v4l2_3280x2464.raw
<

I’ve attached the raw and png files here. Note: Extracted png using my custom ISP in case you don’t have raw image viewer:

file_from_v4l2_3280x2464
file_from_v4l2_3280x24641920×1442 1.92 MB

9

hello talhatahir01022001,

let me have confirmation,
you’ve several sensor modes, this (hallows) is only happened with 3280x2464 sensor mode?

10

Hallows are occuring with all the modes except 720p and 1080p. Happening with:

3280x2464

3280x1848

1640x1232

All 3 resolutions whose width isn’t divisible by 64 i.e., not following VI’s alignment as you suggested above.

11

hello talhatahir01022001,

may I also know which Jetpack public release version you’re working with.

12

5.1

sudo apt show nvidia-jetpack -a
Package: nvidia-jetpack
Version: 5.1.5-b11
Priority: standard
Section: metapackages
Maintainer: NVIDIA Corporation
Installed-Size: 199 kB
Depends: nvidia-jetpack-runtime (= 5.1.5-b11), nvidia-jetpack-dev (= 5.1.5-b11)
Homepage: http://developer.nvidia.com/jetson
Download-Size: 29.3 kB
APT-Sources: https://repo.download.nvidia.com/jetson/common r35.6/main arm64 Packages
Description: NVIDIA Jetpack Meta Package

Package: nvidia-jetpack
Version: 5.1.4-b17
Priority: standard
Section: metapackages
Maintainer: NVIDIA Corporation
Installed-Size: 199 kB
Depends: nvidia-jetpack-runtime (= 5.1.4-b17), nvidia-jetpack-dev (= 5.1.4-b17)
Homepage: http://developer.nvidia.com/jetson
Download-Size: 29.3 kB
APT-Sources: https://repo.download.nvidia.com/jetson/common r35.6/main arm64 Packages
Description: NVIDIA Jetpack Meta Package
13

hello talhatahir01022001,

let’s give it a try to configure preferred_stride=6656 for testing.

14

Thanks it worked. The output image size is now 16,400,384 bytes i.e., 3328x2464x2 with zeroes padded on the right side. ✅

Q1. Any idea why 3328 stride wasn’t working. Because for 1640, I have to use a stride of 3328 instead of 1664 to get correct output

Q2. Is this issue persistent across all Jetson environments (different jetpack versions and jetson boards) or only this one? As I need to write my code that works on all Jetson platforms.

15

hello talhatahir01022001,

thanks for sharing test results, glad to know it works.

>> Q1
you may also dig into VI driver for checking the bytesperline calculated by driver side.
for instance,
$public_sources/kernel_src/kernel/nvidia-oot/drivers/media/platform/tegra/camera/vi/channel.c

static void tegra_channel_update_format(...)
{
        u32 denominator = (!bpp->denominator) ? 1 : bpp->denominator;
        u32 numerator = (!bpp->numerator) ? 1 : bpp->numerator;
        u32 bytesperline = (width * numerator / denominator);

        /* Align stride */
        if (chan->vi->fops->vi_stride_align)
                chan->vi->fops->vi_stride_align(&bytesperline);
        ...
        tegra_channel_fmt_align(chan, chan->fmtinfo,
                                &chan->format.width,
                                &chan->format.height,
                                &chan->format.bytesperline);

>> Q2
yes.. that’s due to VI’s 64-byte alignment, it applies to all Jetson platforms.
you may see-also Xavier TRM for [7.2.2 Video Input (VI)] section for details,
let me re-cap as below for your reference.

The VI5 output FIFO groups by stream number, and then by surface. This behavior is similar to the back-end processors in VI5 ATOMP.
Instead of four-atom boundaries, it operates on boundaries controlled by a programmable address alignment mask (register BATCH_MASK). The boundary can be set from 1 atom (64 bytes) to 1024 atoms (64 kilobytes).
Atoms of a thread are accumulated in the FIFO until at least one of the following conditions is met, upon which atoms are released.