V4L2 controls not working for Leopard Imaging LI-AR0234CS-GMSL2-OWL cameras

[ethan41]

*** Summary ***

I’m capturing frames from a LI-AR0234CS-GMSL2-OWL camera using the V4L2 interface.

For this investigation, I’m using v4l2-ctl to manipulate V4L2 controls and capture raw frame data, but I get similar results with my own code.

I can capture valid raw frames (albeit with a different pixel format than expected) but modifying the standard V4L2 controls (exposure, gain) has no effect.

*** System details ***

$ cat /etc/nv_tegra_release
# R35 (release), REVISION: 1.0, GCID: 31346300, BOARD: t186ref, EABI: aarch64, DATE: Thu Aug 25 18:41:45 UTC 2022

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.5 LTS
Release:	20.04
Codename:	focal

$ uname -a
Linux mtd-orin 5.10.104-tegra #1 SMP PREEMPT Wed Aug 10 20:17:07 PDT 2022 aarch64 aarch64 aarch64 GNU/Linux

*** Procedure and Results ***
I’m using this command line, adapted from the Leopard docs:

$ v4l2-ctl -V --set-fmt-video=width=1920,height=1200,pixelformat=RG10 --set-ctrl bypass_mode=0,gain=1600,exposure=22000,exposure_short=22000  --stream-mmap  --stream-count=10 --stream-to=frames.raw

Those control settings set gain and all exposure controls to max values, subsequently confirmed as having been set by running with -L to show current control values.
The above does yield 10 frames of data that I am able to subsequently decode, but after issuing the following complaint:

The pixelformat 'RG10' is invalid
Format Video Capture:
Width/Height      : 1920/1200
Pixel Format      : 'BA10' (10-bit Bayer GRGR/BGBG)
Field             : None
Bytes per Line    : 3840
Size Image        : 4608000
Colorspace        : sRGB
Transfer Function : Default (maps to sRGB)
YCbCr/HSV Encoding: Default (maps to ITU-R 601)
Quantization      : Default (maps to Full Range)
Flags             :

So I’m getting data, but the RG10 format is not available (only BA10, which is a different Bayer pattern). That is not the primary problem, though.

Here is the debayered image (last of 10 captured frames):

It looks pretty dark. This room is well lit.

Some details:

• I’m debayering in the crudest possible way, so ignore any small-scale color aberration, etc.
• I’m decoding as though the input Bayer data were 16 bits per pixel (despite BA10 claiming to be the bottom 10 bits) because the input in fact occupies the full range (Input bounds: 2114 → 65535).

Then I run again with gain and exposure all set to their minimum values, like this:

$ v4l2-ctl -V --set-fmt-video=width=1920,height=1200,pixelformat=BA10 --set-ctrl bypass_mode=0,gain=100,exposure=28,exposure_short=28  --stream-mmap  --stream-count=10 --stream-to=frames.raw -d /dev/video0

I get this:
[ not allowed to post second image here but it’s a new image that’s similarly dark]

Essentially identical despite totally different control settings (which all succeeded, as confirmed by running v4l2-ctl
with -L).

The controls enumerated on the V4L2 interface are inert. Exposure and gain controls reflect my settings but have no effect on the captured data.

What might I doing wrong?

Here’s what the controls show after the first run (max exposure, etc.):

$ v4l2-ctl -V  -d /dev/video0 -L
Format Video Capture:
Width/Height      : 1920/1200
Pixel Format      : 'BA10' (10-bit Bayer GRGR/BGBG)
Field             : None
Bytes per Line    : 3840
Size Image        : 4608000
Colorspace        : sRGB
Transfer Function : Default (maps to sRGB)
YCbCr/HSV Encoding: Default (maps to ITU-R 601)
Quantization      : Default (maps to Full Range)
Flags             :

Camera Controls

                     group_hold 0x009a2003 (bool)   : default=0 value=0 flags=execute-on-write
                     hdr_enable 0x009a2004 (intmenu): min=0 max=1 default=0 value=0
0: 0 (0x0)
1: 1 (0x1)
                    eeprom_data 0x009a2005 (str)    : min=0 max=1024 step=2 value='ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000' flags=read-only, has-payload
                    sensor_mode 0x009a2008 (int64)  : min=0 max=1 step=1 default=0 value=0 flags=slider
                           gain 0x009a2009 (int64)  : min=100 max=1600 step=1 default=100 value=1600 flags=slider
                       exposure 0x009a200a (int64)  : min=28 max=22000 step=1 default=22000 value=22000 flags=slider
                     frame_rate 0x009a200b (int64)  : min=30000000 max=30000000 step=30000000 default=30000000 value=30000000 flags=slider
                 exposure_short 0x009a200c (int64)  : min=28 max=22000 step=1 default=22000 value=22000 flags=slider
                  stereo_eeprom 0x009a200d (u8)     : min=0 max=4096 step=2 default=0 [160] flags=read-only, has-payload
           sensor_configuration 0x009a2032 (u32)    : min=0 max=4294967295 step=1 default=0 [22] flags=read-only, volatile, has-payload
         sensor_mode_i2c_packet 0x009a2033 (u32)    : min=0 max=4294967295 step=1 default=0 [1026] flags=read-only, volatile, has-payload
      sensor_control_i2c_packet 0x009a2034 (u32)    : min=0 max=4294967295 step=1 default=0 [1026] flags=read-only, volatile, has-payload
                    bypass_mode 0x009a2064 (intmenu): min=0 max=1 default=0 value=0
0: 0 (0x0)
1: 1 (0x1)
                override_enable 0x009a2065 (intmenu): min=0 max=1 default=0 value=0
0: 0 (0x0)
1: 1 (0x1)
                   height_align 0x009a2066 (int)    : min=1 max=16 step=1 default=1 value=1
                     size_align 0x009a2067 (intmenu): min=0 max=2 default=0 value=0
0: 1 (0x1)
1: 65536 (0x10000)
2: 131072 (0x20000)
               write_isp_format 0x009a2068 (int)    : min=1 max=1 step=1 default=1 value=1
       sensor_signal_properties 0x009a2069 (u32)    : min=0 max=4294967295 step=1 default=0 [30][18] flags=read-only, has-payload
        sensor_image_properties 0x009a206a (u32)    : min=0 max=4294967295 step=1 default=0 [30][16] flags=read-only, has-payload
      sensor_control_properties 0x009a206b (u32)    : min=0 max=4294967295 step=1 default=0 [30][36] flags=read-only, has-payload
              sensor_dv_timings 0x009a206c (u32)    : min=0 max=4294967295 step=1 default=0 [30][16] flags=read-only, has-payload
               low_latency_mode 0x009a206d (bool)   : default=0 value=0
               preferred_stride 0x009a206e (int)    : min=0 max=65535 step=1 default=0 value=0
                   sensor_modes 0x009a2082 (int)    : min=0 max=30 step=1 default=30 value=1 flags=read-only

[ethan41]

Here’s the second captured image (with min exposure and gain) that I was forbidden from adding to the description above:

[JerryChang]

hello ethan41,

it looks the settings has applied with your CID controls.
for example, --set-ctrl bypass_mode=0,gain=1600,exposure=22000,exposure_short=22000

@ethan41

                           gain 0x009a2009 (int64)  : min=100 max=1600 step=1 default=100 value=1600 flags=slider
                       exposure 0x009a200a (int64)  : min=28 max=22000 step=1 default=22000 value=22000 flags=slider
                 exposure_short 0x009a200c (int64)  : min=28 max=22000 step=1 default=22000 value=22000 flags=slider

since R35.1 is quite an early JP-5 release version,
is it possible for moving to R35.3.1 or even the latest release R35.4.1 for confirmation?

[ethan41]

Yes, the point I’m making is that the settings have been applied with the CID controls, but they have no effect on the captured data.

The controls do not have any effect.

[JerryChang]

hello ethan41,

is it possible to add some debug prints in sensor driver layer?
please examine the registers are writing with those settings correctly.

[JerryChang]

hello ethan41,

so… you’re going to dump a Bayer raw with those gain/exposure controls, right?
you may see-also Argus NvRaw Tool for capturing nvraw with your exposure time and sensor gain settings.
please share the file for reference if the results still not match your expectation.

[jeremy.shannon1]

I’ve confirmed that the behavior is identical with R35.3.1 - gain/exposure settings have no apparent effect on the output image(s).

[JerryChang]

hello jeremy.shannon1,

do you also using v4l2-ctl to set controls for capturing raw frame data?
could you please have a try with Argus NvRaw Tool for capturing nvraw with your exposure time and sensor gain settings.

[jeremy.shannon1]

I tried these commands with the nvargus_nvraw command line tool:

nvargus_nvraw --c 0 --mode 0 --file ./nvraw_low.jpg --format “jpg” --skipframes 20 --exp0 “0.000028, 1”

nvargus_nvraw --c 0 --mode 0 --file ./nvraw_high.jpg --format “jpg” --skipframes 20 --exp0 “0.022000, 16”

… and got these results

So I suppose that confirms that Owl camera gain/exposure can be controlled with the nvraw utility but not v4l2-ctl. I think the question, then, is: is it impossible to control the Owl camera exposure/gain through the v4l2 interface?

[jeremy.shannon1]

By the way, here’s the result of the

$ nvargus_nvraw --c 0 --mode 0 --sensorinfo

command:

[JerryChang]

hello jeremy.shannon1,

all right, it looks like gain/exposure can be controlled with the nvraw utility.
as mentioned in comment #4, set-ctrl were sending gain/exposure to low-level driver, please also dig into sensor driver layer to check registers were writing with those settings correctly.

[ethan41]

Jerry,

Can we take this as confirmation that the V4L2 interface/driver does not work correctly for this camera?

It’s not surprising that a proprietary interface can actuate the camera controls, but we’re interested in using the standard V4L2 interface. We haven’t modified it and we’re not building it from source.

Please confirm that, as far as you can tell, the V4L2 driver is not functioning correctly and that we’re not just holding it wrong.

Thanks,
Ethan

[JerryChang]

hello ethan41,

I’ve confirmed I cannot reproduce this locally.
you should have 2-step approach, please execute v4l pipeline to set gain/exposures first, and later to capture the frames with the corresponding settings.
for example,
$ v4l2-ctl -V --set-fmt-video=width=2592,height=1944,pixelformat=BG10 --set-ctrl bypass_mode=0 --set-ctrl gain=128 --set-ctrl exposure=250000
$ v4l2-ctl --set-fmt-video=width=2592,height=1944,pixelformat=BG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=3 --stream-to=frames-adjust.raw -d /dev/video0

here’re kernel layer debug messages.
it’s also identical with those v4l control settings, and I can see the difference from debayer results.

[ 5690.915396] JC: ov5693_set_gain: val: 128 gain 205
[ 5690.915875] JC: ov5693_set_exposure: val: 250000 coarse_time: 1978
[ 5690.916500] JC: ov5693_set_frame_rate: val: 30000000 frame_length: 1984

please give it a try, thanks

[jeremy.shannon1]

I gave this a try and it did not work for me. No apparent difference between the output images.

Here’s my dmesg output:

[JerryChang]

hello jeremy.shannon1,

is it really capture a frame? it looks some RCE error has reported.

[jeremy.shannon1]

It did capture a frame, though it came out as ten frames stacked vertically (as it tends to do with some, but rarely all, of the various outputs of the Leopard raw conversion tool… and I have no clue why).

[JerryChang]

I doubt it’s VI to reset the capture channel to waive camera controls.
how about using --stream-skip to drop some frames.
for example, v4l2-ctl --set-fmt-video=width=1920,height=1200,pixelformat=BA10 --set-ctrl bypass_mode=0 --stream-mmap --stream-skip=9 --stream-count=1 --set-ctrl=gain=200 --set-ctrl=exposure=2800 --stream-to=ar0234.raw -d /dev/video0

[jeremy.shannon1]

We did try --stream-skip with no change in results. Good news, though - we were able to determine in email communications with Nvidia reps that the controls do work as long as the stream is started through some other non-V4L2 method before issuing the commands. This is apparently a bug in the Owl camera and Leopard is looking into it.

Thanks for the assistance Jerry.

[JerryChang]

@jeremy.shannon1

This is apparently a bug in the Owl camera and Leopard is looking into it.

thanks for issue narrow down, and status update.