How to calculate device-tree parameters for WDR DOL-2 frame [IMX585]

Here is the IMX585 image drawing for DOL 2-frame all-pixels Line Information (LI):

Here are the values from IMX585 DOL application note document:

HORIZONTAL DIRECTION
W1 = 8
W2 = 3840
W3 = 8
LI = 8 (see image drawing)

VERTICAL DIRECTION
H0* = (RHS1-2)/2 = (78-2)/2 = 38
H1 = 1
H2 = 1
H3 = 10
H4 = 10
H5 = 4
H6 = 8
H7 = 2160
H8 = 8
H9 = 1
H10*= (RHS1-2)/2 = (78-2)/2 = 38


*RHS1 register is set to 0x4E = 78.

Now, the question is how to properly set DOL-related device-tree properties:

active_w = W2 + LI + W1 + W3 = 3840 + 8 + 8 + 8 = 3864

active_h = (H7 + H4 + H5 + H6 + H8 + H9 + H0)*2 =
(2160 + 10 + 4 + 8 + 8 + 1 + 38) * 2 = 4458 ???

num_of_exposure = 2 (always 2 for DOL-2 frame)

num_of_ignored_lines = H3 + H4 + H5 + H6 =
10 + 10 + 4 + 8 = 32 ???

num_of_lines_offset_0 = H0 = 38

num_of_ignored_pixels = LI = 8

num_of_left_margin_pixels = W1 = 8

num_of_right_margin_pixels = W3 = 8

Uncertainties:

  1. We are not sure for active_h if we shall include H3 (Ignored OB_1)?

Can you write down formula for active_h with the above information?

  1. We are not sure for num_of_ignored_lines if we shall include H6 (Effective margin for color processing)?

Can you write down formula for num_of_ignored_lines with the above information?

  1. Do you perhaps see any errors in calculation of other parameters? For now, we only think issues might be in the active_h and num_of_ignored_lines parameters.

NOTE: I have read this documentation many times for the IMX274 but cannot make sense how to calculate those values for IMX585.

Thanks,
Primoz

hello primoz.fiser,

your settings looks almost correct.
but… you may double check the active_h and image height, as we do not support odd resolution.
here’s an example,
active_h = [OB (12) + height (1080) + VBP (5)+ effective (3) ]* 2 = 2200
height = (active_h (2200) /2) - num_of_ignored_lines (12) - num_of_lines_offset (5) = 1083

however, such setting, height = 1083 is not supported.
in this scenario, here’s a little trick to increase num_of_ignored_lines by 1-line to make it as even numbers.

FYI, num_of_ignored_lines is used to crop lines from top.
this applies to both short and long exposures frames, increasing num_ignored_lines will result in a crop of the 1st line of the actual image.

some Q&As as below.
>> (1) Ignored OB_1 this is usually not considered into active_h
>> (2) & (3) please refer to above example for num_of_ignored_lines settings.

Hi @JerryChang ,

I applied your suggestions and came up with the following values for device-tree:

active_w = "3864";
active_h = "4458";
num_of_exposure = "2";
num_of_ignored_lines = "33"; // NOTE: Add + 1 to have even number
num_of_lines_offset_0 = "38";
num_of_ignored_pixels = "8";
num_of_left_margin_pixels = "8";
num_of_right_margin_pixels = "8";

NOTE: I added 1 to num_of_ignored_lines, so 32 + 1 = 33

However, nvargus_nvraw tool still doesn’t report any modes:

nvidia@jetson:~$ nvargus_nvraw --lps
nvargus_nvraw version 1.14.0
Number of supported sensor entries 0
Entry  Source Mode      Uniquename             Resolution   FR  BitDepth  Mode
Index  Index  Index                                             CSI Dyn   Type

NOTE: If I remove DOL mode from device-tree, I can see other non-DOL modes just fine.

What am I doing wrong?

hello primoz.fiser,

you should also have below settings defined in ISP override file to enable WDR support.
for instance, /var/nvidia/nvcam/settings/camera_overrides.isp
wdr.DOL.v4.EnableDOL = TRUE;
and, please also adding below to overwrite DRE values for HDR ratio.
ae.wdr.DreMin = 16;
ae.wdr.DreMax = 16;

Hi @JerryChang ,

I have played with DTS settings and somehow got the DOL mode working.

My current settings for IMX585 DOL mode are:

                mode6 { // IMX585_MODE_3856x2180_DOL
                    mclk_khz = "37125";
                    num_lanes = "4";
                    tegra_sinterface = "serial_a";
                    phy_mode = "DPHY";
                    discontinuous_clk = "no";
                    dpcm_enable = "false";
                    cil_settletime = "0";
                    /* FIXME: DOL related settings */
                    active_w = "3864";
                    active_h = "4444";

                    dynamic_pixel_bit_depth = "12";
                    csi_pixel_bit_depth = "12";
                    mode_type = "bayer_wdr_dol";
                    pixel_phase = "rggb";

                    readout_orientation = "0";
                    line_length = "6600";
                    inherent_gain = "1";
                    mclk_multiplier = "40.40";
                    pix_clk_hz = "1500000000";

                    gain_factor = "10";
                    min_gain_val = "0"; /* 0dB */
                    max_gain_val = "720"; /* 72dB */
                    step_gain_val = "3"; /* 0.3 */
                    default_gain = "0";
                    min_hdr_ratio = "1";
                    max_hdr_ratio = "64";
                    framerate_factor = "1000000";
                    min_framerate = "1500000"; /* 1.5 */
                    max_framerate = "30000000"; /* 30 */
                    step_framerate = "1";
                    default_framerate= "30000000";
                    exposure_factor = "1000000";
                    min_exp_time = "27";  /* us */
                    max_exp_time = "660000"; /* us */
                    step_exp_time = "1";
                    default_exp_time = "10000";/* us */
                    embedded_metadata_height = "0";

                    /* FIXME: DOL related settings */
                    num_of_exposure = "2";
                    num_of_ignored_lines = "24";
                    num_of_lines_offset_0 = "38";
                    num_of_ignored_pixels = "8";
                    num_of_left_margin_pixels = "8";
                    num_of_right_margin_pixels = "8";
                };

And in camera_override.isp I have:

# FIXME: DOL support
wdr.DOL.v4.EnableDOL = TRUE;
ae.wdr.DreMin = 16;
ae.wdr.DreMax = 16;

However DOL frames look a bit distorted full of strange artifacts?

Non-DOL mode frame output:


DOL mode frame output:

Do you have any ideas?

Thanks,
BR,
Primoz

hello primoz.fiser,

it’s glad to know that you’ve DOL sensor mode enabled.
here’re several suggestions.
(1) since you’re using 72dB gain, you may try lower the gain settings.
(2) you should have min_hdr_ratio = max_hdr_ratio, and also the same DRE values within ISP override file.
(3) indoor + outdoor scene might be too complex to identify the failure. please try change your target to LED lights to check long/short exposure frame fusion results.


BTW,
your following up issue looks like related to WDR image tuning.
unfortunately, we don’t support tuning stuffs via forum discussion threads,
please contact with sensor vendors, Jetson Camera Partners for further supports.
thanks

1 Like

Hi @JerryChang ,

thanks for you suggestions and answers.

I guess sensor tuning for DOL is indeed required.
LEF:


SEF:

As it seems, LEF is too exposed and SEF is too dark…

One more question regarding DOL and the expected 30 FPS.

We are only getting 15 FPS instead of 30 FPS?

I am using the following gst pipeline to check frame rate:

$ gst-launch-1.0 nvarguscamerasrc sensor-mode=6 sensor-id=0 ! 'video/x-raw(memory:NVMM), width=3840, height=2160, format=NV12, framerate=30/1' ! queue ! fpsdisplaysink video-sink=fakesink -v
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0: sync = true
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)3840, height=(int)2160, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)3840, height=(int)2160, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)3840, height=(int)2160, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)3840, height=(int)2160, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = video/x-raw(memory:NVMM), width=(int)3840, height=(int)2160, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay.GstPad:src: caps = video/x-raw(memory:NVMM, meta:GstVideoOverlayComposition), width=(int)3840, height=(int)2160, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0.GstPad:sink: caps = video/x-raw(memory:NVMM, meta:GstVideoOverlayComposition), width=(int)3840, height=(int)2160, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay.GstPad:video_sink: caps = video/x-raw(memory:NVMM), width=(int)3840, height=(int)2160, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink: caps = video/x-raw(memory:NVMM), width=(int)3840, height=(int)2160, format=(string)NV12, framerate=(fraction)30/1
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3856 x 2180 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 3981.070801; Exposure Range min 27000, max 660000000;

GST_ARGUS: 3840 x 2160 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 3981.070801; Exposure Range min 27000, max 660000000;

GST_ARGUS: 1920 x 1080 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 3981.070801; Exposure Range min 27000, max 660000000;

GST_ARGUS: 3856 x 2180 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 15.848933; Exposure Range min 27000, max 660000000;

GST_ARGUS: 3840 x 2160 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 15.848933; Exposure Range min 27000, max 660000000;

GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 15.848933; Exposure Range min 27000, max 660000000;

GST_ARGUS: 3840 x 2160 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 7.079458; Exposure Range min 864000, max 20480000;

GST_ARGUS: 3816 x 2160 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 3981.070801; Exposure Range min 27000, max 660000000;

GST_ARGUS: 1924 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 3981.070801; Exposure Range min 27000, max 660000000;

GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 6 
   Output Stream W = 3840 H = 2160 
   seconds to Run    = 0 
   Frame Rate = 29.999999 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0: sync = true
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 9, dropped: 0, current: 16.98, average: 16.98
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 9, dropped: 0, current: 16.98, average: 16.98
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 17, dropped: 0, current: 14.99, average: 15.98
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 17, dropped: 0, current: 14.99, average: 15.98
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 25, dropped: 0, current: 15.01, average: 15.66
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 25, dropped: 0, current: 15.01, average: 15.66
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 33, dropped: 0, current: 15.00, average: 15.49
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 33, dropped: 0, current: 15.00, average: 15.49
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 41, dropped: 0, current: 14.99, average: 15.39
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 41, dropped: 0, current: 14.99, average: 15.39
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 49, dropped: 0, current: 15.00, average: 15.33
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 49, dropped: 0, current: 15.00, average: 15.33
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 57, dropped: 0, current: 15.01, average: 15.28
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 57, dropped: 0, current: 15.01, average: 15.28
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 65, dropped: 0, current: 15.00, average: 15.25
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 65, dropped: 0, current: 15.00, average: 15.25
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 73, dropped: 0, current: 15.00, average: 15.22
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 73, dropped: 0, current: 15.00, average: 15.22
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 81, dropped: 0, current: 15.01, average: 15.20
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 81, dropped: 0, current: 15.01, average: 15.20
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 89, dropped: 0, current: 14.99, average: 15.18
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 89, dropped: 0, current: 14.99, average: 15.18
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 97, dropped: 0, current: 15.00, average: 15.16
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 97, dropped: 0, current: 15.00, average: 15.16
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:06.965034937
Setting pipeline to NULL ...
GST_ARGUS: Cleaning up
CONSUMER: Done Success
GST_ARGUS: Done Success
Freeing pipeline ...

Can you suggest how to achieve 30 FPS?

BR,
Primoz

hello primoz.fiser,

usually, I’ll add VIC to convert the formats for checking.
for instance,
$ gst-launch-1.0 nvarguscamerasrc sensor-id=0 sensor-mode=0 ! 'video/x-raw(memory:NVMM),width=1920, height=1080, framerate=30/1, format=NV12' ! nvvidconv ! fpsdisplaysink text-overlay=0 name=sink_0 video-sink=fakesink sync=0 -v

Hi @JerryChang ,

even when using:
gst-launch-1.0 nvarguscamerasrc sensor-mode=6 sensor-id=0 ! 'video/x-raw(memory:NVMM), width=3840, height=2160, format=NV12, framerate=30/1' ! nvvidconv ! fpsdisplaysink text-overlay=0 name=sink_0 video-sink=fakesink sync=0 -v

we only get 15 FPS.

Do you have any reference information regarding DOL IMX274 Frame rate (FPS)?

@primoz.fiser
Please check by v4l2-ctl to confirm able get 30fps.

v4l2-ctl --stream-mmap -c bypass_mode=0

Hi @JerryChang and @ShaneCCC ,

I did some more debugging and found out we had XVS sub-sampling enabled in the IMX585 driver.

After disabling XVS sub-sampling, the frame rate is now correct at 30 FPS but output frames are no longer in sync anymore!

Here is the imx585 driver change:

+++ b/kernel/nvidia/drivers/media/i2c/imx585_mode_tbls.h
@@ -943,7 +943,7 @@ static const imx585_reg mode_3856x2180_dol[] = {
     {DATARATE_SEL,         0x02}, /* 1782 Mbps */
     {WDMODE,               0x01}, /* DOL 2 frame mode */
     {ADDMODE,              0x00}, /* all pixel */
-    {THIN_V_EN,            0x01}, /* enable (subsampling) */
+    {THIN_V_EN,            0x00}, /* disable (subsampling) */
     {VCMODE,               0x00}, /* Enable Line information (LI) output */
     {LANEMODE,             0x03}, /* 4 lane */
     {SHR0_LOW,             0x06},

Here is the resulting output fusioned frame:

Do you know if the reference IMX274 implementation had XVS sub-sampling enabled or not?

Or maybe why are we getting frame sync issues when disabling XVS subsampling…

BR,
Primoz

hello primoz.fiser,

may I double confirm the Jetpack/L4T release version you’re working with.
and… what’s this XVS sub-sampling actually did. for example, is 3856x2180 the full resolution of IMX585?

@JerryChang

may I double confirm the Jetpack/L4T release version you’re working with.

nvidia@jetson:~$ cat /etc/nv_tegra_release
# R35 (release), REVISION: 4.1, GCID: 33958178, BOARD: t186ref, EABI: aarch64, DATE: Tue Aug  1 19:57:35 UTC 2023
nvidia@jetson:~$

and… what’s this XVS sub-sampling actually did.

I disabled subsampling because VMAX register in our case is set to VMAX = 4500 for 30 FPS.

Here is snippet from IMX585 datasheet:

and short explanation:

for example, is 3856x2180 the full resolution of IMX585?

Total number of pixels 3856 x 2220

Number of effective pixels 3856 x 2180

Number of active pixels 3856 x 2176

Number of recommended recording pixels 3840 x 2160

hello primoz.fiser,

IMX274 doesn’t had XVS sub-sampling enabled

such fusion shift issue has happened after you revise VMAX, right?
if yes, you may give it a try to configure VBP (Vertical Blank Period) to adjust long/short frames.