How to calculate device-tree parameters for WDR DOL-2 frame [IMX585]

primoz.fiser · May 9, 2024, 10:21am

Here is the IMX585 image drawing for DOL 2-frame all-pixels Line Information (LI):

Here are the values from IMX585 DOL application note document:

HORIZONTAL DIRECTION
W1 = 8
W2 = 3840
W3 = 8
LI = 8 (see image drawing)

VERTICAL DIRECTION
H0* = (RHS1-2)/2 = (78-2)/2 = 38
H1 = 1
H2 = 1
H3 = 10
H4 = 10
H5 = 4
H6 = 8
H7 = 2160
H8 = 8
H9 = 1
H10*= (RHS1-2)/2 = (78-2)/2 = 38

*RHS1 register is set to 0x4E = 78.

Now, the question is how to properly set DOL-related device-tree properties:

active_w = W2 + LI + W1 + W3 = 3840 + 8 + 8 + 8 = 3864

active_h = (H7 + H4 + H5 + H6 + H8 + H9 + H0)*2 =
(2160 + 10 + 4 + 8 + 8 + 1 + 38) * 2 = 4458 ???

num_of_exposure = 2 (always 2 for DOL-2 frame)

num_of_ignored_lines = H3 + H4 + H5 + H6 =
10 + 10 + 4 + 8 = 32 ???

num_of_lines_offset_0 = H0 = 38

num_of_ignored_pixels = LI = 8

num_of_left_margin_pixels = W1 = 8

num_of_right_margin_pixels = W3 = 8

Uncertainties:

We are not sure for active_h if we shall include H3 (Ignored OB_1)?

Can you write down formula for active_h with the above information?

We are not sure for num_of_ignored_lines if we shall include H6 (Effective margin for color processing)?

Can you write down formula for num_of_ignored_lines with the above information?

Do you perhaps see any errors in calculation of other parameters? For now, we only think issues might be in the active_h and num_of_ignored_lines parameters.

NOTE: I have read this documentation many times for the IMX274 but cannot make sense how to calculate those values for IMX585.

Thanks,
Primoz

JerryChang · May 10, 2024, 3:28am

hello primoz.fiser,

your settings looks almost correct.
but… you may double check the active_h and image height, as we do not support odd resolution.
here’s an example,
active_h = [OB (12) + height (1080) + VBP (5)+ effective (3) ]* 2 = 2200
height = (active_h (2200) /2) - num_of_ignored_lines (12) - num_of_lines_offset (5) = 1083

however, such setting, height = 1083 is not supported.
in this scenario, here’s a little trick to increase num_of_ignored_lines by 1-line to make it as even numbers.

FYI, num_of_ignored_lines is used to crop lines from top.
this applies to both short and long exposures frames, increasing num_ignored_lines will result in a crop of the 1st line of the actual image.

some Q&As as below.
>> (1) Ignored OB_1 this is usually not considered into active_h
>> (2) & (3) please refer to above example for num_of_ignored_lines settings.

primoz.fiser · May 10, 2024, 8:03am

Hi @JerryChang ,

I applied your suggestions and came up with the following values for device-tree:

active_w = "3864";
active_h = "4458";
num_of_exposure = "2";
num_of_ignored_lines = "33"; // NOTE: Add + 1 to have even number
num_of_lines_offset_0 = "38";
num_of_ignored_pixels = "8";
num_of_left_margin_pixels = "8";
num_of_right_margin_pixels = "8";

NOTE: I added 1 to num_of_ignored_lines, so 32 + 1 = 33

However, nvargus_nvraw tool still doesn’t report any modes:

nvidia@jetson:~$ nvargus_nvraw --lps
nvargus_nvraw version 1.14.0
Number of supported sensor entries 0
Entry  Source Mode      Uniquename             Resolution   FR  BitDepth  Mode
Index  Index  Index                                             CSI Dyn   Type

NOTE: If I remove DOL mode from device-tree, I can see other non-DOL modes just fine.

What am I doing wrong?

JerryChang · May 13, 2024, 2:52am

hello primoz.fiser,

you should also have below settings defined in ISP override file to enable WDR support.
for instance, /var/nvidia/nvcam/settings/camera_overrides.isp
wdr.DOL.v4.EnableDOL = TRUE;
and, please also adding below to overwrite DRE values for HDR ratio.
ae.wdr.DreMin = 16;
ae.wdr.DreMax = 16;

primoz.fiser · May 13, 2024, 7:48am

Hi @JerryChang ,

I have played with DTS settings and somehow got the DOL mode working.

My current settings for IMX585 DOL mode are:

                mode6 { // IMX585_MODE_3856x2180_DOL
                    mclk_khz = "37125";
                    num_lanes = "4";
                    tegra_sinterface = "serial_a";
                    phy_mode = "DPHY";
                    discontinuous_clk = "no";
                    dpcm_enable = "false";
                    cil_settletime = "0";
                    /* FIXME: DOL related settings */
                    active_w = "3864";
                    active_h = "4444";

                    dynamic_pixel_bit_depth = "12";
                    csi_pixel_bit_depth = "12";
                    mode_type = "bayer_wdr_dol";
                    pixel_phase = "rggb";

                    readout_orientation = "0";
                    line_length = "6600";
                    inherent_gain = "1";
                    mclk_multiplier = "40.40";
                    pix_clk_hz = "1500000000";

                    gain_factor = "10";
                    min_gain_val = "0"; /* 0dB */
                    max_gain_val = "720"; /* 72dB */
                    step_gain_val = "3"; /* 0.3 */
                    default_gain = "0";
                    min_hdr_ratio = "1";
                    max_hdr_ratio = "64";
                    framerate_factor = "1000000";
                    min_framerate = "1500000"; /* 1.5 */
                    max_framerate = "30000000"; /* 30 */
                    step_framerate = "1";
                    default_framerate= "30000000";
                    exposure_factor = "1000000";
                    min_exp_time = "27";  /* us */
                    max_exp_time = "660000"; /* us */
                    step_exp_time = "1";
                    default_exp_time = "10000";/* us */
                    embedded_metadata_height = "0";

                    /* FIXME: DOL related settings */
                    num_of_exposure = "2";
                    num_of_ignored_lines = "24";
                    num_of_lines_offset_0 = "38";
                    num_of_ignored_pixels = "8";
                    num_of_left_margin_pixels = "8";
                    num_of_right_margin_pixels = "8";
                };

And in camera_override.isp I have:

# FIXME: DOL support
wdr.DOL.v4.EnableDOL = TRUE;
ae.wdr.DreMin = 16;
ae.wdr.DreMax = 16;

However DOL frames look a bit distorted full of strange artifacts?

Non-DOL mode frame output:

DOL mode frame output:

Do you have any ideas?

Thanks,
BR,
Primoz

JerryChang · May 13, 2024, 8:09am

hello primoz.fiser,

it’s glad to know that you’ve DOL sensor mode enabled.
here’re several suggestions.
(1) since you’re using 72dB gain, you may try lower the gain settings.
(2) you should have min_hdr_ratio = max_hdr_ratio, and also the same DRE values within ISP override file.
(3) indoor + outdoor scene might be too complex to identify the failure. please try change your target to LED lights to check long/short exposure frame fusion results.

BTW,
your following up issue looks like related to WDR image tuning.
unfortunately, we don’t support tuning stuffs via forum discussion threads,
please contact with sensor vendors, Jetson Camera Partners for further supports.
thanks

primoz.fiser · May 15, 2024, 6:19am

Hi @JerryChang ,

thanks for you suggestions and answers.

I guess sensor tuning for DOL is indeed required.
LEF:

SEF:

As it seems, LEF is too exposed and SEF is too dark…

One more question regarding DOL and the expected 30 FPS.

We are only getting 15 FPS instead of 30 FPS?

I am using the following gst pipeline to check frame rate:

$ gst-launch-1.0 nvarguscamerasrc sensor-mode=6 sensor-id=0 ! 'video/x-raw(memory:NVMM), width=3840, height=2160, format=NV12, framerate=30/1' ! queue ! fpsdisplaysink video-sink=fakesink -v
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0: sync = true
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)3840, height=(int)2160, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)3840, height=(int)2160, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)3840, height=(int)2160, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)3840, height=(int)2160, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = video/x-raw(memory:NVMM), width=(int)3840, height=(int)2160, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay.GstPad:src: caps = video/x-raw(memory:NVMM, meta:GstVideoOverlayComposition), width=(int)3840, height=(int)2160, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0.GstPad:sink: caps = video/x-raw(memory:NVMM, meta:GstVideoOverlayComposition), width=(int)3840, height=(int)2160, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay.GstPad:video_sink: caps = video/x-raw(memory:NVMM), width=(int)3840, height=(int)2160, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink: caps = video/x-raw(memory:NVMM), width=(int)3840, height=(int)2160, format=(string)NV12, framerate=(fraction)30/1
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3856 x 2180 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 3981.070801; Exposure Range min 27000, max 660000000;

GST_ARGUS: 3840 x 2160 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 3981.070801; Exposure Range min 27000, max 660000000;

GST_ARGUS: 1920 x 1080 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 3981.070801; Exposure Range min 27000, max 660000000;

GST_ARGUS: 3856 x 2180 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 15.848933; Exposure Range min 27000, max 660000000;

GST_ARGUS: 3840 x 2160 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 15.848933; Exposure Range min 27000, max 660000000;

GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 15.848933; Exposure Range min 27000, max 660000000;

GST_ARGUS: 3840 x 2160 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 7.079458; Exposure Range min 864000, max 20480000;

GST_ARGUS: 3816 x 2160 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 3981.070801; Exposure Range min 27000, max 660000000;

GST_ARGUS: 1924 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 3981.070801; Exposure Range min 27000, max 660000000;

GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 6 
   Output Stream W = 3840 H = 2160 
   seconds to Run    = 0 
   Frame Rate = 29.999999 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0: sync = true
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 9, dropped: 0, current: 16.98, average: 16.98
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 9, dropped: 0, current: 16.98, average: 16.98
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 17, dropped: 0, current: 14.99, average: 15.98
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 17, dropped: 0, current: 14.99, average: 15.98
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 25, dropped: 0, current: 15.01, average: 15.66
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 25, dropped: 0, current: 15.01, average: 15.66
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 33, dropped: 0, current: 15.00, average: 15.49
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 33, dropped: 0, current: 15.00, average: 15.49
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 41, dropped: 0, current: 14.99, average: 15.39
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 41, dropped: 0, current: 14.99, average: 15.39
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 49, dropped: 0, current: 15.00, average: 15.33
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 49, dropped: 0, current: 15.00, average: 15.33
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 57, dropped: 0, current: 15.01, average: 15.28
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 57, dropped: 0, current: 15.01, average: 15.28
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 65, dropped: 0, current: 15.00, average: 15.25
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 65, dropped: 0, current: 15.00, average: 15.25
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 73, dropped: 0, current: 15.00, average: 15.22
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 73, dropped: 0, current: 15.00, average: 15.22
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 81, dropped: 0, current: 15.01, average: 15.20
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 81, dropped: 0, current: 15.01, average: 15.20
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 89, dropped: 0, current: 14.99, average: 15.18
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 89, dropped: 0, current: 14.99, average: 15.18
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 97, dropped: 0, current: 15.00, average: 15.16
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 97, dropped: 0, current: 15.00, average: 15.16
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:06.965034937
Setting pipeline to NULL ...
GST_ARGUS: Cleaning up
CONSUMER: Done Success
GST_ARGUS: Done Success
Freeing pipeline ...

Can you suggest how to achieve 30 FPS?

BR,
Primoz

JerryChang · May 15, 2024, 9:06am

hello primoz.fiser,

usually, I’ll add VIC to convert the formats for checking.
for instance,
$ gst-launch-1.0 nvarguscamerasrc sensor-id=0 sensor-mode=0 ! 'video/x-raw(memory:NVMM),width=1920, height=1080, framerate=30/1, format=NV12' ! nvvidconv ! fpsdisplaysink text-overlay=0 name=sink_0 video-sink=fakesink sync=0 -v

primoz.fiser · May 15, 2024, 9:56am

Hi @JerryChang ,

even when using:
gst-launch-1.0 nvarguscamerasrc sensor-mode=6 sensor-id=0 ! 'video/x-raw(memory:NVMM), width=3840, height=2160, format=NV12, framerate=30/1' ! nvvidconv ! fpsdisplaysink text-overlay=0 name=sink_0 video-sink=fakesink sync=0 -v

we only get 15 FPS.

Do you have any reference information regarding DOL IMX274 Frame rate (FPS)?

ShaneCCC · May 16, 2024, 3:10am

@primoz.fiser
Please check by v4l2-ctl to confirm able get 30fps.

v4l2-ctl --stream-mmap -c bypass_mode=0

primoz.fiser · May 20, 2024, 11:02am

Hi @JerryChang and @ShaneCCC ,

I did some more debugging and found out we had XVS sub-sampling enabled in the IMX585 driver.

After disabling XVS sub-sampling, the frame rate is now correct at 30 FPS but output frames are no longer in sync anymore!

Here is the imx585 driver change:

+++ b/kernel/nvidia/drivers/media/i2c/imx585_mode_tbls.h
@@ -943,7 +943,7 @@ static const imx585_reg mode_3856x2180_dol[] = {
     {DATARATE_SEL,         0x02}, /* 1782 Mbps */
     {WDMODE,               0x01}, /* DOL 2 frame mode */
     {ADDMODE,              0x00}, /* all pixel */
-    {THIN_V_EN,            0x01}, /* enable (subsampling) */
+    {THIN_V_EN,            0x00}, /* disable (subsampling) */
     {VCMODE,               0x00}, /* Enable Line information (LI) output */
     {LANEMODE,             0x03}, /* 4 lane */
     {SHR0_LOW,             0x06},

Here is the resulting output fusioned frame:

Do you know if the reference IMX274 implementation had XVS sub-sampling enabled or not?

Or maybe why are we getting frame sync issues when disabling XVS subsampling…

BR,
Primoz

JerryChang · May 21, 2024, 2:52am

hello primoz.fiser,

may I double confirm the Jetpack/L4T release version you’re working with.
and… what’s this XVS sub-sampling actually did. for example, is 3856x2180 the full resolution of IMX585?

primoz.fiser · May 21, 2024, 8:51am

@JerryChang

may I double confirm the Jetpack/L4T release version you’re working with.

nvidia@jetson:~$ cat /etc/nv_tegra_release
# R35 (release), REVISION: 4.1, GCID: 33958178, BOARD: t186ref, EABI: aarch64, DATE: Tue Aug  1 19:57:35 UTC 2023
nvidia@jetson:~$

and… what’s this XVS sub-sampling actually did.

I disabled subsampling because VMAX register in our case is set to VMAX = 4500 for 30 FPS.

Here is snippet from IMX585 datasheet:

and short explanation:

for example, is 3856x2180 the full resolution of IMX585?

Total number of pixels 3856 x 2220

Number of effective pixels 3856 x 2180

Number of active pixels 3856 x 2176

Number of recommended recording pixels 3840 x 2160

JerryChang · May 22, 2024, 2:49am

hello primoz.fiser,

IMX274 doesn’t had XVS sub-sampling enabled

such fusion shift issue has happened after you revise VMAX, right?
if yes, you may give it a try to configure VBP (Vertical Blank Period) to adjust long/short frames.

system · June 19, 2024, 3:59am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Failing to get input from CSI camera (IMX219-83) Jetson Nano camera , ubuntu , gstreamer	29	1526	September 25, 2023
Why is the imx477 camera image not being captured via FPD Link ds90ub95x FPD Link via virtual channels Jetson Xavier NX camera	9	203	August 2, 2024
Jetson AGX Orin FPD-LinkIII_ds90ub95x Devicetree How to configuration? Jetson AGX Orin camera	21	422	July 2, 2024
Gstreamer reports Raspberry Pi camera streaming at 120fps when in reality it is only 60fps Jetson Nano camera , gstreamer	53	3835	July 19, 2022
IMX477 camera is not working on my Jetson Nano B01 Jetson Nano camera	9	976	August 25, 2022
Need help on H264 encoded 4K@30fps video framerate Jetson Xavier NX camera , gstreamer	10	1142	October 18, 2021
External Trigger works but camera cuts FPS in half Jetson Xavier NX camera	8	1290	August 3, 2022
IMX378 camera freezes on Jetpack 4.2 with jetson nano Jetson Nano	20	2122	October 18, 2021
Multi-camera Jetson Xavier NX configuration, more than 6 cameras Jetson Xavier NX camera , nvbugs	24	2907	November 4, 2021
Problems with nvarguscamerasrc using the last imx565 mode Jetson AGX Xavier camera , gstreamer	43	368	July 11, 2024

How to calculate device-tree parameters for WDR DOL-2 frame [IMX585]

Related topics