Issues with Streaming Modes on Jetson Xavier NX (JetPack 5.1.3)

Robotics & Edge Computing Jetson & Embedded Systems Jetson Xavier NX
1

Hello,

I am working with a Jetson Xavier NX running JetPack version 5.1.3. In my DTS file, I have defined three sensor modes (Mode 0, Mode 1, and Mode 2), but none of these modes are streaming. However, I am able to stream Mode 3, which is not defined in the DTS file.

Here’s the configuration for Mode 0 (which is similar for Mode 1 and Mode 2 with slight variations):

define SENSOR_MODE0 *
*** mclk_khz = “24000”; **
*** discontinuous_clk = “no”; ***
*** dpcm_enable = “false”; ***
*** cil_settletime = “0”; ***
*** dynamic_pixel_bit_depth = “16”; ***
*** csi_pixel_bit_depth = “16”; ***
*** mode_type = “yuv”; ***
*** pixel_phase = “uyvy”; ***
*** active_w = “1280”; ***
*** active_h = “720”; ***
*** readout_orientation = “0”; ***
*** line_length = “2250”; ***
*** inherent_gain = “1”; ***
*** pix_clk_hz = “74250000”; ***
*** serdes_pix_clk_hz = “350000000”; ***
*** gain_factor = “10”; ***
*** min_gain_val = “0”; ***
*** max_gain_val = “300”; ***
*** step_gain_val = “3”; ***
*** default_gain = “0”; ***
*** min_hdr_ratio = “1”; ***
*** max_hdr_ratio = “1”; ***
*** framerate_factor = “1000000”; ***
*** min_framerate = “30000000”; ***
*** max_framerate = “30000000”; ***
*** step_framerate = “1”; ***
*** default_framerate = “30000000”; ***
*** exposure_factor = “1000000”; ***
*** min_exp_time = “24”; ***
*** max_exp_time = “33333”; ***
*** step_exp_time = “1”; ***
*** default_exp_time = “33333”; ***
*** embedded_metadata_height = “0”; ***

In the driver, I am using four modes (Mode 0, Mode 1, Mode 2, and Mode 3). Mode 3 is streaming successfully, but Mode 0, Mode 1, and Mode 2 are not streaming.

How is it possible that Mode 3, which is not even defined in the DTS file, is streaming? And which configurations Mode 3 is using by default? Why are the first three modes (Mode 0, Mode 1, and Mode 2) not streaming, even though they are defined in the DTS?

Additionally, I would like to understand how Mode X in the DTS file maps to resolution_array[X] in the driver. Can someone explain how these mappings work, and what might be causing the streaming issue with the first three modes?

Thank You

3

The modex define in device tree is Argus using.
Suppose you have 4 mode by v4l2-ctl --list-formats-ext you need to confirm both of these modes able to streaming by v4l2-ctl command first.

4

Hi ShaneCCC,

Thanks for the reply!

/*static const struct camera_common_frmfmt sensor_frmfmt = {
{{640, 480}, &__60fps, 1, 0, 0},
{{1280, 720}, &__60fps, 1, 0, 1},
{{1920, 1080}, &__60fps, 1, 0, 2},
{{2592, 1944}, &__60fps, 1, 0, 3},
{{3840, 2160}, &__60fps, 1, 0, 4},
{{4208, 3120}, &__60fps, 1, 0, 5},

};*/

The sensor_frmfmt[] array consists of elements of type camera_common_frmfmt. The first three elements in the sensor_frmfmt[] array (i.e., the 0th, 1st, and 2nd index elements) are not streaming, regardless of the sensor mode. However, the remaining elements are streaming. Could you clarify the reason for this behavior?

Thank you!

5

What’s the trace log.
Suppose the problem relative with the sensor REG configuration.

6

Hi Shane,

There is no need to doubt the sensor configurations. Changing the resolution index positions is causing the issue. If we swap the 0th index resolution with the 3rd index resolution, then 640x480 will stream, and 2592x1944 won’t stream. It’s not a sensor configuration issue; it’s clearly an index issue.

/*static const struct camera_common_frmfmt sensor_frmfmt = {
{{2592, 1944}, &__60fps, 1, 0, 3},
{{1280, 720}, &__60fps, 1, 0, 1},
{{1920, 1080}, &__60fps, 1, 0, 2},
{{640, 480}, &__60fps, 1, 0, 0},
{{3840, 2160}, &__60fps, 1, 0, 4},
{{4208, 3120}, &__60fps, 1, 0, 5},

};*/

After swapping the indexes, I used the same sensor configurations for all resolutions, which I verified by debugging.

Thank You!

7

What’s you command to select the sensor mode? Do you confirm xxx_set_mode in your sensor driver to initialize correct sensor mode?

8

Hi Shane,

I am not using any specific command to select the sensor mode directly. Instead, I am triggering the stream using the following downstream command:

gst-launch-1.0 v4l2src device="/dev/video0" ! video/x-raw,width=1920,height=1080,pixelformat=UYVY ! fpsdisplaysink

I have confirmed that the sensor_set_mode is correctly setting the sensor according to the driver configuration, as per the frame formats defined in the driver:

static const struct camera_common_frmfmt sensor_frmfmt = {
    {{640, 480}, &__60fps, 1, 0, 0},
    {{1280, 720}, &__60fps, 1, 0, 1},
    {{1920, 1080}, &__60fps, 1, 0, 2},
    {{2592, 1944}, &__60fps, 1, 0, 3},
    {{3840, 2160}, &__60fps, 1, 0, 4},
    {{4208, 3120}, &__60fps, 1, 0, 5},
};

This ensures that the sensor is initialized correctly in the respective modes.

However, as I mentioned before, sensor modes 0, 1, and 2 (corresponding to the resolutions 640x480, 1280x720, and 1920x1080) are not streaming, regardless of the sensor mode. When I change the index to values above 2, the stream works as expected.

My query is: Why are the first 3 resolutions not streaming, irrespective of the selected sensor mode?

Thank You!

9

There’s no this kind of problem for current camera framework.
You may confirm by v4l2-ctl and make sure the sensor driver set to correct sensor mode.

10

I’ve used the v4l2-ctl command to check the sensor configuration:

$ v4l2-ctl -d /dev/video0 --all

Here’s the relevant output:

sensor_modes 0x009a2082 (int)    : min=0 max=30 step=1 default=30 value=3 flags=read-only

Irrespective of the streaming mode, the sensor mode is always set to mode 3. Since this is marked as a read-only flag, I am unable to change it directly.

Could you help me understand how I can change the sensor mode if it’s read-only? Additionally, could this be the cause of the issue with the first three resolutions not streaming properly?

Also, the rest of the streaming works fine for modes above 2, so it seems that the other configurations are not being affected.

Looking forward to your insights!

11

Confirm by below command. Modify the width and height to select correct sensor mode.

v4l2-ctl --set-fmt-video=width=1920,height=1080 --set-ctrl bypass_mode=0 --stream-mmap -d /dev/video0
12

Thank you for the suggestion. I’ve tried using the command you provided:

v4l2-ctl --set-fmt-video=width=2592,height=1944 --set-ctrl bypass_mode=0 --stream-mmap -d /dev/video0

This command works perfectly with the 2592x1944 resolution which is index 3, but unfortunately, it doesn’t work with resolutions which are under index 3 (640x480, 1280x720, and 1920x1080).

Additionally, even after running the command, the sensor mode is still not changing, and it remains stuck at mode 3, as I mentioned earlier. The sensor mode is still set to read-only, so I’m unable to modify it directly.

Could you help clarify if this is causing the issue, and whether there is any way to change the sensor mode to the desired one for the first three resolutions?

Thank You!

13

Hi Shane,

Additionally, I’ve made some observations regarding the behavior of the sensor_modes:

  • After giving the v4l2 command, the sensor_modes change based on the sensor modes defined in the driver. If two sensor modes are set, the sensor_modes is set to 2; if only one is set, it’s set to 1.
  • When I only define two sensor modes in the DTS, index 0 and index 1 do not stream, but all modes starting from index 2 and above stream correctly.
  • If I remove all sensor modes, the driver fails to load, and the board gets stuck.
  • If I only define one sensor mode, index 0 does not stream, but all remaining indices work fine.

Now, my question is: If only two sensor modes are defined, how is it that the third index (which should not be part of the configuration) is still streaming?

Thank You!

14

Have a confirm by v4l2-ctl --list-formats-ext for the sensor mode that reported by your driver.

15

Hi Shane,

I used the command you mentioned, but I’m getting three formats as output. I only need one format. Could you also tell me where I can disable the remaining formats?"

v4l2-ctl -d /dev/video0 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
Type: Video Capture

[0]: 'UYVY' (UYVY 4:2:2)
	Size: Discrete 640x480
		Interval: Discrete 0.033s (30.000 fps)
	Size: Discrete 1280x720
		Interval: Discrete 0.033s (30.000 fps)
	Size: Discrete 1920x1536
		Interval: Discrete 0.033s (30.000 fps)
	Size: Discrete 1920x1080
		Interval: Discrete 0.033s (30.000 fps)
	Size: Discrete 2592x1944
		Interval: Discrete 0.033s (30.000 fps)
[1]: 'NV16' (Y/CbCr 4:2:2)
	Size: Discrete 640x480
		Interval: Discrete 0.033s (30.000 fps)
	Size: Discrete 1280x720
		Interval: Discrete 0.033s (30.000 fps)
	Size: Discrete 1920x1536
		Interval: Discrete 0.033s (30.000 fps)
	Size: Discrete 1920x1080
		Interval: Discrete 0.033s (30.000 fps)
	Size: Discrete 2592x1944
		Interval: Discrete 0.033s (30.000 fps)
[2]: 'UYVY' (UYVY 4:2:2)
	Size: Discrete 640x480
		Interval: Discrete 0.033s (30.000 fps)
	Size: Discrete 1280x720
		Interval: Discrete 0.033s (30.000 fps)
	Size: Discrete 1920x1536
		Interval: Discrete 0.033s (30.000 fps)
	Size: Discrete 1920x1080
		Interval: Discrete 0.033s (30.000 fps)
	Size: Discrete 2592x1944
		Interval: Discrete 0.033s (30.000 fps)
16

Maybe check the sensor mode if define multiple sensor mode with different pixel_phase.

17

Hi Shane,

Initially, three sensor modes were defined, all with the same UYVY pixel_phase, and at that time, I was getting three pixel_phase as mentioned earlier. Afterward, I removed two sensor modes and defined only one, but I am still getting the same three pixel_phase.

Where exactly are these pixel_phase being enumerated?

Thank You!

18

I would suspect could be the sensor driver cause the problem.
Could you use reference driver like nv_ov5693.c as template to modify to verify.

19

I have compared my driver file with the nv_ov5693.c file, and both follow the same structure.

My query is regarding the following section in the code:

static const struct camera_common_frmfmt sensor_frmfmt[] = {
    {{640, 480}, &__90fps, 1, 0, 0},
    {{1280, 720}, &__90fps, 1, 0, 1},
    {{1920, 1536}, &__30fps, 1, 0, 2},

    /* Add modes with no device tree support after this */
    
    {{1920, 1080}, &__30fps, 1, 0, 3},
    {{3840, 2160}, &__30fps, 1, 0, 4},
};

Currently, we are streaming with no device tree support. The first three sensor modes, which are defined in the device tree, are not streaming. However, after those entries, the sensor modes without device tree support are working, and I am able to stream successfully.

Now, I am focusing on enabling streaming with device tree support. Could you guide me on how to stream with device tree support properly?

Thank You!

20

I don’t think the problem relative with the sensor mode in device tree.
And remove all sensor modes have problem could be lost some necessary parameters like pixel_phase. You can reference to tc358840.c to report the necessary parameters instead instead from device tree.

21

Hi Shane,

I have checked the tc358840.c , but it did not help me resolve the issue.

Could you review the parameters I am using in the device tree source and check if there are any mistakes? If there are any incorrect configurations, please provide the correct ones to ensure proper streaming.

Here are the parameters I have set:

mclk_khz = "24000";  
phy_mode = "DPHY";  
discontinuous_clk = "yes";  
dpcm_enable = "false";  
cil_settletime = "0";  
dynamic_pixel_bit_depth = "16";  
csi_pixel_bit_depth = "16";  
mode_type = "yuv";  
pixel_phase = "uyvy";  
active_w = "1920";  
active_h = "1080";  
readout_orientation = "0";  
deskew_initial_enable = "false";  
pix_clk_hz = "186250000";  
gain_factor = "10";  
framerate_factor = "1000000";  
exposure_factor = "1000000";  
serdes_pix_clk_hz = "400000000";  
line_length = "1920";  
inherent_gain = "1";  
mclk_multiplier = "6.67";  
min_gain_val = "16";  
max_gain_val = "128";  
step_gain_val = "2";  
default_gain = "16";  
min_hdr_ratio = "1";  
max_hdr_ratio = "1";  
min_framerate = "1000000";  
max_framerate = "60000000";  
step_framerate = "1";  
default_framerate = "60000000";  
min_exp_time = "10"; /* 1ms */  
max_exp_time = "500000"; /* 500ms */  
step_exp_time = "1";  
default_exp_time = "2000";  
embedded_metadata_height = "0";

Thank You!