Xavier NX camera RAW16

Autonomous Machines Jetson & Embedded Systems Jetson Xavier NX
1

I’m attempting to interface a camera with the Jetson Xavier NX via CSI. I have the camera producing images in raw8 (GREY) mode. however I cannot get the CSI interface to work correctly with raw16.

I have made what I believe to be the correct alterations to the drivers and device tree in order to get both ‘GREY’ and 'Y16 ’ encodings to work (as described in this post), however only raw8 is working.

This command produces an Image I can use:

v4l2-ctl --set-fmt-video=width=1920,height=1080,pixelformat=GREY  --stream-mmap --set-ctrl=bypass_mode=0 --stream-count=1 -d /dev/video0 --stream-to=test_cam_data --verbose

And the following does not:

v4l2-ctl --set-fmt-video=width=1920,height=1080,pixelformat=Y16  --stream-mmap --set-ctrl=bypass_mode=0 --stream-count=1 -d /dev/video0 --stream-to=test_cam_data --verbose

Using the second command gives the following errors:

From Trace:

kworker/0:5-5703  [000] ....  5669.259421: rtcpu_vinotify_error: tstamp:177529759064 tag:CHANSEL_NOMATCH channel:0x04 frame:0 vi_tstamp:177529756175 data:0x00000249
kworker/0:5-5703  [000] ....  5669.259422: rtcpu_vinotify_event: tstamp:177530195091 tag:FS channel:0x00 frame:0 vi_tstamp:177529756152 data:0x00000012
kworker/0:5-5703  [000] ....  5669.259423: rtcpu_vinotify_event: tstamp:177530195230 tag:CHANSEL_NOMATCH channel:0x04 frame:0 vi_tstamp:177529756175 data:0x00000249
kworker/0:5-5703  [000] ....  5669.315416: rtcpu_vinotify_event: tstamp:177530737908 tag:FE channel:0x00 frame:0 vi_tstamp:177530246540 data:0x00000022
kworker/0:5-5703  [000] ....  5669.315418: rtcpu_vinotify_error: tstamp:177530930937 tag:CHANSEL_NOMATCH channel:0x04 frame:0 vi_tstamp:177530928094 data:0x00000249
kworker/0:5-5703  [000] ....  5669.315419: rtcpu_vinotify_event: tstamp:177531280784 tag:FS channel:0x00 frame:0 vi_tstamp:177530928072 data:0x00000012
kworker/0:5-5703  [000] ....  5669.315420: rtcpu_vinotify_event: tstamp:177531280942 tag:CHANSEL_NOMATCH channel:0x04 frame:0 vi_tstamp:177530928094 data:0x00000249
kworker/0:5-5703  [000] ....  5669.315421: rtcpu_vinotify_event: tstamp:177531822794 tag:FE channel:0x00 frame:0 vi_tstamp:177531418461 data:0x00000022

From dmesg:

[   63.390233] tegra194-vi5 15c10000.vi: corr_err: discarding frame 0, flags: 128, err_data 131072
[   66.093659] tegra194-vi5 15c10000.vi: no reply from camera processor
[   66.093821] tegra194-vi5 15c10000.vi: uncorr_err: request timed out after 2500 ms
[   66.093968] tegra194-vi5 15c10000.vi: err_rec: attempting to reset the capture channel
[   66.098344] tegra194-vi5 15c10000.vi: err_rec: successfully reset the capture channel

I have tried the following:

  • Setting the embedded_metadata_height to 0, 1, or 2.
    - this resulted in different behaviour, but didn’t correct the issue
    - currently this value is set to 2.
  • reducing active_w and active_h

Any assistance would be great. How do I interpret the errors ‘CHANSEL_NOMATCH’, and ‘FS’? are there any hints on where I should be looking in the TRM?

3

Have a reference to below

CHANSEL_NOMATCH
bit 0        no_match
bit 1-4    CTYPE
bit 5-10  DTYPE

CTYPE:
    FS    = 0x8, FE   = 0x1 
    LS    = 0x4, LE   = 0x2  
    LS_LE = 0x6, DATA = 0x9

DTYPE:
      NvCsiDataType_Unspecified             = 0,
      NvCsiDataType_YUV420_10               = 25,
      NvCsiDataType_LEG_YUV420_8            = 26,
      NvCsiDataType_YUV420_8                = 24,
      NvCsiDataType_YUV420CSPS_8            = 28,
      NvCsiDataType_YUV420CSPS_10           = 29,
      NvCsiDataType_YUV422_8                = 30,
      NvCsiDataType_YUV422_10               = 31,
      NvCsiDataType_RGB444                  = 32,
      NvCsiDataType_RGB555                  = 33,
      NvCsiDataType_RGB565                  = 34,
      NvCsiDataType_RGB666                  = 35,
      NvCsiDataType_RGB888                  = 36,
      NvCsiDataType_RAW6                    = 40,
      NvCsiDataType_RAW7                    = 41,
      NvCsiDataType_RAW8                    = 42,
      NvCsiDataType_RAW10                   = 43,
      NvCsiDataType_RAW12                   = 44,
      NvCsiDataType_RAW14                   = 45,
      NvCsiDataType_RAW16                   = 46,
      NvCsiDataType_RAW20                   = 47,
      NvCsiDataType_User_1                  = 48,
      NvCsiDataType_User_2                  = 49,
      NvCsiDataType_User_3                  = 50,
      NvCsiDataType_User_4                  = 51,
      NvCsiDataType_User_5                  = 52,
      NvCsiDataType_User_6                  = 53,
      NvCsiDataType_User_7                  = 54,
      NvCsiDataType_User_8                  = 55,
      NvCsiDataType_Unknown                 = 64,
4

Thanks for getting back to me so quickly Shane!

I’m a bit unsure how to work with that information. the CHANSEL_NOMATCH data I got was 0x00000249 in binary that’s
1001001001.

bit 0 is 1, which I assume means no_match? (to a data type?)

bits 1-4 are 0100 (0x04) which I assume means LS? (what does LS mean?)

the last bits (5-10) are 10010 (0x12). This doesn’t match the table, which makes me assume I’ve made a mistake somewhere.

Sorry for not understanding, how am I meant to interpret this table? Also is this same table used to interpret the FS errors on the subsequent lines?

5

The LS means line start

6

Thanks for clarifying that Shane. From above, am I interpreting the value from the trace correctly?

7

Based on this post I think that I am interpreting the table ShaneCCC provided correctly.

For code 0x00000249 (1001001001) I get the following:

NO_MATCH: True (1)
CTYPE:  0x4 (0100)
DTYPE:  18  (10010)???

I am unsure what this implies, as from the table ShaneCCC provided, there is no match for a DTYPE of 18. have I made a mistake in my interpretation?

8

hello benm12y6,

may I know what’s your actual sensor output formats? according to your descriptions, I don’t think it’s RAW16.
please refer to Xavier TRM, check Table-7.9 for supported data types.
could you please also share more details of your sensor capability.
thanks

9

hello JerryChang,

I’m fairly confident the sensor is outputting in a raw16 format. I have configured it based on its user manual. The raw8 output is working when configured that way. What makes you suspect I haven’t configured it for raw16 in this case?

10

hello benm12y6,

image date type (DTYPE) should return 46 if you’d configure it as Raw16.

DTYPE
...
        NvCsiDataType_RAW16                   = 46,

may I know what’s the color pattern orders (i.e. RGGB), also, what’s the bits per pixels in your configuration?
thanks

11

Where does the value in the trace come from? is it saying the format of the data it is receiving from the device, or the format it is configured to receive?

The camera is a mono sensor, and I have set bits per pixel to 16 in the device tree.

12

hello benm12y6,

it’s due to monochrome sensor is not a RAW16 formats. you may also refer to Bayer filter from Wikipedia.

13

Sorry for the confusion JerryChang, which formats should I be using with a mono sensor with a bits per pixel of 16?

Thanks,
Ben

14

hello benm12y6,

please check Xavier TRM, you’ll need to set pixel formats as T_R16 in the VI drivers.
you may also have modifications in VI drivers to make it supported.
for example, $L4T_Sources/r32.4.3/Linux_for_Tegra/source/public/kernel/nvidia/drivers/media/platform/tegra/camera/vi/*

BTW,
there’s Multimedia API Sample Applications, 12_camera_v4l2_cuda.
you may allocate CUDA buffers and do frame capture through v4l2, using CUDA engine to have format converting.
however, if you would like to leverage other hardware engines such encoder or VIC, you would also need to convert the formats to YUV420 8-bit or 10-bit.
thanks

15

Thanks for getting back to me, I have made these changes to vi5_formats.h

	/* GRAY */
TEGRA_VIDEO_FORMAT(RAW8, 8, Y8_1X8, 1, 1, T_R8,
		RAW8, GREY, "GRAY8"),
TEGRA_VIDEO_FORMAT(RAW10, 10, Y10_1X10, 2, 1, T_R16_I,
		RAW10, Y10, "GRAY10"),
TEGRA_VIDEO_FORMAT(RAW10, 16, Y16_1X16, 2, 1, T_R16_I,
		RAW10, Y16, "GRAY16"),

Is this what you were referring to ?

16

Just following up on this further. I have changed the camera to output other data formats, while maintaining the same device tree and driver for RAW16, and I still get the same CHANSEL 0x249 error, this would indicate to me that I have configured the Jetson incorrectly for receiving RAW16 data, as I see this error regardless which data format is provided.

17

The CHANSEL 0x249 means NVCSI/VI receive a 0x12 data type package. That’s mens you need to check the sensor output package.

18

Okay so I am a bit more confused now. If look at the trace while getting RAW8 data from the camera I get also get CHANSEL errors, however I am still getting frames from the camera (images which are slightly malformed).

Here is the trace for that:

 kworker/0:4-1654  [000] ....   340.837137: rtcpu_vinotify_event: tstamp:11019259503 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:11018760588 data:0x00000000
 kworker/0:4-1654  [000] ....   340.837139: rtcpu_vinotify_event: tstamp:11019259649 tag:RESERVED_19 channel:0x23 frame:0 vi_tstamp:9002975648 data:0x000203e8
 kworker/0:4-1654  [000] ....   340.837139: rtcpu_vinotify_event: tstamp:11019259812 tag:RESERVED_19 channel:0x23 frame:0 vi_tstamp:9002997184 data:0x070203e9
 kworker/0:4-1654  [000] ....   340.837140: rtcpu_vinotify_error: tstamp:11019450763 tag:CHANSEL_NOMATCH channel:0x04 frame:0 vi_tstamp:11019445325 data:0x00000269
 kworker/0:4-1654  [000] ....   340.837146: rtos_queue_peek_from_isr_failed: tstamp:11019452373 queue:0x0bcbcf78
 kworker/0:4-1654  [000] ....   340.837147: rtcpu_vinotify_event: tstamp:11019695252 tag:FS channel:0x00 frame:0 vi_tstamp:11019442115 data:0x00000012
 kworker/0:4-1654  [000] ....   340.837147: rtcpu_vinotify_event: tstamp:11019695414 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:11019442117 data:0x00000000
 kworker/0:4-1654  [000] ....   340.837147: rtcpu_vinotify_event: tstamp:11019695554 tag:CHANSEL_EMBED_SOF channel:0x23 frame:0 vi_tstamp:11019442138 data:0x00000004
 kworker/0:4-1654  [000] ....   340.837148: rtcpu_vinotify_event: tstamp:11019695715 tag:CHANSEL_EMBED_EOF channel:0x23 frame:0 vi_tstamp:11019442248 data:0x00010008
 kworker/0:4-1654  [000] ....   340.837148: rtcpu_vinotify_event: tstamp:11019695851 tag:ATOMP_EMB_DATA_DONE channel:0x23 frame:0 vi_tstamp:11019442270 data:0x00000000
 kworker/0:4-1654  [000] ....   340.837148: rtcpu_vinotify_event: tstamp:11019696009 tag:RESERVED_19 channel:0x23 frame:0 vi_tstamp:9024787808 data:0x030203e9
 kworker/0:4-1654  [000] ....   340.837148: rtcpu_vinotify_event: tstamp:11019696148 tag:CHANSEL_NOMATCH channel:0x04 frame:0 vi_tstamp:11019445325 data:0x00000269
 kworker/0:4-1654  [000] ....   340.837149: rtcpu_vinotify_event: tstamp:11019696305 tag:CHANSEL_PXL_SOF channel:0x23 frame:0 vi_tstamp:11019447629 data:0x00000001

When outputting in RAW16, I still get the CHANSEL errors (with code 249 instead, I am also seeing the following in dmesg:

[  274.312147] tegra194-vi5 15c10000.vi: corr_err: discarding frame 0, flags: 128, err_data 131072
[  276.818836] tegra194-vi5 15c10000.vi: no reply from camera processor
[  276.819033] tegra194-vi5 15c10000.vi: uncorr_err: request timed out after 2500 ms
[  276.819198] tegra194-vi5 15c10000.vi: err_rec: attempting to reset the capture channel
[  276.821005] t194-nvcsi 15a00000.nvcsi: csi5_stop_streaming: csi_pt=2, st_id=2, vc_id=0, pg_mode=0x0
[  276.821014] t194-nvcsi 15a00000.nvcsi: csi5_stream_close: stream_id=2, csi_port=2
[  276.821024] t194-nvcsi 15a00000.nvcsi: csi5_start_streaming: csi_pt=2, st_id=2, vc_id=0, pg_mode=0x0
[  276.821032] t194-nvcsi 15a00000.nvcsi: csi5_stream_set_config: stream_id=2, csi_port=2
[  276.821039] t194-nvcsi 15a00000.nvcsi: csi5_stream_open: stream_id=2, csi_port=2
[  276.821383] tegra194-vi5 15c10000.vi: err_rec: successfully reset the capture channel
[  276.838861] [RCE] Configuring VI GoS.
[  276.838871] [RCE] VM GOS[#0] addr=0xc2100000
[  276.838878] [RCE] VM GOS[#1] addr=0xc2101000
[  276.838884] [RCE] VM GOS[#2] addr=0xc2102000
[  276.838891] [RCE] VM GOS[#3] addr=0xc2103000
[  276.838896] [RCE] VM GOS[#4] addr=0xc2104000
[  276.838930] [RCE] VM GOS[#5] addr=0xc2105000
[  276.862219] tegra194-vi5 15c10000.vi: corr_err: discarding frame 0, flags: 128, err_data 131072
[  279.378722] tegra194-vi5 15c10000.vi: no reply from camera processor
[  279.378890] tegra194-vi5 15c10000.vi: uncorr_err: request timed out after 2500 ms
[  279.379069] tegra194-vi5 15c10000.vi: err_rec: attempting to reset the capture channel
[  279.381247] t194-nvcsi 15a00000.nvcsi: csi5_stop_streaming: csi_pt=2, st_id=2, vc_id=0, pg_mode=0x0
[  279.381256] t194-nvcsi 15a00000.nvcsi: csi5_stream_close: stream_id=2, csi_port=2
[  279.381266] t194-nvcsi 15a00000.nvcsi: csi5_start_streaming: csi_pt=2, st_id=2, vc_id=0, pg_mode=0x0
[  279.381273] t194-nvcsi 15a00000.nvcsi: csi5_stream_set_config: stream_id=2, csi_port=2
[  279.381280] t194-nvcsi 15a00000.nvcsi: csi5_stream_open: stream_id=2, csi_port=2
[  279.381636] tegra194-vi5 15c10000.vi: err_rec: successfully reset the capture channel
[  279.412313] tegra194-vi5 15c10000.vi: corr_err: discarding frame 0, flags: 128, err_data 131072
[  279.414812] [RCE] Configuring VI GoS.
[  279.414856] [RCE] VM GOS[#0] addr=0xc2100000
[  279.414863] [RCE] VM GOS[#1] addr=0xc2101000
[  279.414880] [RCE] VM GOS[#2] addr=0xc2102000
[  279.414886] [RCE] VM GOS[#3] addr=0xc2103000
[  279.414892] [RCE] VM GOS[#4] addr=0xc2104000
[  279.414905] [RCE] VM GOS[#5] addr=0xc2105000

I don’t think the camera is outputting data in an incorrect format, as I can receive frames using a Jetson Nano with the following debug settings:

echo 1 > /sys/kernel/debug/tracing/tracing_on
echo 30720 > /sys/kernel/debug/tracing/buffer_size_kb
echo 1 > /sys/kernel/debug/tracing/events/tegra_rtcpu/enable
echo 1 > /sys/kernel/debug/tracing/events/freertos/enable
echo 2 > /sys/kernel/debug/camrtc/log-level
echo 1 > /sys/kernel/debug/tracing/events/camera_common/enable
echo > /sys/kernel/debug/tracing/trace
cd /sys/kernel/debug/dynamic_debug/
echo file csi2_fops.c +p > control
echo file vi2_fops.c +p > control

and my resulting trace doesn’t show any of those errors:

 vi-output, vg57-23093 [000] .... 20677.954570: csi_s_stream: enable : 0x1
 vi-output, vg57-23093 [000] .... 20677.954598: tegra_channel_set_stream: vg5761 6-0010 : 0x1
 vi-output, vg57-23093 [002] .... 20680.924387: tegra_channel_capture_frame: sof:20681.254560510
 vi-output, vg57-23093 [001] .... 20680.941126: tegra_channel_capture_frame: sof:20681.271199378
 vi-output, vg57-23093 [000] .... 20680.957642: tegra_channel_capture_frame: sof:20681.287857621
 vi-output, vg57-23093 [002] .... 20680.974345: tegra_channel_capture_frame: sof:20681.304519302
 v4l2-ctl-23092 [003] .... 20680.990538: tegra_channel_capture_done: mw_ack_done:20681.320422320
 v4l2-ctl-23092 [003] .... 20680.990565: tegra_channel_set_stream: enable : 0x0
 v4l2-ctl-23092 [003] .... 20680.990571: tegra_channel_set_stream: vg5761 6-0010 : 0x0
 v4l2-ctl-23092 [000] .... 20681.112934: tegra_channel_set_stream: nvcsi--1 : 0x0
 v4l2-ctl-23092 [000] .... 20681.112938: csi_s_stream: enable : 0x0

At this point I am assuming that my configuration of the NX must be wrong, but I am still unsure how.

19

I have found a work around, which is to set the camera to output raw16, and then setup the device tree, and driver for raw8 but with double the frame width. this allows me to capture the full resolution frames from the camera, but isn’t an ideal solution. I still have some issues with the NX which I don’t see on the Nano, but I will make a separate post to address this.