[R36.3] Can't capture images successfully with ISX031, Need some help

Hi, we are doing ISX031 bringup on our own board now, and can’t capture images successfully. The following is our information, let me know if you need more.

Release Info

$ cat /etc/nv_tegra_release
# R36 (release), REVISION: 3.0, GCID: 36191598, BOARD: generic, EABI: aarch64, DATE: Mon May  6 17:34:21 UTC 2024

Block Diagram
6 * GMSL_CAM:
ISX031 —> MAX96717F —> MAX96714 —> Orin

HW Connect

DTS
Currently only configured for Cam_4:
tegra234-camera-isx031-a00.dtsi.txt (9.0 KB)

Linux Driver
Just used to create v4l devices, ISX031, MAX96717 and MAX96714 are configured in userspace.

Config Script
2 Lanes, 1500Mbps
config_cam.bash.txt (3.6 KB)

v4l2 cmd
v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1300 --stream-mmap -c bypass_mode=0 --stream-count=1 --stream-to=isx031.raw

Trace
Boost clock:

echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrq_rate_locked
cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate
cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate
cat /sys/kernel/debug/bpmp/debug/clk/nvcsi/max_rate

Found CHANSEL_SHORT_FRAME and CHANSEL_NOMATCH in the trace log:

     kworker/7:2-228     [007] .......   224.177587: rtcpu_vinotify_event: tstamp:7640432731 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:244483677888 data:0x359d580010000000
     kworker/7:2-228     [007] .......   224.177589: rtcpu_vinotify_event: tstamp:7640432897 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:244483684288 data:0x0000000031000001
     kworker/7:2-228     [007] .......   224.177589: rtcpu_vinotify_event: tstamp:7640433052 cch:0 vi:0 tag:VIFALC_ACTIONLST channel:0x23 frame:0 vi_tstamp:244483687296 data:0x0000000007020001
     kworker/7:2-228     [007] .......   224.177590: rtcpu_vinotify_event: tstamp:7640433184 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:244483734848 data:0x359d550010000000
     kworker/7:2-228     [007] .......   224.177590: rtcpu_vinotify_event: tstamp:7640433337 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:244483741344 data:0x0000000031000002
     kworker/7:2-228     [007] .......   224.177591: rtcpu_vinotify_event: tstamp:7640433481 cch:0 vi:0 tag:FS channel:0x00 frame:0 vi_tstamp:244492456480 data:0x0000000000000014
     kworker/7:2-228     [007] .......   224.177591: rtcpu_vinotify_event: tstamp:7640433634 cch:0 vi:0 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:244492456512 data:0x0000000800000000
     kworker/7:2-228     [007] .......   224.177592: rtcpu_vinotify_event: tstamp:7640770761 cch:0 vi:0 tag:CHANSEL_PXL_SOF channel:0x23 frame:0 vi_tstamp:244494739968 data:0x0000000000000001
     kworker/7:2-228     [007] .......   224.177593: rtcpu_vinotify_event: tstamp:7640770916 cch:0 vi:0 tag:VIFALC_ACTIONLST channel:0x23 frame:0 vi_tstamp:244494741344 data:0x0000000008020001
     kworker/7:2-228     [007] .......   224.177593: rtcpu_vinotify_event: tstamp:7641237212 cch:0 vi:0 tag:CHANSEL_PXL_EOF channel:0x23 frame:0 vi_tstamp:244519493056 data:0x0000000005130002
     kworker/7:2-228     [007] .......   224.177594: rtcpu_vinotify_event: tstamp:7641237385 cch:0 vi:0 tag:ATOMP_FRAME_DONE channel:0x23 frame:0 vi_tstamp:244519493664 data:0x0000000000000000
     kworker/7:2-228     [007] .......   224.177594: rtcpu_vinotify_event: tstamp:7641237518 cch:0 vi:0 tag:VIFALC_ACTIONLST channel:0x23 frame:0 vi_tstamp:244519496608 data:0x0000000002020001
     kworker/7:2-228     [007] .......   224.177595: rtcpu_vinotify_event: tstamp:7641237669 cch:0 vi:0 tag:FE channel:0x00 frame:0 vi_tstamp:244519493696 data:0x0000000000000024
     kworker/7:2-228     [007] .......   224.177595: rtcpu_vinotify_event: tstamp:7641237804 cch:0 vi:0 tag:CHANSEL_SHORT_FRAME channel:0x10 frame:0 vi_tstamp:244519493696 data:0x0000200000000000
     kworker/7:2-228     [007] .......   224.177596: rtcpu_vinotify_event: tstamp:7641237956 cch:0 vi:0 tag:VIFALC_ACTIONLST channel:0x23 frame:0 vi_tstamp:244519506816 data:0x0000000001020001
     kworker/7:2-228     [007] .......   224.177596: rtcpu_vinotify_event: tstamp:7641238089 cch:0 vi:0 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:244519493728 data:0x0000000800000000
     kworker/7:2-228     [007] .......   224.177597: rtcpu_vinotify_event: tstamp:7641548186 cch:0 vi:0 tag:FS channel:0x00 frame:0 vi_tstamp:244525789856 data:0x0000000000000014
     kworker/7:2-228     [007] .......   224.177597: rtcpu_vinotify_event: tstamp:7641548335 cch:0 vi:0 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:244525789888 data:0x0000000800000000
     kworker/7:2-228     [007] .......   224.177598: rtcpu_vinotify_event: tstamp:7641548491 cch:0 vi:0 tag:CHANSEL_PXL_SOF channel:0x23 frame:0 vi_tstamp:244528073376 data:0x0000000000000001
     kworker/7:2-228     [007] .......   224.233543: rtcpu_vinotify_error: tstamp:7642544786 cch:0 vi:0 tag:CHANSEL_NOMATCH channel:0x10 frame:0 vi_tstamp:244561406912 data:0x00000000000003c9
     kworker/7:2-228     [007] .......   224.233544: rtcpu_vinotify_event: tstamp:7642562402 cch:0 vi:0 tag:CHANSEL_PXL_EOF channel:0x23 frame:0 vi_tstamp:244552826624 data:0x0000000005130002
     kworker/7:2-228     [007] .......   224.233544: rtcpu_vinotify_event: tstamp:7642562560 cch:0 vi:0 tag:ATOMP_FRAME_DONE channel:0x23 frame:0 vi_tstamp:244552827200 data:0x0000000000000000
     kworker/7:2-228     [007] .......   224.233545: rtcpu_vinotify_event: tstamp:7642562695 cch:0 vi:0 tag:FE channel:0x00 frame:0 vi_tstamp:244552827264 data:0x0000000000000024
     kworker/7:2-228     [007] .......   224.233545: rtcpu_vinotify_event: tstamp:7642562845 cch:0 vi:0 tag:CHANSEL_SHORT_FRAME channel:0x10 frame:0 vi_tstamp:244552827264 data:0x0000200000000000
     kworker/7:2-228     [007] .......   224.233546: rtcpu_vinotify_event: tstamp:7642562991 cch:0 vi:0 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:244552827296 data:0x0000000800000000
     kworker/7:2-228     [007] .......   224.233546: rtcpu_vinotify_event: tstamp:7642563142 cch:0 vi:0 tag:FS channel:0x00 frame:0 vi_tstamp:244559123424 data:0x0000000000000014
     kworker/7:2-228     [007] .......   224.233547: rtcpu_vinotify_event: tstamp:7642563276 cch:0 vi:0 tag:CHANSEL_NOMATCH channel:0x10 frame:0 vi_tstamp:244561406912 data:0x00000000000003c9
     kworker/7:2-228     [007] .......   224.233547: rtcpu_vinotify_event: tstamp:7643576393 cch:0 vi:0 tag:FE channel:0x00 frame:0 vi_tstamp:244586160640 data:0x0000000000000024
     kworker/7:2-228     [007] .......   224.233548: rtcpu_vinotify_event: tstamp:7643576528 cch:0 vi:0 tag:FS channel:0x00 frame:0 vi_tstamp:244592456768 data:0x0000000000000014

trace.txt (312.0 KB)

Sensor Output
Active: 1920 x 1300
Line width: 2200
Format: YUV422 8-bit

Hi,

For the camera basic functionality first needs to check the device and driver configuration.
You can reference to below program guide for the detailed information of device tree and driver implementation.
https://docs.nvidia.com/jetson/archives/r36.3/DeveloperGuide/SD/CameraDevelopment/SensorSoftwareDriverProgramming.html?highlight=programing#sensor-software-driver-programming

Please refer to Applications Using V4L2 IOCTL Directly by using V4L2 IOCTL to verify basic camera functionality.
https://docs.nvidia.com/jetson/archives/r36.3/DeveloperGuide/SD/CameraDevelopment/SensorSoftwareDriverProgramming.html?highlight=programing#to-run-a-v4l2-ctl-test

Once confirm the configure and still failed below link help to get log and some information and some tips for debug.
https://elinux.org/Jetson/l4t/Camera_BringUp#Steps_to_enable_more_debug_messages

Thanks!

Hi,

These documents you mentioned have been read, please help to analyze the problem, thanks!

Hi,

Using GStreamer preview to preview on the screen, I can see the camera image, but there are still some problems:

  • There’s a big delay.
  • The frame rate is much less than 30.
  • The green screen can be seen at intervals.

GStreamer cmd:

gst-launch-1.0 v4l2src device=/dev/video0 ! 'video/x-raw, format=(string)UYVY, width=(int)1920, height=(int)1300, framerate=(fraction)30/1' ! xvimagesink -ev

Any suggestions?

hello xxeevm,

could you please share the tracing logs upon your latest test results.
for instance, do you still seeing those CHANSEL_SHORT_FRAME and CHANSEL_NOMATCH failures while running camera stream?

besides, let’s give it a try to disable preview rendering, and shows frame-rate only for quick checking.
i.e. $ gst-launch-1.0 v4l2src device=/dev/video1 num-buffers=300 ! 'video/x-raw, width=1920, height=1080, framerate=30/1, format=YUY2' ! fpsdisplaysink text-overlay=0 name=sink_0 video-sink=fakesink sync=0 -v

Hi JerryChang,

Latest test results tracing logs is the file “trace_1202.txt”, and still seeingCHANSEL_SHORT_FRAME and CHANSEL_NOMATCH failures.

Disable preview rendering and shows frame-rate only cmd:
gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=300 ! 'video/x-raw, format=(string)UYVY, width=(int)1920, height=(int)1300, framerate=(fraction)30/1' ! fpsdisplaysink text-overlay=0 name=sink_0 video-sink=fakesink sync=0 -v
Result:
trace_1202.txt (398.4 KB)

Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0/GstFakeSink:fakesink0: sync = false
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-raw, format=(string)UYVY, width=(int)1920, height=(int)1300, framerate=(fraction)30/1, interlace-mode=(string)progressive, colorimetry=(string)2:4:7:1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw, format=(string)UYVY, width=(int)1920, height=(int)1300, framerate=(fraction)30/1, interlace-mode=(string)progressive, colorimetry=(string)2:4:7:1
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = video/x-raw, format=(string)UYVY, width=(int)1920, height=(int)1300, framerate=(fraction)30/1, interlace-mode=(string)progressive, colorimetry=(string)2:4:7:1
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0/GstFakeSink:fakesink0.GstPad:sink: caps = video/x-raw, format=(string)UYVY, width=(int)1920, height=(int)1300, framerate=(fraction)30/1, interlace-mode=(string)progressive, colorimetry=(string)2:4:7:1
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0.GstGhostPad:sink: caps = video/x-raw, format=(string)UYVY, width=(int)1920, height=(int)1300, framerate=(fraction)30/1, interlace-mode=(string)progressive, colorimetry=(string)2:4:7:1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw, format=(string)UYVY, width=(int)1920, height=(int)1300, framerate=(fraction)30/1, interlace-mode=(string)progressive, colorimetry=(string)2:4:7:1
Redistribute latency...
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0/GstFakeSink:fakesink0: sync = false
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 2, dropped: 0, current: 0.77, average: 0.77
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 7, dropped: 0, current: 1.95, average: 1.36
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 12, dropped: 0, current: 1.95, average: 1.56
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 17, dropped: 0, current: 1.95, average: 1.65
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 22, dropped: 0, current: 1.78, average: 1.68
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 27, dropped: 0, current: 1.96, average: 1.73
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 32, dropped: 0, current: 1.95, average: 1.76
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 37, dropped: 0, current: 1.95, average: 1.78
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 42, dropped: 0, current: 1.95, average: 1.80
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 47, dropped: 0, current: 1.95, average: 1.82
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 52, dropped: 0, current: 1.95, average: 1.83
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 57, dropped: 0, current: 1.96, average: 1.84
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 62, dropped: 0, current: 1.95, average: 1.85
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 67, dropped: 0, current: 1.95, average: 1.85

hello xxeevm,

please try using nvvidconv to convert the video formats as I420,
for instance, $ gst-launch-1.0 -v nvv4l2camerasrc device=/dev/video0 ! 'video/x-raw(memory:NVMM), width=1920, height=1300, format=UYVY' ! videorate ! nvvidconv ! 'video/x-raw(memory:NVMM), format=I420' ! fpsdisplaysink text-overlay=0 video-sink=fakesink sync=0

Hi JerryChang

Using nvvidconv to convert the video formats as I420, result is following:

Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0: sync = false
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstNvV4l2CameraSrc:nvv4l2camerasrc0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1300, format=(string)UYVY, interlace-mode=(string)progressive, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1300, format=(string)UYVY, interlace-mode=(string)progressive, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstVideoRate:videorate0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1300, format=(string)UYVY, interlace-mode=(string)progressive, framerate=(fraction)30/1
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1300, interlace-mode=(string)progressive, framerate=(fraction)30/1, format=(string)I420
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1300, interlace-mode=(string)progressive, framerate=(fraction)30/1, format=(string)I420
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1300, interlace-mode=(string)progressive, framerate=(fraction)30/1, format=(string)I420
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1300, interlace-mode=(string)progressive, framerate=(fraction)30/1, format=(string)I420
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1300, interlace-mode=(string)progressive, framerate=(fraction)30/1, format=(string)I420
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1300, interlace-mode=(string)progressive, framerate=(fraction)30/1, format=(string)I420
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1300, format=(string)UYVY, interlace-mode=(string)progressive, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstVideoRate:videorate0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1300, format=(string)UYVY, interlace-mode=(string)progressive, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1300, format=(string)UYVY, interlace-mode=(string)progressive, framerate=(fraction)30/1
Redistribute latency...
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0: sync = false
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 93, dropped: 0, current: 33.92, average: 33.92
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 175, dropped: 0, current: 29.11, average: 31.48
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 260, dropped: 0, current: 30.18, average: 31.05
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 344, dropped: 0, current: 29.84, average: 30.74
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 429, dropped: 0, current: 30.19, average: 30.63
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 513, dropped: 0, current: 29.81, average: 30.49
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 598, dropped: 0, current: 30.18, average: 30.45

hello xxeevm,

it looks promising, the reported frame-rate is around 30-fps now.
you may adding nvvidconv to convert the video formats as I420 into your original pipeline for verification.

Hi JerryChang

Adding nvvidconv to convert the video formats as I420, the problem is same:

cmd is following:

gst-launch-1.0 nvv4l2camerasrc device=/dev/video0 ! \
    'video/x-raw(memory:NVMM), format=(string)UYVY, width=(int)1920, height=(int)1300, interlace-mode=progressive, framerate=(fraction)30/1' ! \
    nvvidconv ! \
    'video/x-raw(memory:NVMM), format=I420' ! \
    nv3dsink -ev

hello xxeevm,

please try other sink elements, such as xvimagesink.
for instance,
$ gst-launch-1.0 nvv4l2camerasrc device=/dev/video0 ! 'video/x-raw(memory:NVMM), format=(string)UYVY, width=(int)1920, height=(int)1300, interlace-mode=progressive, framerate=(fraction)30/1' ! nvvidconv ! 'video/x-raw(memory:NVMM), format=I420' ! queue ! xvimagesink -e

Hi JerryChang

This cmd only got a warning message and exit.

WARNING: erroneous pipeline: could not link queue0 to xvimagesink0

hello xxeevm,

please remove queue element.
$ gst-launch-1.0 nvv4l2camerasrc device=/dev/video0 ! 'video/x-raw(memory:NVMM), format=(string)UYVY, width=(int)1920, height=(int)1300, framerate=(fraction)30/1' ! nvvidconv ! 'video/x-raw(memory:NVMM), format=I420' ! xvimagesink -e

besides, you may also test with v4l2src plugin for confirmation.

Hi JerryChang,

Problem is solved, the root cause is camera output without embedded data. After modify in dts embedded_metadata_height = "0", v4l2-ctl and gstreamer work fine.

Thanks very much for your help~

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.