Orin NX v4l2/gstreamer starting stream timeout error

[jafeth.garcia]

Hi, I am working with an Orin NX JP 5.1.1 on a custom carrier board. I have been running streaming stability tests for 2 MIPI sensors. I am able to capture frames from both sensors. I have been doing start/stop stream cycles on the sensors and I have seen the stream has errors when starting a stream.

This is the command I used for captures in v4l2 with a 1 second delay between each capture cycle:

v4l2-ctl -d /dev/video0 --set-fmt-video=width=3840,height=2160,pixelformat=BG12 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=150

I will see 30fps output from the sensor for a while but after ~20min v4l2 will not show a framerate output and I get these error messages in the kernel log:

[ 2007.143818] ox08b40 2-0030: ox08b40_power_on: power on
[ 2007.172784] ox08b40 2-0030: ox08b40_software_reset: Sending first software reset command
[ 2007.173107] ox08b40 2-0030: ox08b40_software_reset: Sending second software reset command
[ 2007.191366] bwmgr API not supported
[ 2007.199513] ox08b40 2-0030: ox08b40_set_mode: selecting mode 0
[ 2007.271310] ox08b40 2-0030: ox08b40_set_gain: gain requested: hcg {12, 0}, lcg {3, 0}, spd {1, 0}
[ 2007.275503] ox08b40 2-0030: ox08b40_set_exposure: requested exposure time: 4945 us
[ 2007.275511] ox08b40 2-0030: ox08b40_set_exposure: exposure time (double rows): 168
[ 2007.279389] ox08b40 2-0030: ox08b40_set_frame_rate: refuse frame rate change
[ 2007.279396] ox08b40 2-0030: ox08b40_start_streaming
[ 2007.279402] ox08b40 2-0030: ox08b40_start_streaming: setting frame sync follower
[ 2009.798336] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[ 2009.807479] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[ 2009.818481] (NULL device *): vi_capture_control_message: NULL VI channel received
[ 2009.826206] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_close: Error in closing stream_id=0, csi_port=0
[ 2009.836869] (NULL device *): vi_capture_control_message: NULL VI channel received
[ 2009.844588] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_open: VI channel not found for stream- 0 vc- 0
[ 2009.855296] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel

I have also done tests with gstreamer using a similar capture cycle with 1s delays. The gstreamer pipeline is:

gst-launch-1.0 -v nvarguscamerasrc sensor-id=0 num-buffers=150 ! "video/x-raw(memory:NVMM),format=NV12,width=3840,height=2160" ! nvvidconv ! fakesink

After ~30min these cycles also trigger an error:

Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)3840, height=(int)2160, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)3840, height=(int)2160, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)3840, height=(int)2160, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstFakeSink:fakesink0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)3840, height=(int)2160, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)3840, height=(int)2160, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)3840, height=(int)2160, format=(string)NV12, framerate=(fraction)30/1
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3840 x 2160 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 1.375000; Exposure Range min 59000, max 32003000;

GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 0 
   Output Stream W = 3840 H = 2160 
   seconds to Run    = 0 
   Frame Rate = 29.999999 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
nvbuf_utils: dmabuf_fd -1 mapped entry NOT found
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, threadExecute:694 NvBufSurfaceFromFd Failed.
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, threadFunction:247 (propagating)
Got EOS from element "pipeline0".

The kernel log does not show an error but seems to be hanging after starting a stream:

[  489.946628] ox08b40 2-0030: ox08b40_power_on: power on
[  489.974948] ox08b40 2-0030: ox08b40_software_reset: Sending first software reset command
[  489.975090] ox08b40 2-0030: ox08b40_software_reset: Sending second software reset command
[  489.991278] bwmgr API not supported
[  489.999219] ox08b40 2-0030: ox08b40_set_mode: selecting mode 0
[  490.073833] ox08b40 2-0030: ox08b40_set_gain: gain requested: hcg {12, 0}, lcg {3, 0}, spd {1, 0}
[  490.077361] ox08b40 2-0030: ox08b40_set_exposure: requested exposure time: 32003 us
[  490.077367] ox08b40 2-0030: ox08b40_set_exposure: exposure time (double rows): 1092
[  490.082563] ox08b40 2-0030: ox08b40_set_frame_rate: refuse frame rate change
[  490.082570] ox08b40 2-0030: ox08b40_start_streaming
[  490.082576] ox08b40 2-0030: ox08b40_start_streaming: setting frame sync follower

What could cause these issues? Is there a fix to improve the stability of the capture?

[ShaneCCC]

I would suspect it could be the sensor power sequency or the sensor didn’t reset well cause the problem.
Please probe the signal to confirm the power sequency first.

[JerryChang]

hello jafeth.garcia,

BTW,
please also include -e options to the gst pipeline, it will send an EoS at shutdown.
i.e. $ gst-launch-1.0 nvarguscamerasrc sensor-id=0 num-buffers=150 ... -e

[jafeth.garcia]

Hi, I checked the sensor power sequence and added missing wait time on the sensor start stream sequence. This fixed the issue with v4l2 capture.

The gstreamer pipeline capture is having issues with the mentioned error:

nvbuf_utils: dmabuf_fd -1 mapped entry NOT found
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, threadExecute:694 NvBufSurfaceFromFd Failed.
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, threadFunction:247 (propagating)

[ShaneCCC]

Please check if infinite timeout help on your case.
If yes try to add set_mode_delay_ms in device tree to try.

sudo service nvargus-daemon stop
sudo enableCamInfiniteTimeout=1 nvargus-daemon

[jafeth.garcia]

Hi, I tried enabling infiniteTimeout but the error still appeared after around ~15min. Does the Orin NX have a way to enable the VI trace in order to get further information on what is happening?

[ShaneCCC]

Please adjust the pix_clk_hz/serdes_pix_clk_hz > 1.5Gbps to confirm.

Skew calibration is required if sensor or deserializer is using DPHY, and the output data rate is > 1.5Gbps.
An initiation deskew signal should be sent by sensor or deserializer to perform the skew calibration. If the deskew signals is not sent, the receiver will stall, and the capture will time out.
You can calculate the output data rate with the following equation:

Output data rate = (sensor or deserializer pixel clock in hertz) * (bits per pixel) / (number of CSI lanes)

[jafeth.garcia]

How does the serdes_pix_clk_hz property get calculated? Does it follow the same formula you gave?
Also is there more documentation on what this serdes clock does/is?

The sensor I am using also is using HDR compression so there is dynamic_pixel_depth=16 and csi_pixel_depth=12, which of these two should be used for the data rate calculation?

[ShaneCCC]

Both of serdes_pix_clk_hz and pix_clk_hz are the same calculation.
For HDR suppose use dynamic_pixel_depth for bits per pixel in the calculation.

[jafeth.garcia]

Hi, I have tested adjusting the clocks and this did seem to help with the stability in the capture. Original calculation for pix_clk_hz was:

2300 Mbps * 2 lanes / 12 bits per pixel = 383333333 Hz = ~383 MHz

This was changed to:

2300 Mbps * 2 lanes / 16 bits per pixel = 287500000 Hz = ~287.5 MHz

However I am seeing that capture can result in a distorted frame such as below (distorted vs normal frame). Does this mean the pix_clk_hz is not correct yet?

[ShaneCCC]

Does the sensor driver have any calculation depend on pix_clk_hz?
For Orin the pix_clk_hz/serdes_pix_clk_hz only for acquire the NVCSI/VI/ISP bandwidth.
However you can set them to max to ignore it for Orin side.

sudo su
echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/emc/mrq_rate_locked
cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate |tee /sys/kernel/debug/bpmp/debug/clk/vi/rate
cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate | tee  /sys/kernel/debug/bpmp/debug/clk/isp/rate
cat /sys/kernel/debug/bpmp/debug/clk/nvcsi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/nvcsi/rate
cat /sys/kernel/debug/bpmp/debug/clk/emc/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/emc/rate