RAW capture over CSI works half speed only

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

We are implementing some driver modifications based on the Leopard Imaging IMX377 driver, since that is almost unusable demo code.

We are experiencing weird bandwidth limitation issues - it seems that the sensor can be run only at half speed. We got working 3x 2lane CSI RAW capture, and 1x 4 lane CSI RAW was working situation initially with the LI driver, LI claiming that multiple RAW are not possible.

But when we run an ISP process, eg a SCF based camera-daemon + gstreamer pipeline (e.g. to show a live image), then we are able to capture 2 more RAW streams. By the strace we see that nvcamera-daemon is using some secret calls to configure the VI or ensure some ISOBW.

How can we achieve the same configuration without using any ISP related code (Argus or SCF/camera_daemon)?

The error which is presented to us is:

[  841.797075] tegra-vi4 15700000.vi: master error

Looking to sources, it refers to about 6-7 conditions, from which 4 are non-resolveable because the Parker TRM is limited and does not show which bits belong to the specific conditions.

Does this relate to some VI clocking, MC clocking, or any special setup?

The device tree does list proper values for the VI BW allocation, and dump from sysfs does show increased allocation when multiple RAW captures are ran, but it still fails.

DTSI:

tegra-camera-platform {
		num_csi_lanes = <12>;
		max_lane_speed = <1440000>;
		min_bits_per_pixel = <12>;
		vi_peak_byte_per_pixel = <2>;
		vi_bw_margin_pct = <25>;
		max_pixel_rate = <1200000>;
		isp_peak_byte_per_pixel = <5>;
		isp_bw_margin_pct = <25>;

The first four lines show a 12 lane x 1440mbit/s I/O with 12 into 16 bpp memory writes. And it does not work.

Where can we find a proper CSI driver development guide ?

Our use case is three simultaneous 12 MPx @ 35 fps (12bit RAW)

2

Sorry, really don’t understand what your question and what you what. If you capture raw data from v4l2-ctl and got frame rate problem that should be the sensor output problem. Could you probe the mipi to make sure the output frame rate is as expect.

3

@ShaneCCC the sensors work individually very well, that is what we have tested before approaching simultaneous capture.

Test cases:

  • capture RAW from Sensor 1 in 4 lane cfg: PASS
  • capture RAW from Sensor 2 in 4 lane cfg: PASS
  • capture RAW from Sensor 3 in 4 lane cfg: PASS
  • capture RAW from Sensor 1 in 2 lane cfg: PASS
  • capture RAW from Sensor 2 in 2 lane cfg: PASS
  • capture RAW from Sensor 3 in 2 lane cfg: PASS
  • capture RAW from Sensor 1,2,3 in 2 lane cfg - simultaneous: PASS

Does not work:

  • capture RAW from Sensor 1,2 in 4 lane cfg - simultaneous: FAIL
  • capture RAW from Sensor 1,3 in 4 lane cfg - simultaneous: FAIL
  • capture RAW from Sensor 2,3 in 4 lane cfg - simultaneous: FAIL
  • capture RAW from Sensor 1,2,3 in 4 lane cfg - simultaneous: FAIL

Very weirdly however:

  • start ISP process on Sensor 1 in 4 lane cfg (argus or gstreamer) and capture RAW from Sensor 2,3 in 4 lane cfg - simultaneous: PASS

We do see, that the camera-daemon (SCF) makes some settings with clock rates to the kernel, which is likely why the capture works when such an ISP related process is running. But that portion of your code is a blackbox, with no sources provided :(

PASS = we get expected framerate
FAIL = we get nothing: (tegra-vi4: master error)

4

Did you capture RAW by v4l2-ctl? How do you capture RAW from 2 sensor source?

5

We run multiple v4l2-ctl instances in parallel on different /dev/videoX nodes

6

What’s the resolution? It could be bandwidth problem. Could you try if lower resolution is working.

7

4 lane: 4104 x 3046 @ 30 fps
2 lane: 4104 x 3046 @ 15 fps

It is not a resolution problem. It is a pixel rate problem - since the “master error” kernel message refers to some FIFO overflows according to the Parker TRM. The 2-lane mode has naturally only the half the pixel rate and this is why it works.

So tell me - where do we set the VI pipeline to be able to capture full-speed?
What is the secret call from nvcamera-daemon, which makes these things work?

8

For the v4l2-ctl pipeline all of the source are in the kernel/kernel-4.4/drivers/media/platform/tegra/camera
What do you mean “What is the secret call from nvcamera-daemon, which makes these things work?”

9

When the gstreamer is ran on sensor 1, then we can capture two more RAWs from sensor 2 and 3.

Stoppping the gstreamer on sensor 1, one or both the raw captures from sensor 2 and 3 will immediately crash with the “master error” message, and the system is in half-speed mode.

From the verbose SCF logs we can identify these messages that may relate to enabling the full speed:

PowerServiceUtils:calculateReqClock: entered
PowerServiceHw:addRequest: table size: before: 0, after:1
	request table for VI 0:
	req[0]: guID=2, stageID=SensorCapture
	req[0]: inW=4104, inH=3046, inBpp = 12, fps=30
	req[0]: outW=4104, outH=3046, outBpp=12
	req[0]: out1W=0, out1H=0, out1Bpp=0
	req[0]: out2W=0, out2H=0, out2Bpp=0
	req[0]: clock=100800000, pixelRate=576000000, timeout=900
	req[0]: isoBw=1209600, timeout=900
	req[0]: non_isoBw=0, timeout=900
PowerServiceUtils:calculateReqClock: entered
PowerServiceHw:addRequest: table size: before: 0, after:1
	request table for CSI 0:
	req[0]: guID=2, stageID=SensorCapture
	req[0]: inW=4104, inH=3046, inBpp = 12, fps=30
	req[0]: outW=4104, outH=3046, outBpp=12
	req[0]: out1W=0, out1H=0, out1Bpp=0
	req[0]: out2W=0, out2H=0, out2Bpp=0
	req[0]: clock=140400000, pixelRate=576000000, timeout=900
	req[0]: isoBw=0, timeout=900
	req[0]: non_isoBw=0, timeout=900
PowerServiceHwVi:setIso: m_bwVal_Iso=1209600
PowerServiceHw:setClock: PowerServiceHw[1]: requested_clock_Hz=100800000
PowerServiceHw:setClock: PowerServiceHw[0]: requested_clock_Hz=140400000
PowerServiceCore:setCameraBw: totalIsoBw=1209600

Looking into kernel sources I have not identified any cause why the pipeline runs at half the speed.

The cause may be maybe that the memory interface from VI is too slow, and VI can not push enough data. This is overriden by the nvcamera-daemon, when it starts using the ISP. As there are no SCF sources, I am unable to make a proof of concept program which would make the proper calls, without any relation to ISP - just to make possible the three simultaneous RAW captures.

10

How about run the ./jetson_clcoks.sh to boost as performance mode?

11

That we do always and it does not help.

Only running the gstreamer or argus makes the difference. But that blocks one of the cameras, which is not acceptable, we want all three in RAW mode.

12

Could you try below command to increase the vi clock

sudo su
$ cd /sys/kernel/debug/bpmp/debug/clk
$ cat vi/rate
$ echo 1 > vi/state
$ echo 1 > vi/mrq_rate_locked
$ cat vi/max_rate > vi/rate
13

It works! Thank you very much ShaneCCC.

$ sudo su
# cd /sys/kernel/debug/bpmp/debug/clk
# cat vi/rate
409600000
# cat vi/max_rate
1036800000
# echo 1 > vi/state
# echo 1 > vi/mrq_rate_locked
# cat vi/max_rate > vi/rate
# cat vi/rate
1036800000