NVidia Jetson AGX Xavier Module 32GB - Non Industrial Version
Xavier is connected to 3 MAX96712 deserializers
All Cameras on all deserializers are identical
Each MAX96712 deserialzier is connected via a 4 lane CSI port to the Xavier AGX.
Using the existing vi ports and csi channel nodes as a pattern we have the vi ports and csi channel nodes in the device tree.
Currently we can successfully acquire images on cameras 0-3 connected to Xavier via port-index 0
Currently we can successfully acquire images on cameras 4-7 connected to Xavier via port-index 2
However we can NOT acquire images on cameras 8-11 connected to Xavier via port-index 4
All cameras serailizers and deserialziers are accessble on the I2C busses
Following command is used to perform test capture:
gst-launch-1.0 v4l2src device=/dev/gmsl/cam8 num-buffers=1 ! jpegenc ! filesink location=/home/root/pics/cam8.jpg;
Using the trace debugging commands we observe the following when attempting to acquire and image we observed the following: trace-20240104-1330.BAD.txt (97.6 KB)
Looks like output indicates a multibit error in the stream.
Hard to imagine that is happening since all 3 deserializers are programmed virtually identicaly. The first 2 deserializers work fine and the third deserializer experiences this error with known good cameras.
So is their anything different about Xavier port-index 4 vs. (port-index 0 and port-index 2)?
the only difference for point-index=4 is there’s only single VI brick. but, this may not the root cause of your failure.
you may see-also Port Index section for reference.
anyways,
please give it a try with below commands to boost all the VI/CSI/ISP clocks.
for instance,
sudo su
echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/emc/mrq_rate_locked
cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate |tee /sys/kernel/debug/bpmp/debug/clk/vi/rate
cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/isp/rate
cat /sys/kernel/debug/bpmp/debug/clk/nvcsi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/nvcsi/rate
cat /sys/kernel/debug/bpmp/debug/clk/emc/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/emc/rate
We verified the following in the live devicetree /sys/firmware/devicetree/base:
For the camera that produces the NOVC errors we followed the phandle “links” between the camera, csi, and vi nodes and verified that:
*The vc-id property in the camera and phandle linked vi port all had vc-id property = 1
*The port-index property in the camera and phandle linked csi channel and phandle linked vi port all had port-index property = 4
verified in live /host1x/nvcsi@15a00000/num-channels = 12
@JerryChang can you verify that port-index = <4>; is the correct value for the port-index when the CSI lanes are connected to Xavier CSI 4 and CSI 5 lanes?
yes, that’s correct. you may also share the device tree setting for reference, please re-cap part of settings of camera8-11 for checking.
anyways, PHY_INTR0 with error code 0x01 it means there’s LP sequence error has detected on clock lane.
please give it a try to tune set_mode_delay_ms and cil_settletime property settings by Sensor Software Driver Programming.
We have attached an excerpt from the live device tree (/sys/firmware/devicetree/base/…) of the camera, csi, vi, and module device tree nodes for cameras 8-11 for you to check. devtreeExcerptForCheck.txt (27.6 KB)
We will now tune the paramters set_mode_delay_ms and cil_settletime
To set cil_settletime we used the following for a reference
your VI/CSI/Sensor port bindings looks correct.
please also execute v4l2-compliance for sanity test… i.e. $ v4l2-compliance -d /dev/video<n>
I suspect you have duplicated position properties for those camera nodes.
for instance,
tegra-camera-platform {
module8 {
position = "centerleft";
module9 {
position = "centerright";
module10 {
position = "topleft";
module11 {
position = "topright";
please check all these 12 camera nodes were using different position property values.
per developer guide, it’s only given example for 6-cam system, you may try using hex values instead for your use-case.
and… this may be minor.
please try remove those focuser node definitions.
@JerryChang Following are the results of the tests we ran for set_mode_delay_ms and cil_settletime parameters.
We assigned set_mode_delay_ms = “100”; and tested cil_settletime = “[13-27]”.
The results of all 15 tests were no successful image acquisition with the same “NOVC” errors.
All of our MIPI clock and data pairs are matched length for each of the 3 deserializers.
The table below shows the length differences as well as the corresponding time delay for the length difference.
One can see that the delay is less than 1 picosecond for any of the buses.
So unless delay added in the Xavier module, the signals should be arriving virtualy simultaneously.
FYI
-the buses for the derserializers that work are MIPI-0 and MIPI-1.
-the bus for the deserializer that does not work is MIPI-2.
the root cause should be PHY_INTR0 instead of STREAM_NOVC.
there’s workaround to ignore CRC/ECC report (but… I don’t kept r32.7.2 code base, is it possible for moving to r32.7.4?)
anyways, PHY_INTR0 with error code 0x01 it means there’s LP sequence error has detected on clock lane.
some experience to resolve such failure is adding software reset on serdes chip before s_stream().
may I also know what’s your hardware setups for testing camera:8-11?
for example, did you have all 3 sets of SerDes chips connected?
Moving from 32.7.2 would require a great deal of work. Before deciding to move to another version we would like to try the work arround.
@JerryChang Is it possible for you to show me the work around logic that needs to be placed in the code?
If you show me the lines of code for the work arround in a another version I can place them in the right spot in 32.7.2.
Our test setup for testing cameras 8 - 11 is shown below.
All cameras are known good cameras.
Cameras 0-7 are known to work.
Durning the tests that have have been running only cams 8-11 and cam 2 were connected.
We had cam 2 connected just to have a known working camera avaiable during testing.
Cameras 0,1,3-7 are grayed out to show they were not connected during the tests.
sorry, part of workaround to ignore CRC/ECC was handled by rce-fw, so, sharing the kernel change cannot helps.
this connection diagram helps greatly.
the truth is… camera device tree settings must be identical with your physical setups.
so,
you may give another try by disconnect deserializer0, deserializer1, and also removing those DT settings for confirmation.
We have the ability to power down the deserializers on our board. So we powered down deserializers 0 and 1.
We also removed all camera, csi, vi, and module device tree nodes from the device tree that were not being used to supoort the cameras on deserializer 2.
We have included a copy of this modifyed device tree, just in case you would like to see how we changed the device tree. devtreeExcerptForCheck2.txt (17.3 KB)
We enabled the debug trace.
We then attempted to capture an image.
We observed the following dmesg output while trying to capture image:
according to the kernel logs. you should have <bus>-<address> for each SerDes.
please confirm… is this still indicate your camera nodes from 1st deserializer? i.e. 0-000c.
device tree port bindings looks correct.
please download Jetson AGX Xavier Series OEM Product Design Guide, for checking [10.1.1 CSI D-PHY Design Guidelines] of Signal Routing Requirements.
and, please see-also [Figure 10-2. Camera CSI D-PHY Connections], to confirm it’s using CSI_4_CLK_P/N only for 4-lane sensor on CSI-E/F.
@JerryChang we have found that the MIPI CLK + DATA do not look correct coming out of deserializer. We will continue to look into this matter and report back what we find.
