MAX9286 frame sync works with 1 camera but fails with 2 cameras

ahuska · March 8, 2020, 10:11pm

Hello,

This question is not specific to the Jetson TX2, but rather the MAX9286 GMSL1 deserializer.

I have been using the MAX9286 with a single camera (image sensor plus MAX9271 serializer). The deserializer is configured in manual frame sync with the frame period registers set appropriately for my frame rate. With a single camera I can stream successfully with no issues…the deserializer get a lock on the video link, it detects the vsync from the camera, the frame sync lock bit is set, and valid CSI data is received by the Jetson.

When I connect a second camera the deserializer gets a lock on both video links, but it does not detect vsync from either camera, the frame sync bit stays clear, and no CSI data is received by the Jetson. I have tried changing the order in which the cameras begin to stream, the serializers are set to enabled, and the deserializer is set to enable output, but there is no change in frame sync lock behavior.

I tried configuring the MAX9286 to mask out one of the active cameras prior to enabling the video link and that allows me to successfully stream both virtual channels on my host. The vsyncs are detected and the frame lock happens as would be expected. Naturally the masked out link is all green, but the unmasked camera is coming through perfectly and I’m getting the full frame rate on both channels. Unfortunately, when I unmask the 2nd link (or change the mask to the other camera) by sending an i2c command from the console I lose the frame sync lock, the lock is not restored, and there’s no longer any CSI output. Oddly the MAX9286 does not set the “frame sync lock lost” bit, it just clears the lock bit.

With frame sync set to Automatic or Semi-automatic I can not get frame sync to work even with the masking, so I am using Manual frame sync mode for all testing. The MAX9286 is not generating vsync internally.

I assume that I am very close but have an issue in the order of initialization or a timing issue.

I’ve seen at least two people on this forum get the 9286 to work with 2 or more cameras, can anyone provide some assistance? I also have an inquiry in with Maxim, and I will update this topic if I learn more from them.

Andy

Related threads:

kayccc · March 9, 2020, 11:05pm

Hi ahus,ka

We don’t have experience on MAX9286 GMSL1 deserializer, may other developers share their thoughts with you.

greivin.fallas · March 11, 2020, 12:20am

Hi ahuska,

At RidgeRun we have worked with GMSL SerDes and frame synchronization.

I haven’t used this kind of chips but I referred your question to a Team with more experience in GMSL technologies.

Note about GMSL drivers: https://developer.ridgerun.com/wiki/index.php?title=MAX96705_MAX9286_GMSL_SERDES_Linux_Drivers#MAX96705_serializer_and_MAX9286_deserializer_chips

Greivin F.

CarlosR92 · March 11, 2020, 2:47pm

Hi ahuska,

As Greivin, introduced, at RidgeRun we have worked a lot with GMSL/GMSL2 SerDes and frame synchronization.

Attempting to capture from 2 cameras simultaneously is kind of tricky and depending of the sensors, all the chips require special configuration. In GMSL1, there are 2 ways to achieve camera synchronization:

1 ) <b>Automatic Frame Sync</b>: The deserializer generates its own sync signals based on VSYNC analysis of the signals coming from the serializer. In case of multiple serializers such as the MAX9286, it tries to sync the VSYNC of all the cameras based on the number of cameras connected and the VSYNC delays present on each video link.
2 ) <b>External Frame Sync</b>: By using the GPI pin in the deserializer, and configuring the registers accordingly, you can forward any signal used as trigger to the serializers. This method should be the one used in case you want to sync multiple cameras connected to multiple deserializers, because the same external sync source can be used in every deserializer.

I’m not sure which mode you are using, but for example, for one customer we were using the OV10635 sensor in Automatic frame sync, and we were getting frame sync problems. Digging a lot into the problem we identified that the OV10635 was not using the sync signal generated by the deserializer and forwarded by the serializer, so that the deserializer was not able to sync between the 2 cameras. Finally we managed to configure the OV10635 sensor to use the sync signal, in trigger mode, and suddenly we got frame sync and frames coming out of the deserializer, this because both cameras were synchronized.

So, my recommendation would be to make sure your sensors are using the signal provided by the deserializer, either in Automatic or External frame sync modes. Otherwise the deserializer won’t be able to get the proper synchronization between them.

Carlos R.

ahuska · March 13, 2020, 12:42am

Hello All,

Thank you for your input, it is very appreciated.

I am using Manual frame sync where the 9286 generates the frame sync signal internally and sends it to the serializer and sensor. I have verified that the sensor has FSIN enabled and that the signal is going out of the serializer GPO pin.

This week I started probing the camera and determined that Vsync is not coming out of the sensor after initialization. During initialization I can measure Vsync pulses of proper duration on my scope, but they stop after ~10 frames. I was able to manually enable the Vsync output but when I do that the Vsync pulse period is only 10ns so it is not being detected by the serializer and deserializer. According to the sensor datasheet Vsync should be active for about 1280ns, so I must have a setting wrong. I have reached out to the vendor’s FAE for assistance in configuring the sensor for Vsync.

I will keep you updated.

ahuska · March 19, 2020, 6:51pm

Hi All,

I experimented with the Vsync registers of the sensor and was able to increase the vsync period to 1.28us and get all of the Vsycn inversion modes aligned in the sensor, serializer, and deserializer. The MAX9286 is now able to recognize the Vsync, get a frame sync lock, and stream both virtual channels to the TX2.

Now I have a couple of details to work out:

The sensor must be configured with Frame Sync Input (FSIN) enabled or the 9286 loses frame sync with a few frames, but when FSIN is enabled the image received often contains a bright row of pixels, sometimes two, see the embedded screenshot:

Screenshot from 2020-03-19 10-49-021864×1048 563 KB

If there are no major highlights in the scene then there is no line for some reason, and disabling HDR has some effect on the bright line behavior as well. Does any one know what might be causing this?
I can stream both virtual channels on the TX2 for 10 or 15 minutes then I start to get a reduced frame rate with dmesg errors and ultimately the cameras stop streaming completely with timeout errors that look like the 9286 is no longer sending data, the 9286 loses frame lock, and I lose I2C communication to the camera until I reboot the system. Oddly I don’t see any obvious errors in the “cat /sys/kernel/debug/tracing/trace” output so I suspect a DeSer configuration problem related to signal strength although my Fakra cables are only 2m long. I will experiment more and report back.

Andy

ahuska · March 26, 2020, 5:34pm

In the interest of supporting the community here are some things I learned:

Regarding the bright line issue above: This is solved by setting the frame sync pulse period to be EXACTLY was is specified by the image sensor datasheet. My calculations were off by 200 pclk periods with a total frame time of 3,200,000 counts (so about 0.6% error) and that was close enough for it to work, but not enough to look right.

I am still adjusting settings and performing long term tests on the frame lock failure, but setting the exact frame timing will likely help this, too.

One implication of this is that when using virtual channels on a MAX9286 you must use identical sensors with identical frame and format settings.

More to come,
Andy

ahuska · March 27, 2020, 7:39pm

I have confirmed that setting the frame time to be exactly the correct number of pclk periods as defined in the datasheet fixed the loss of frame sync lock. I ran two cameras on one MAX9286 for 20 hours with no loss of lock.

One new problem found is that after very long streaming sessions, in this case 14 hours, I had a single camera attached to another MAX9286 and it froze up with a very very dark image on the screen and the i2c bus is no longer functional. When I try to access the bus I get the following error in serial console:

[84444.162662] tegra-i2c 3180000.i2c: arb lost in communicate to add 0x2c                                                
[84454.383280] tegra-i2c 3180000.i2c: timed out for bus clear                                                            
[84454.388946] max9286 2-002c: I2C write failed

Unfortunately I can not find any software command to restore communication to the i2c bus.

More to come…
Andy

ahuska · March 30, 2020, 9:03pm

Hello again,

In case anyone is interested, I ran the long term test again with all 6 cameras running (2 cams per DeSer). It took about 48 hours for the failure to occur during with my RTP client received 4.5TB of data…just a fun statistic.

This time when the failure occurred two camera streams were frozen, possibly because the MAX9286 did not detect a fault from the frozen camera. In any case, the I2C bus was locked up again so I unplugged one of the cameras and the other one immediately started streaming again so the MAX9286 was working fine and as soon as it lost the connection to the first camera it masked out that link and started working again. There is something going on in the camera (MAX9271 or OV10635) that is causing the bus to be locked. Next time this occurs I will probe to see if the SDA or SCL lines are held low. If they are then I will try to disable the reverse control channel after the sensor is configured so the I2C activity of the camera module does not affect my host (theory at this point).

Andy

qichen.ou · March 31, 2020, 9:11am

Hi ahuska,
Thank you for sharing, I’m new for this,I want to achieve 2 or 4 cameras simultaneously in one Max9286 .Should I refer to imx390.c, max9296.c, max9295.c in l4t 32.3.1?

hardware: max9286+max96705+ar0143(with ap0101)

thanks!

ahuska · March 31, 2020, 10:12am

Hi qichen.ou, please start a new topic and I will give you advice there. PM me a link to the new thread when you create it.

CarlosR92 · April 1, 2020, 9:54pm

Hi @ahuska,

Have you checked registers 0x31 and 0x27 of the MAX9286 when the issue happens? May be it is losing synchronization after some time running (it happened to me once).

ahuska · April 1, 2020, 10:08pm

Hi Carlos,

Thanks for the thought. The strange thing is that my I2C bus is locked up when this happens. The Jetson reports an I2C lost arbitration error and I can not read or write anything until I disconnect the frozen camera. It is possible I have a bad camera, so I have numbered them and will see if it is always the same one.

Andy

zhouqiang · May 5, 2020, 1:09pm

Hi Ahuska,
Does it solved,we have the similar issue,mx9288+max96705.the SOC I2C arb loss,disconnect camera and can not retry I2C.the SDA latch low and SCL change from 400K to 300K.

ahuska · May 5, 2020, 4:30pm

Hello zhouqiang,

I did solve this issue just this week with the help of Maxim. The solution is to enable High Immunity Mode! I do not know if the 9288 supports HIM, but you can contact Maxim FAE for more information and detailed support for your platform, they have been very helpful, but it does include NDA materials so I can’t share it all here.

Andy

ahuska · May 5, 2020, 4:35pm

Hello All,

On my quest for 100% stability with the MAX9286 and MAX96705 + OV10635 I am very close. With FAE help (mentioned above) I am now able to stream multiple cameras using VCIDs for 60 hours without a single frame dropped!

The final remaining issue is i2c communications on cold boot of the system. Most of the time when the system boots I can initialize the 9286 and 96705 without error, but immediately afterwords I can no longer read or write to the serializer or image sensor. When it does this I can reinitialize the Des and Ser with no power cycle and the init works, but still no i2c comms. When the system boots and i2c comms are good then I never have an issue, everything works perfectly, but when it boots then fails after init it never recovers without a power cycle.

I have rearranged my init sequence, isolated registers, slowed down the i2c host bus and the i2c link between Ser and Des, and nothing solves the problem. Has anyone seen this?

Andy

Luna2020 · May 20, 2020, 3:52am

Hi ahuska,

I’m new to GMSL Camera too, and debug with max9286 and max96705 using v4l2, can you give me a favor on it?
Now I can open the first gmsl camera and preview successfully, and get two camera successfully after initialization but can’t open the second gmsl camera. (using command to open camera0 or 1: gst-launch-1.0 v4l2src device=/dev/video0/1 ! ‘video/x-raw,width=1280,height=720,format=(string)YUY2’ ! nvvidconv ! ‘video/x-raw(memory:NVMM),format=(string)NV12’ ! nvoverlaysink)

I have some question about two-cameras, I appreciate your suggestions:

Can you open the second camera use the same command as the first camera but just change the camera node ?
How to open and preview mulit-cameras such as two cameras, can you share the commands you are using to open multi-cameras?
After you implement frame-sync, each time to open single or multi-cameras you can get the correct MAX9286 frame-sync(0x31=0x40) status ? I’m using an external signal(50Hz square wave) to trigger frame-sync for multi-cameras frame sync, but unfortunately I can’t get the correct frame sync in MAX9286 0x31 register.

nvidia@xavier:~$ ./i2cdump.sh (Opening the second camera but failed)

	Max9286 dump
	No size specified (using byte-data access)
		 0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
	00: e2 e2 00 01 00 00 00 00 26 90 ff e4 91 f3 50 0b    ??.?....&?.???P?
	10: 00 00 f3 0f e4 1b 00 01 00 0c 60 00 f4 ff 40 02    ..????.?.?`.?.@?
	20: 02 00 20 00 00 00 00 92 00 00 00 00 00 00 00 00    ?. ....?........
	30: 00 00 99 99 b5 00 00 00 00 00 00 19 24 54 c8 22    ..???......?$T?"
	40: 5a 10 00 a0 00 00 04 22 c2 02 ff 00 00 00 00 00    Z?.?..?"??......
	50: 00 00 05 00 00 00 00 00 00 00 00 02 00 00 00 0f    ..?........?...?
	60: 45 1f ff 00 00 12 96 00 50 00 00 00 00 00 00 00    E?...??.P.......
	70: 20 12 00 00 00 00 00 00 00 00 00 00 00 00 00 00     ?..............
	80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
	90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
	a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
	b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
	c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
	d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
	e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
	f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
	 
	Max96705 dump  
	No size specified (using byte-data access)
		 0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
	00: 84 90 1f 00 83 00 a0 84 00 38 ba 8a 84 b6 02 bf    ???.?.??.8??????
	10: 20 00 40 02 60 e3 1f 93 ff ff ff ff 00 00 41 01    !.@?`???......A?
	20: 00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f    .???????????????
	30: 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 0e    ????????????????
	40: 0f 0d 5b 05 00 00 00 00 00 00 00 00 00 c0 00 00    ??[?.........?..
	50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
	60: 00 00 00 00 00 00 61 c7 19 0f 00 00 00 00 00 00    ......a???......
	70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
	80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
	90: 00 00 00 00 00 00 06 1f 4a 0d 10 00 00 00 00 00    ......??J??.....
	a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
	b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
	c0: 00 00 00 00 00 00 00 00 0a ff 00 00 00 00 00 00    ........?.......
	d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
	e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
	f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................

Thanks so much.

nadeemm · October 18, 2021, 6:18pm