TX2 NX, Lontium lt6911uxc and V4L2

Greetings,

We’ve been working on using the lt6911uxc to input HDMI video to the TX2 NX.

We cannot capture video with v4l2. We have probed the CSI lines from the lt6911uxc chip and it appears to have data coming out.

Questions:

1) We put in mode info in the DT (Device Tree) and now we see the video input change from HDMI to Cameral when we use ‘v4l2-ctl --all’. Why has the video input changed from HDMI to Camera? I have a Toshiba tc358743 running with and Xavier NX that shows up as HDMI not a camera.

Before:  
Video input : 0 (HDMI 0: no signal)

After:
Video input : 0 (Camera 0: no power)

2) We have noticed that the lt6911uxc driver is loaded but it shows up as not being used until we try and capture video with the following. Again, the Toshiba setup shows the driver in use even when video is not being captured. Should the driver be in use?

v4l2-ctl --device /dev/video0 --set-fmt-video=width=1280,height=720,pixelformat=YUV --stream-mmap --stream-to=frame.jpg --stream-count=1

3) When trying to capture video the v4l2-ctl command just hangs trying to capture video and produces these repeating errors in dmesg:

Nov 3 16:27:23 ams1pdev5 kernel: [ 2666.220223] tegra-vi4 [15700000.vi](http://15700000.vi/): PXL_SOF syncpt timeout! err = -11 

Nov 3 16:27:23 ams1pdev5 kernel: [ 2666.226739] tegra-vi4 [15700000.vi](http://15700000.vi/): tegra_channel_error_recovery: attempting to reset the

4) Is anyone using the Lontium dev/evk board and wiring the CSI lines to a carrier board? If so is there a particular carrier board that exposes all 4 lines on each CSI port of the SoM? We tried using the Xavier NX Dev board but the carrier board only exposes 2 lines. In theory we only need this for 4kp60 but part of what we are trying to do is prove the lt6911uxc is capable of capturing 4kp60. At this point however if we could do 4kp30 with 1 lane I’d be happy. Heck, I’d even take 480i just to prove that we get something.

Thanks in advance for help with this.

>> Q1
please check Kconfig, you may modify that similar to Toshiba bridge driver for confirmation.

>> Q2
may I know what’s the sources of this lt6911uxc bridge? or, had you enable test-pattern-generator?

>> Q3
can you confirm there’s signaling sending to CSI brick?
this error log PXL_SOF, it means VI engine cannot receive start-of-frame correctly.
it may due to you’ve longer initial time then the timeout limit, i.e. 2500ms.
please add set_mode_delay_ms DT settings to configure wait time for the first frame after capture starts, the unit is in milliseconds.

>> Q4
I am not quite understand this part.
there’s lane configuration in device tree, which correspond to the actual hardware settings.
for example, if your bridge used 4-lane, you need to configure bus-width = <4>.
moreover, for TX2 series, due to the bandwidth, you should enable 8-lane to bring-up 4kp60.

@JerryChang Thanks for the help with this.

Q1
This is what I’ve got for the lt6911uxc in /Linux_for_Tegra/source/public/kernel/nvidia/drivers/media/i2c/Kconfig

config VIDEO_LT6911UXC
        tristate "Lontium LTX6911UXC HDMI to CSI bridge"
        depends on VIDEO_V4L2 && VIDEO_V4L2_SUBDEV_API
        ---help---
          This driver supports LT6911UXC bridge from Lontium

          To compile this driver as a module, choose M here: the module
          will be called lt6911uxc.

For the Toshiba the Kconfig is over here :
/Linux_for_Tegra/source/public/kernel/kernel-4.9/drivers/media/i2c/Kconfig

config VIDEO_TC358743
        tristate "Toshiba TC358743 decoder"
        depends on VIDEO_V4L2 && I2C && VIDEO_V4L2_SUBDEV_API
        select HDMI
        ---help---
          Support for the Toshiba TC358743 HDMI to MIPI CSI-2 bridge.

          To compile this driver as a module, choose M here: the
          module will be called tc358743.

So we would add the “select HDMI” in the Lontium section and re-make the kernel first re-running “menuconfig”?
In order for it to work correctly do we need to add this or does it matter?

Q2
We are using a TV set top box as the source video. It is not using HDCP. It is the same source we are using with the Toshiba chip/bridge.

Q3
How exactly can we confirm the signaling to the CSI brick? We have probed the CSI lines coming from the Lontium chip and can verify there is a signal.
Would we set the 'set_mode_delay_ms` in the the VI section of the DT or somewhere else?

Q4.
This is good information. The way we understand it the TX2 has 2 CSI ports and each port has 4 CSI lanes. This would give us the 8 we need for 4kp60. We also understand there is a clock tied to each CSI port. In order to use all 8 lanes we have to use one clock for both ports. How is this set?
Also, the Lontium firmware is configured to use 1 port ( 4 CSI lanes) for 4kp30 and below. If it is 4kp60 it will use 2 ports ( 8 CSI lanes). Is this possible to configure so that we can switch between resolutions?

Our first step is to verify the Lontium chip is capable of ingesting video.
Second we need to verify the Lontium chip is capable of ingesting 4kp60 video.
We have been unable to do either of these steps. Part of our problem is we are working with a newly developed carrier board for the SOM that integrates the Lontium chip in the design. We would love to first work with a known hardware platform e.g. an Nvidia Dev Kit with a dev board from Lontium in order to prove out the solution but nothing seems to exist. Please share if you know of something that exists. A platform must exist as something was used to develop the Nvidia driver for the Lontium chip.

hello edward.dow,

actually, I did not notice the difference before you said that. so, please have a try to update kernel config.

please refer to Sensor Software Driver Programming, it’s device tree property, set_mode_delay_ms.

one CSI brick capable to 1-lane, 2-lane, and 4-lane.
there’s gang mode mechanism, (for only 4k resolutions). if you assign bus-width = <8>; in device tree. VI driver will enable two CSI bricks to handle the signaling. these must be continuous 8-lanes configuration. for example, CSI-A,B/CSI-C,D.
you may dig into VI driver for gang mode implementation.
for example, $public_sources/kernel_src/kernel/nvidia/drivers/media/platform/tegra/camera/vi/channel.c

this looks promising.
so, basically, you’ll need to configure device tree as same as hardware config.
you should have correct port bindings. please see-also bridge driver, tc358840 for reference.

VI tracing log it helps to dig into the issue.
here’re steps to enable that, please share VI tracing logs for reference if Tegra side still not receiving the signaling.
for example,

echo 1 > /sys/kernel/debug/tracing/tracing_on
echo 30720 > /sys/kernel/debug/tracing/buffer_size_kb
echo 1 > /sys/kernel/debug/tracing/events/tegra_rtcpu/enable
echo 1 > /sys/kernel/debug/tracing/events/freertos/enable
echo 2 > /sys/kernel/debug/camrtc/log-level
echo > /sys/kernel/debug/tracing/trace
cat /sys/kernel/debug/tracing/trace
# to review dynamic tracing log 
# cat /sys/kernel/debug/tracing/trace_pipe 

Q2
Made the change in the DT but haven’t seen any difference.

Q3
We have gotten new information from Lontium that 4kp60 4:2:2 can be done with 1 CSI port and 4 lanes. They have said the FW needs to be set up correctly so working with them to provide a new FW after we clarify our 4kp60 is in a 4:2:2 color space

Q4
Set up the VI tracing and get the following:

cat /sys/kernel/debug/tracing/trace
# tracer: nop
#
# entries-in-buffer/entries-written: 473/473   #P:4
#
#                              _-----=> irqs-off
#                             / _----=> need-resched
#                            | / _---=> hardirq/softirq
#                            || / _--=> preempt-depth
#                            ||| /     delay
#           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION
#              | |       |   ||||       |         |
     kworker/0:2-16555 [000] .... 166367.524262: rtos_queue_peek_from_isr_failed: tstamp:5199320602634 queue:0x0b4b4940
     kworker/0:2-16555 [000] .... 166367.692245: rtos_queue_peek_from_isr_failed: tstamp:5199325602624 queue:0x0b4b4940
     kworker/0:2-16555 [000] .... 166367.860267: rtos_queue_peek_from_isr_failed: tstamp:5199330602620 queue:0x0b4b4940
     kworker/0:2-16555 [000] .... 166368.028372: rtos_queue_peek_from_isr_failed: tstamp:5199335602611 queue:0x0b4b4940
     kworker/0:2-16555 [000] .... 166368.196249: rtos_queue_peek_from_isr_failed: tstamp:5199340602603 queue:0x0b4b4940
     kworker/0:2-16555 [000] .... 166368.364265: rtos_queue_peek_from_isr_failed: tstamp:5199345602597 queue:0x0b4b4940
     kworker/0:2-16555 [000] .... 166368.532256: rtos_queue_peek_from_isr_failed: tstamp:5199350602593 queue:0x0b4b4940
     kworker/0:2-16555 [000] .... 166368.644263: rtos_queue_peek_from_isr_failed: tstamp:5199355602585 queue:0x0b4b4940
     kworker/0:2-16555 [000] .... 166368.812262: rtos_queue_peek_from_isr_failed: tstamp:5199360602578 queue:0x0b4b4940
     kworker/0:2-16555 [000] .... 166368.980252: rtos_queue_peek_from_isr_failed: tstamp:5199365602608 queue:0x0b4b4940
     kworker/0:2-16555 [000] .... 166369.148274: rtos_queue_peek_from_isr_failed: tstamp:5199370602569 queue:0x0b4b4940
     kworker/0:2-16555 [000] .... 166369.316270: rtos_queue_peek_from_isr_failed: tstamp:5199375602555 queue:0x0b4b4940
     kworker/0:2-16555 [000] .... 166369.484286: rtos_queue_peek_from_isr_failed: tstamp:5199380602549 queue:0x0b4b4940
     kworker/0:2-16555 [000] .... 166369.652277: rtos_queue_peek_from_isr_failed: tstamp:5199385602544 queue:0x0b4b4940
     kworker/0:2-16555 [000] .... 166369.764285: rtos_queue_peek_from_isr_failed: tstamp:5199390602538 queue:0x0b4b4940
     kworker/0:2-16555 [000] .... 166369.932280: rtos_queue_peek_from_isr_failed: tstamp:5199395602528 queue:0x0b4b4940
     kworker/0:2-16555 [000] .... 166370.100287: rtos_queue_peek_from_isr_failed: tstamp:5199400602521 queue:0x0b4b4940
     kworker/0:2-16555 [000] .... 166370.268295: rtos_queue_peek_from_isr_failed: tstamp:5199405602515 queue:0x0b4b4940
     kworker/0:2-16555 [000] .... 166370.436292: rtos_queue_peek_from_isr_failed: tstamp:5199410602508 queue:0x0b4b4940
     kworker/0:2-16555 [000] .... 166370.604314: rtos_queue_peek_from_isr_failed: tstamp:5199415602502 queue:0x0b4b4940
     kworker/0:2-16555 [000] .... 166370.772502: rtos_queue_peek_from_isr_failed: tstamp:5199420602505 queue:0x0b4b4940
     kworker/0:2-16555 [000] .... 166370.884300: rtos_queue_peek_from_isr_failed: tstamp:5199425602489 queue:0x0b4b4940
     kworker/0:2-16555 [000] .... 166371.052307: rtos_queue_peek_from_isr_failed: tstamp:5199430602480 queue:0x0b4b4940
     kworker/0:2-16555 [000] .... 166371.164299: rtos_queue_peek_from_isr_failed: tstamp:5199433571715 queue:0x0b4b4940
     kworker/0:2-16555 [000] .... 166389.140663: rtos_queue_peek_from_isr_failed: tstamp:5199995171907 queue:0x0b4b4940
     kworker/0:2-16555 [000] .... 166389.140683: rtcpu_start: tstamp:5199995172880
     kworker/0:2-16555 [000] .... 166389.140690: rtos_queue_send_from_isr_failed: tstamp:5199995179559 queue:0x0b4a7698
     kworker/0:2-16555 [000] .... 166389.140692: rtos_queue_send_from_isr_failed: tstamp:5199995179665 queue:0x0b4ab1a8
     kworker/0:2-16555 [000] .... 166389.140694: rtos_queue_send_from_isr_failed: tstamp:5199995179773 queue:0x0b4acdd8
     kworker/0:2-16555 [000] .... 166389.140697: rtos_queue_send_from_isr_failed: tstamp:5199995179882 queue:0x0b4af718
     kworker/0:2-16555 [000] .... 166389.140699: rtos_queue_send_from_isr_failed: tstamp:5199995179985 queue:0x0b4b04d8
     kworker/0:2-16555 [000] .... 166389.140701: rtos_queue_send_from_isr_failed: tstamp:5199995180090 queue:0x0b4b1298
     kworker/0:2-16555 [000] .... 166389.140702: rtos_queue_send_from_isr_failed: tstamp:5199995180194 queue:0x0b4b2058
     kworker/0:2-16555 [000] .... 166389.140710: rtos_queue_send_failed: tstamp:5199995180634 queue:0x0b4a7698
     kworker/0:2-16555 [000] .... 166389.140712: rtos_queue_send_from_isr_failed: tstamp:5199995182763 queue:0x0b4a7698
     kworker/0:2-16555 [000] .... 166389.140714: rtos_queue_send_from_isr_failed: tstamp:5199995182869 queue:0x0b4ab1a8
     kworker/0:2-16555 [000] .... 166389.140717: rtos_queue_send_from_isr_failed: tstamp:5199995182974 queue:0x0b4acdd8
     kworker/0:2-16555 [000] .... 166389.140718: rtos_queue_send_from_isr_failed: tstamp:5199995183083 queue:0x0b4af718
     kworker/0:2-16555 [000] .... 166389.140720: rtos_queue_send_from_isr_failed: tstamp:5199995183192 queue:0x0b4b04d8
     kworker/0:2-16555 [000] .... 166389.140722: rtos_queue_send_from_isr_failed: tstamp:5199995183296 queue:0x0b4b1298
     kworker/0:2-16555 [000] .... 166389.140724: rtos_queue_send_from_isr_failed: tstamp:5199995183401 queue:0x0b4b2058
     kworker/0:2-16555 [000] .... 166389.140726: rtos_queue_send_failed: tstamp:5199995184314 queue:0x0b4a7698
     kworker/0:2-16555 [000] .... 166389.140745: rtcpu_vinotify_event: tstamp:5199995224752 tag:CSIMUX_STREAM channel:0xff frame:0 vi_tstamp:5199995224354 data:0x00000001
     kworker/0:2-16555 [000] .... 166389.140747: rtcpu_vinotify_event: tstamp:5199996050341 tag:CSIMUX_FRAME channel:0x00 frame:1 vi_tstamp:5199996049488 data:0x00c00060
     kworker/0:2-16555 [000] .... 166389.140748: rtcpu_vinotify_event: tstamp:5199996050654 tag:CHANSEL_SHORT_FRAME channel:0x01 frame:1 vi_tstamp:5199996049488 data:0x00000001
     kworker/0:2-16555 [000] .... 166389.140749: rtcpu_vinotify_event: tstamp:5199996050953 tag:ATOMP_FE channel:0x00 frame:1 vi_tstamp:5199996049491 data:0x00000001
     kworker/0:2-16555 [000] .... 166389.196674: rtcpu_vinotify_event: tstamp:5199996591499 tag:CSIMUX_FRAME channel:0x00 frame:1 vi_tstamp:5199996591006 data:0x000000a0
     kworker/0:2-16555 [000] .... 166389.196682: rtcpu_vinotify_event: tstamp:5199997112860 tag:CSIMUX_FRAME channel:0x00 frame:1 vi_tstamp:5199997112364 data:0x000000a0
     kworker/0:2-16555 [000] .... 166389.196686: rtcpu_vinotify_event: tstamp:5199997614056 tag:CSIMUX_FRAME channel:0x00 frame:1 vi_tstamp:5199997613563 data:0x00e00060
     kworker/0:2-16555 [000] .... 166389.252657: rtcpu_vinotify_event: tstamp:5199998155469 tag:CSIMUX_FRAME channel:0x00 frame:1 vi_tstamp:5199998155081 data:0x000000a0
     kworker/0:2-16555 [000] .... 166389.252664: rtcpu_vinotify_event: tstamp:5199998656773 tag:CSIMUX_FRAME channel:0x00 frame:1 vi_tstamp:5199998656281 data:0x00c00060
     kworker/0:2-16555 [000] .... 166389.252668: rtcpu_vinotify_event: tstamp:5199999178133 tag:CSIMUX_FRAME channel:0x00 frame:1 vi_tstamp:5199999177639 data:0x00c00060
     kworker/0:2-16555 [000] .... 166389.252671: rtcpu_vinotify_event: tstamp:5199999699495 tag:CSIMUX_FRAME channel:0x00 frame:1 vi_tstamp:5199999698997 data:0x00c00060
     kworker/0:2-16555 [000] .... 166389.308595: rtos_queue_peek_from_isr_failed: tstamp:5200000172394 queue:0x0b4b4940
     kworker/0:2-16555 [000] .... 166389.308600: rtcpu_vinotify_event: tstamp:5200000240984 tag:CSIMUX_FRAME channel:0x00 frame:1 vi_tstamp:5200000240515 data:0x000000a0
     kworker/0:2-16555 [000] .... 166389.308603: rtcpu_vinotify_event: tstamp:5200000762368 tag:CSIMUX_FRAME channel:0x00 frame:1 vi_tstamp:5200000761874 data:0x000000a0
     kworker/0:2-16555 [000] .... 166389.308606: rtcpu_vinotify_event: tstamp:5200001263567 tag:CSIMUX_FRAME channel:0x00 frame:1 vi_tstamp:5200001263074 data:0x00c00060
     kworker/0:2-16555 [000] .n.. 166389.364606: rtcpu_vinotify_event: tstamp:5200001805085 tag:CSIMUX_FRAME channel:0x00 frame:1 vi_tstamp:5200001804591 data:0x000000a0
     kworker/0:2-16555 [000] .... 166389.364661: rtcpu_vinotify_event: tstamp:5200002326443 tag:CSIMUX_FRAME channel:0x00 frame:1 vi_tstamp:5200002325950 data:0x000000a0

hello edward.dow,

according to those VI tracing logs, it tells there’s signaling.
however, it’s interrupted by this failure, tag:CHANSEL_SHORT_FRAME .

     kworker/0:2-16555 [000] .... 166389.140748: rtcpu_vinotify_event: tstamp:5199996050654 tag:CHANSEL_SHORT_FRAME channel:0x01 frame:1 vi_tstamp:5199996049488 data:0x00000001
     kworker/0:2-16555 [000] .... 166389.140749: rtcpu_vinotify_event: tstamp:5199996050953 tag:ATOMP_FE channel:0x00 frame:1 vi_tstamp:5199996049491 data:0x00000001

this usually due to the device tree settings mismatch to sensor hardware configuration.
please review your implementation.
thanks

@JerryChang
Thanks for taking a look at these errors!

The Lontium chip has FW that is loaded on the chip. I’m wondering if the mismatch is occurring due to incorrect FW for the settings we have in the DT.

The current FW we are using indicates RGB444. We do have another FW that is setup for YUV422.

Do you know if this is the pixel_phase attribute in the DT?
Our dtsi file is set to: pixel_phase = “rggb”;

As a reference my dtsi file for the Toshiba chip does not have a pixel_phase set. We are using the Nvidia Lontium driver. Maybe it’s a setting the particular driver needs set?

hello edward.dow,

what’s the pixel formats sending to CSI brick, it should be YUV, right?
actually, you may refer to Toshiba driver to leave only port bindings in device tree.

I believe it is. We received a FW for the Lontium chip that indicates it is YUV422.

With respect to the Toshiba driver are you saying to make the Lontium dtsi file look like the Toshiba dtsi?

hello edward.dow,

why don’t you share the device tree settings of Lontium for reference, thanks