4K HDMI Capture Module based on TC358840

Hi everybody

Our research group has developed a 4K HDMI capture module (named HDMI2CSI) that plugs into the Camera Connector (J22) on the Jetson TX1 (EDIT: and TX2) Development Board.

The module has served us as development platform for the Toshiba TC358840 HDMI2CSI bridge and we want to share our driver development with the community. The driver is based on V4L2 and Video can be captured with a GStreamer pipeline. If you are interested in the driver development, check out our Github Wiki.
EDIT: For new projects, please use our new L4T 28.1 source code at https://github.com/InES-HPMM/linux-l4t-4.4 This version works on both TX1 and TX2 and uses newer drivers that are better integrated with Nvidia and are well tested.
And the Wiki: https://github.com/InES-HPMM/linux-l4t-4.4/wiki/hdmi2csi
The deprecated 24.2.1 (and previous) L4T branches are still available at: https://github.com/InES-HPMM/linux-l4t/wiki/hdmi2csi

The hardware module is available for sale (in small quantities) at https://blog.zhaw.ch/high-performance/2016/05/20/4k-hdmi-to-csi-interface-for-tx1-evalboard/

Our hardware offers:

  • one 4K HDMI input, which is connected to 2 x 4 CSI lanes
  • one 1080p input connected to 1 x 4 CSI lanes
  • HDMI 1.4b
  • Ability to capture Audio over HDMI (in development)

The current state of the driver allows capturing and displaying 2160p30 on the basis of the Linux4Tegra 24.1 release.

Regards
Tobias

Do you have an updated driver for Linux4Tegra R24.2? The V4L2 kernel stack has been modified a lot between R24.1 and R24.2.

Hi, we intend to port our drivers to 24.2 in the coming weeks. Depending on how many of the interfaces that we use have changed, it might take a little more or less time.

Interestingly the driver for the HDMI-to-CSI bridge TC358840 that we originally ported has now been added to L4T 24.2 by Nvidia [1].
Since their integration into L4T differs from ours, we will have to see how to merge the two branches. Of course it would be best to stay as close as possible to the Nvidia implementation, but we also need to achieve the required performance for 2160p30 capturing. I am not sure if their VI driver is capable of capturing concurrently on 2 x 4 CSI-lanes.

We will inform you in this thread and on our Github Wiki as soon as the port to L4T 24.2 is ready. Thanks for the patience.

[1]
http://nv-tegra.nvidia.com/gitweb/?p=linux-3.10.git&a=search&h=33ba00911e9a72a742647030e76b4e355adc9acf&st=commit&s=tc358840

Kamm, thanks for the reply. We have purchased 2 pieces of your HDMI2CSI adaptors a few weeks ago. And I’m trying to do tc358840 driver porting for R24.2, by referencing your R24.1 implementation.

https://github.com/jkjung-avt/l4t-r24-2-tc358840

Keep me updated. We might get this done more quickly together.

Hi Jkjung,

We are working on the same thing here (www.lp-research.com). We are not very far at all yet.
I just cloned your repo to see how it works for us. We are currently on TX1, but are planning to build a custom carrier board with an FPGA on it later on.
How is your progress so far? Anything that we can help you with to accelerate development?

Thanks,
Klaus

Just yesterday I got 720x480p60 HDMI video input to stream through /dev/video0 successfully. I still need to debug why 1080p60 and other resolutions did not work (might be hardware problem on our board). My latest code is already on GitHub.

Sounds great! Just trying to build your source tree, but having some errors. Does it build without problem for you? Running linaro 5.3.1 64-bit here.

Got it to build here now. Just some configuration issues. Now test…

An initial port of our drivers for the TC358840 to L4T R24.2.1 is now available on our Github:

https://github.com/InES-HPMM/linux-l4t/wiki/hdmi2csi#drivers-for-l4t-r2421

We have successfully tested HDMI capture on HDMI-A with 2160p30 and 1080p60. Please note, that this is a Work-In-Progress and there are currently some issues (e.g. HDMI-B not available). We are expecting to solve these issues in the coming days.

If you would like to contribute to the development, please contact us.

@jkjung Did you make any further progress with using the standard Tegra backend for the HDMI input? We’re quite stuck here with development. Also I noticed that your repo is not up anymore :-(

@trilion99, for the past week I was working on something else. I will likely come back to tc358840 driver development work next week. But I might not be able to share the code on GitHub from this point on.

For now, I think you can reference HPMM’s implementation (kamm’s posting). I was referencing their code too.

To adapt the code over to your own board, you’ll need to first make sure the following are defined correctly in your device tree.

  • I2C bus & slave address of tc358840
  • GPIO reset & interrupt pins of tc358840
  • CSI output ports (TX0 & TX1) of tc358840
  • CSI input ports (tegra vi) of Tegra X1

Hopefully you will be making progress along the way.

Hello.
I try to test by capturing a 2160p30 capable HDMI source and displaying it on a local HDMI display.
But error is diplayed on terminal.
I referred to this site[https://github.com/InES-HPMM/linux-l4t/wiki/l4t_prebuilt_image], and can flash the image to TX1.

HDMI Source: Sony FDR-X3000(3840x2160p60/1920x1080p24)

Please give me some advice and Please let us know if you need anything else.


【Error Message】
Setting pipeline to PAUSED …
Inside NvxLiteH264DecoderLowLatencyInitNvxLiteH264DecoderLowLatencyInit set DPB and MjstreamingInside NvxLiteH265DecoderLowLatencyInitNvxLiteH265DecoderLowLatencyInit set DPB and MjstreamingPipeline is live and does not need PREROLL …
Setting pipeline to PLAYING …
New clock: GstSystemClock
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data flow error.
Additional debug info:
gstbasesrc.c(2865): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming task paused, reason not-negotiated (-4)
Execution ended after 0:00:00.185335050
Setting pipeline to PAUSED …
Setting pipeline to READY …
Setting pipeline to NULL …
Freeing pipeline …

Hi takumi_627

Did you load the tc358840 driver before executing a GStreamer pipeline?

sudo modprobe tc358840

Also what is the GStreamer pipeline you executed?

Please note that I am currently generating a new prebuilt image based on the new L4T 24.2.1 that contains improvements compared to the current image. It should become available in the next few hours.

@kamm

Thank you for replying.

yes, I loaded the tc358840 driver before executing a GStreamer pipeline.

I executed the GStreamer pipeline shown below

gst-launch-1.0 v4l2src ! 'video/x-raw, width=3840, height=2160, framerate=60/1, format=UYVY' ! nvvidconv ! 'video/x-raw(memory:NVMM), width=3840, height=2160, framerate=60/1, format=I420' ! nvoverlaysink sync=false
gst-launch-1.0 v4l2src device=/dev/video0 ! 'video/x-raw, width=3840, height=2160, framerate=60/1, format=UYVY' ! nvvidconv ! 'video/x-raw(memory:NVMM), width=3840, height=2160, framerate=60/1, format=I420' ! nvoverlaysink sync=false

I am looking forward to becoming available a new prebuilt image.

The maximum resolution we can capture is currently 3840x2160p30. We are looking to also enable 4096x2160p24, but this is not implemented yet.
Higher rates than that (e.g. 3840x2160p60) are not possible due to limitations of the converter chip (Toshiba TC358840) and the HDMI 1.4b standard.

Another thing to consider is that the HDMI2CSI board acts as HDMI sink (i.e. like a monitor). This means that as per the HDMI standard it provides an EDID which is used to negotiate the best format with the HDMI source. We have enabled some of the most used formats (e.g. 2160p30, 1080p60) in the EDID. If you need a special resolution, I can modify the EDID and send you the modified file via email.

If you can not capture HDMI with the following pipeline (for 2160p30), please send me an email to kamm@zhaw.ch, so we can debug the problem.

gst-launch-1.0 v4l2src ! 'video/x-raw, width=3840, height=2160, framerate=30/1, format=UYVY' ! nvvidconv ! 'video/x-raw(memory:NVMM), width=3840, height=2160, framerate=30/1, format=I420' ! nvoverlaysink sync=false

A new prebuilt image for L4T 24.2.1 is now available at our Github wiki:

Note that additionally to flashing the APP partition it is now also necessary to flash the EBT partition (and thus replace U-Boot).

Hi kamm,

I flashed the pre built image (both APP and EBT), but the board does not boot up after that. The LED at S3, S4 are on, but display, network interface, mouse/keyboard etc are not up.

However if I flash only APP, the board boots up and works without issues, except that the capture card will not function.

Can you please help me resolve this?

Hi zeitgeist

I found a small copy/paste mistake in the command for flashing the EBT partition. This caused the APP partition to be overwritten instead of the EBT partition.

The corrected commands are now available at [1]. Sorry for the trouble…

[1]

Hi kamm,

Thanks for the quick reply. I will try this.

Hi kamm,

With what all 4K media players has the HDMI capture card tested? Can you please let me know the model numbers of these players?