TX2 CSI monochrome sensor 14bit video data From FPGA

liliasojd · July 26, 2020, 1:44am

I want to use FPGA to send 14-bit infrared image to TX2 through CSI, and now I have some problems:

I am using SBGGR10, and now I can use V4l2-ctrl to obtain images correctly, but I cannot find the device in Gstreamer mode.
What should I do if I want to get a 14-bit image? I saw in camera_common.c and sensor_common.c that there are only 10 and 12 bits, and 14 bits are not supported.
Since I am transmitting a monochrome image, it seems that I cannot pass the ISP and can only operate it directly through V4l. Does this mean I can’t use Gstreamer? I want to process the image into an 8-bit image and send it over the network through the RTPS protocol. , How to operate?
All in all, I hope that TX2 can obtain a 14-bit monochrome image through the CSI interface, process it into an 8-bit image, and then transmit it on the network through the RTPS protocol. Is there any way?

nvidia@tegra-ubuntu:~/v4l-utils-1.12.6/utils/v4l2-ctl$ ./v4l2-ctl -d /dev/video3 --set-fmt-video=width=1280,height=720,pixelformat=BG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=100
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 40.59 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 40.50 fps
<<<<<<<<<<<<<<<<<<

nvidia@tegra-ubuntu:~/v4l-utils-1.12.6/utils/v4l2-ctl$ nvgstcapture --video-res=3
…
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute:364 No cameras available
(Argus) Error EndOfFile: Unexpected error in reading socket (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 214)
(Argus) Error EndOfFile: Receive worker failure, notifying 1 waiting threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 289)
(Argus) Error InvalidState: Argus client is exiting with 1 outstanding client threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 306)
(Argus) Error EndOfFile: Receiving thread terminated with error (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadWrapper(), line 317)
(Argus) Error EndOfFile: Client thread received an error from socket (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 149)
(Argus) Error EndOfFile: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 101)

Blockquote

ShaneCCC · July 26, 2020, 2:17pm

Enable the daemon log to get more information.
Also check the dt of the tegra_camera_platform{} check those information is correct by v4l2-ctl --list-devices.

sudo su
kill the process of nvargus-daemon
export enableCamPclLogs=5
export enableCamScfLogs=5
/usr/sbin/nvargus-daemon
lanuch camera APP from another console

enrique.ramirez · July 29, 2020, 8:21pm

Hi @liliasojd,

The v4l2src GStreamer plugin does not have support for SBGGR10 (or any bayer 10-bit format), but you can download GStreamer source code, patch the element, build and re-install GStreamer, here is a reference: GStreamer RAW10 patch.
The TX2 capture subsystem supports RAW6, RAW7, RAW8, RAW10, RAW12, RAW14. So, you should be able to capture 14-bit image. And yes, you’re right, tegra files do not support this, but you extend format the support modifying the following files:

sources/kernel/nvidia/drivers/media/platform/tegra/camera/sensor_common.c

static int extract_pixel_format(
	const char *pixel_t, u32 *format)
{
   .
   .
	else if (strncmp(pixel_t, "bayer_bggr14", size) == 0)
		*format = V4L2_PIX_FMT_SBGGR14;
   .
   .
};

sources/kernel/nvidia/drivers/media/platform/tegra/camera/camera_common.c

static const struct camera_common_colorfmt camera_common_color_fmts[] = {
    .
    .
	{
		MEDIA_BUS_FMT_SBGGR14_1X14,
		V4L2_COLORSPACE_SRGB,
		V4L2_PIX_FMT_SBGGR14,
	},
    .
    .
};

sources/kernel/nvidia/drivers/media/platform/tegra/camera/vi/vi4_formats.h

static const struct tegra_video_format vi4_video_formats[] = {
    .
    .
	TEGRA_VIDEO_FORMAT(RAW14, 14, SBGGR14_1X14, 2, 1, T_R16_I,
				RAW14, SBGGR14, "BGBG.. GRGR.."),
    .
    .
};

The ISP only supports bayer formats as inputs. If you are using SBGGR10 or SBGGR14 you should be able to capture through the ISP using GStreamer Nvarguscamerasrc plugin (the ISP will convert the bayer data into NV12). However, if you are transmitting monochrome using bayer formats (SBGGR10 or SBGGR14), you’ll probably see differences in colors but you should be able to capture the images if everything is correctly configured in the device-tree specially the “tegra-camera-platform node”.
If you want the RAW data (bypassing the ISP), you can also capture using v4l2src (as I mentioned in 1.).

liliasojd · July 31, 2020, 7:07am

I tried the RAW14 method you mentioned, but an error was reported when compiling the kernel. I used the 28R2_1 version.

GZIP kernel/config_data.gz
CHK kernel/config_data.h
DTC drivers/misc/mods/mods.dtb
DTB drivers/misc/mods/mods.dtb.S
AS [M] drivers/misc/mods/mods.dtb.o
LD [M] drivers/misc/mods/mods.o
rm drivers/misc/mods/mods.dtb.S drivers/misc/mods/mods.dtb
CC drivers/media/platform/tegra/camera/vi/vi4_fops.o
In file included from /home/lili/TX2_28R1_ALL/public_release/kernel/kernel-4.4/drivers/media/platform/tegra/camera/vi/mc_common.h:35:0,
from /home/lili/TX2_28R1_ALL/public_release/kernel/kernel-4.4/…/t18x/drivers/video/tegra/host/vi/vi4.h:24,
from /home/lili/TX2_28R1_ALL/public_release/kernel/kernel-4.4/drivers/media/platform/tegra/camera/vi/vi4_fops.c:19:
/home/lili/TX2_28R1_ALL/public_release/kernel/kernel-4.4/drivers/media/platform/tegra/camera/vi/core.h:113:2: error: ‘MEDIA_BUS_FMT_SBGGR14_1X14’ undeclared here (not in a function)
MEDIA_BUS_FMT_##MBUS_CODE,
^
/home/lili/TX2_28R1_ALL/public_release/kernel/kernel-4.4/drivers/media/platform/tegra/camera/vi/vi4_formats.h:120:2: note: in expansion of macro ‘TEGRA_VIDEO_FORMAT’
TEGRA_VIDEO_FORMAT(RAW14, 14, SBGGR14_1X14, 2, 1, T_R16_I,
^
/home/lili/TX2_28R1_ALL/public_release/kernel/kernel-4.4/drivers/media/platform/tegra/camera/vi/core.h:117:2: error: ‘V4L2_PIX_FMT_SBGGR14’ undeclared here (not in a function)
V4L2_PIX_FMT_##FOURCC,
^
/home/lili/TX2_28R1_ALL/public_release/kernel/kernel-4.4/drivers/media/platform/tegra/camera/vi/vi4_formats.h:120:2: note: in expansion of macro ‘TEGRA_VIDEO_FORMAT’
TEGRA_VIDEO_FORMAT(RAW14, 14, SBGGR14_1X14, 2, 1, T_R16_I,
^
/home/lili/TX2_28R1_ALL/public_release/kernel/kernel-4.4/scripts/Makefile.build:261: recipe for target ‘drivers/media/platform/tegra/camera/vi/vi4_fops.o’ failed
make[7]: *** [drivers/media/platform/tegra/camera/vi/vi4_fops.o] Error 1
/home/lili/TX2_28R1_ALL/public_release/kernel/kernel-4.4/scripts/Makefile.build:406: recipe for target ‘drivers/media/platform/tegra/camera/vi’ failed
make[6]: *** [drivers/media/platform/tegra/camera/vi] Error 2
/home/lili/TX2_28R1_ALL/public_release/kernel/kernel-4.4/scripts/Makefile.build:406: recipe for target ‘drivers/media/platform/tegra/camera’ failed
make[5]: *** [drivers/media/platform/tegra/camera] Error 2
/home/lili/TX2_28R1_ALL/public_release/kernel/kernel-4.4/scripts/Makefile.build:406: recipe for target ‘drivers/media/platform/tegra’ failed
make[4]: *** [drivers/media/platform/tegra] Error 2
/home/lili/TX2_28R1_ALL/public_release/kernel/kernel-4.4/scripts/Makefile.build:406: recipe for target ‘drivers/media/platform’ failed
make[3]: *** [drivers/media/platform] Error 2
/home/lili/TX2_28R1_ALL/public_release/kernel/kernel-4.4/scripts/Makefile.build:406: recipe for target ‘drivers/media’ failed
make[2]: *** [drivers/media] Error 2
make[2]: *** Waiting for unfinished jobs…
/home/lili/TX2_28R1_ALL/public_release/kernel/kernel-4.4/Makefile:973: recipe for target ‘drivers’ failed
make[1]: *** [drivers] Error 2
make[1]: Leaving directory ‘/home/lili/TX2_28R1_ALL/kernel28r2_1_out’
Makefile:150: recipe for target ‘sub-make’ failed
make: *** [sub-make] Error 2

enrique.ramirez · July 31, 2020, 2:07pm

Hi @liliasojd,

Try append the following modification to the RAW14 support:

sources/kernel/kernel-4.4/include/uapi/linux/videodev2.h

#define V4L2_PIX_FMT_SBGGR14 v4l2_fourcc('B', 'G', '1', '4') /* 14 BGBG.. GRGR.. */

liliasojd · July 31, 2020, 2:18pm

V4L2_PIX_FMT_SBGGR14 undefined problem is solved, but MEDIA_BUS_FMT_SBGGR14_1X14 is still undefined

enrique.ramirez · July 31, 2020, 2:24pm

That definition is missing on 28R2_1 (Kernel-4.4), the latest versions already contain this support.

You can add it for your version:

sources/kernel/kernel-4.4/include/uapi/linux/media-bus-format.h

#define MEDIA_BUS_FMT_SBGGR14_1X14 0x3019

liliasojd · July 31, 2020, 3:37pm

Thank you very much. At least the compilation is passed. Tomorrow to check if the data is correct. I have one more question. Does the CSI of TX2 only support certain resolutions? I tried it, such as 1920X1080, 1280X720, 1024X768, but there is no problem, but for example 800X600, 720X576, the file size of the image captured using v4l-ctrl is wrong.

enrique.ramirez · July 31, 2020, 4:34pm

@liliasojd,

There’s a bytes-per-line limitation on the TX2. For resolutions where the bytes-per-line value is not multiple of TEGRA_STRIDE_ALIGNMENT (256 on 28R2_1), a padding is added to the image.

So, for example:

Resolution: 800x600
Bytes-per-line: 800 x 2 (Using RAW10 or RAW14)

800 x 2 / 256 = 6,25    ---> not a multiple of 256

Fixed bytes-per-line: 7 x 256 = 1792
Fixed width = 1792 / 2 = 896

A stride of 96 empty pixels is added to the width in this case.

TEGRA_STRIDE_ALIGNMENT value is defined in:
sources/kernel/kernel-4.4/drivers/media/platform/tegra/camera/vi/core.h

You can modify this value, but if a value lower than 64 is set, you’ll probably see a corrupted frame.