How to specify RAW format in device tree

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

I have a camera streaming images in RAW10 format. It works when capturing images via v4l2-ctl, but there’s no way to make it work with gst-launch-1.0 v4l2src. This is how it looks

nvidia@nvidia-desktop:~/Pictures$ v4l2-ctl -d /dev/video0 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
	Index       : 0
	Type        : Video Capture
	Pixel Format: 'RG10'
	Name        : 10-bit Bayer RGRG/GBGB
		Size: Discrete 1920x1080
			Interval: Discrete 0.020s (50.000 fps)

This anyway comes from the device tree, where I’ve set up:

mode_type = "bayer";
pixel_phase = "rggb";

I’ve set it up like this because I see no way of specifying it is RAW data to the device tree. The Sensor Driver Programming Guide details that the possible values for mode_type are
•yuv
•bayer
•bayer_wdr_pwl
I guess the single image capturing is working due to the ISP bypassing. Should I do something similar with the video streaming pipeline? If so, how? Or do I need to modify the device tree?

2

This is not an answer to your question, but gstreamer v4l2src may only support byte aligned formats.
You may have a look to this post for trying offline conversion (not sure this was correct according to TRM for Msb, you may check further if trying it).

3

Thanks for the answer. It seems interesting what you point out. Anyway, I’ve tried what you suggest in the forwarded post, and I’m still not able to stream it. I’ve successfully reached last step, but yet the gst pipeline gives the following error

nvidia@nvidia-desktop:~/Pictures$ gst-launch-1.0 filesrc location=img.bggr blocksize=2073600 ! 'video/x-bayer, width=1920, height=1080, framerate=25/1, format=bggr' ! bayer2rgb ! videoconvert
Setting pipeline to PAUSED ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
ERROR: from element /GstPipeline:pipeline0/GstFileSrc:filesrc0: Internal data stream error.
Additional debug info:
gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstFileSrc:filesrc0:
streaming stopped, reason not-linked (-1)
Execution ended after 0:00:00.028811861
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

Anyway I’ll take a look to the byte alignment fact that could be related to the problems I’m having.

4

Moreover, the pixel is monochrome too (grayscale), with 10bpp.

5

I’ve seen in your other topic you’ve moved on with your dt and have now driver reporting Y10 format as expected.

So you may first try to record 100 frames with v4l2-ctl as Y10 and save to file:

v4l2-ctl --set-fmt-video=width=1920,height=1080,pixelformat=Y10 --stream-mmap -d /dev/video0 --set-ctrl bypass_mode=0 --stream-count=100 --stream-to=test.Y10

Note that this may result in a huge size, so cd to an external disk before trying this.

This would be saved into 16 bits per pixel (1920 * 1080 * 2 = 4147200 bytes per frame).
So you would just try :

gst-launch-1.0 filesrc location=test.Y10 blocksize=4147200 ! video/x-raw,format=GRAY16_LE, width=1920, height=1080, framerate=30/1 ! videoconvert ! xvimagesink

If this works, you would just have to modify the driver for providing Y16 if you want to use it in gstreamer from v4l2src plugin.

6

Hi Honey_Patouceul,
It worked pretty good. Thanks for the answer!
I have some points yet. The image I get is darker than the one I get by processing the raw file. Does this make sense? Could it be related to the Y10 format?


(although the image seems wrong, it is OK. The black part of the bottom is due to the repeating pattern, as 1024 different lines are being sent and the height is 1080)

Moreover, I’m sending the data in RAW10 format. As far as I see in the TRM, RAW10 corresponds to 32 bpp. This would be inconsistent with the “GRAY16_LE” format right?

Moreover, regarding the Y16 format, I cannot see it in the soc_mediabus file. Where should it be located?

7

Sorry, I have no Y10 sensor and no experience in sensor porting, however my guess would be that you are zooming into a corner because the blocksize or width and height are wrong.

Try recording one frame only and check size:

v4l2-ctl --set-fmt-video=width=1920,height=1080,pixelformat=Y10 --stream-mmap -d /dev/video0 --set-ctrl bypass_mode=0 --stream-count=1 --stream-to=test.Y10
ls -l test.Y10

and use this as blocksize for filesrc when playing a longer record.

I would expect 192010802 = 4147200 for Y10 saved into 16 bits, but again not sure.

If this is ok, then it might be a color scaling issue because in Y16 white would be 0xff, while in 10 bits it is 0x3f only.

Maybe using a YUV format with 0-sized U and V would be an option. Someone with better skills in sensor porting would better advise.

8

Another thought: when you process the image are you taking into account the pixel packing? I see that you’ve referenced the TRM so you have probably noticed that some formats are not packed in the lower bits. The relevant information is on page 2794 of the TRM.

For raw10:

bit:  15 14 13 12 11 10 09 08 07 06 05 04 03 02 01 00
data:  0  0 d9 d8 d7 d6 d5 d4 d3 d2 d1 d0 d9 d8 d7 d6

Note that the 4 most significant bits are replicated in the 4 least significant bit positions.

This is for T_R16_I and you specifically mentioned T_R32 however.

9

I did the test you suggested, and it is showing 4147200, so I guess the blocksize is well set.

Regarding the Y16 format, any tip for its implementation? Because I cannot see it in the soc_mediabus (not like Y10 at least). Does it have another name?

Lastly, for the scaling theory, I’ve tested to take an equal picture in RAW and in Y10, and this is the result.
RAW

v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat=GRAY16_LE --set-ctrl bypass_mode=0 --stream-mmap --stream-count=1 --stream-to=img_005.raw

Processed with ImageJ, 16-bit unsigned LE

Y10

v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat=Y10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=200 --stream-to=test200.y10

It does not seem to be scaled, just darker, so maybe you are right in the 0xff for Y16 and 0x3f for Y10 fact for the white.

10

I had in mind that, although I was not taking it much into consideration. I’ve seen that they are not packed in the lower bits, but I thought the TX2 is taking that into consideration when correctly specifying the format.

Does the TX2 not consider it? So if it doesn’t, how could I indicate the packet formatting for the gst framework?

Moreover, I’m using RAW10 with 10-bit depth grayscale pixels. As far as I know, this would correspond to 16 bits, and indeed, treating it as 16 bits seems to be working (I’ve declared it as T_R16_I in the vi5_formats when giving support to grayscale). Anyway, I indicated what I saw in the TRM as it makes no sense for me. Why does it indicate that RAW10 corresponds to 32 bpp?

11

If the frames are processed through the ISP then you don’t have to worry about it. If you bypass the ISP using v4l2 you need to account for the pixel packing.

You could write a GStreamer plugin to do the conversion. Or you could create a GStreamer app sink. If you went the plugin route (as opposed to the app sink route) you would put it early in your pipeline and then the rest of your application shouldn’t care.

I agree that the TRM is confusing and I’ve largely ignored that particular table because most sensors I’ve used are raw12 bayer sensors. D3 has a monochrome sensor we are supporting now so I am more interested in these details now. I just haven’t had a need to understand the processing of monochrome sensors on Jetson to much degree yet.

Regards,
Greg

12

This is really interesting. I’ll probably need to create an app sink for that, I didn’t even consider that as a potential problem, but now I can see it is much more important than what I though. Thanks for the answer.
P.D. I’ll accept Honey’s answer for this post as it is more related to the original topic. Anyway your answer helped me pretty much too in understanding the issue I might be having

13

I’ve given support to Y16 and I’m able to stream it with the qV4L2 tool (although a bit darker, but probably due to bit allignment). Thanks for the tips!