Hello,
I work with two different image sensors on a orin nx, sony imx565 and sony imx567. For both, images captured using v4l2 contain garbled pixels at the right end of the lines, replacing the true pixels sent by the sensor. The problem does not appear when capturing using e.g. nvargus_raw, but pure v4l2 must work too.
For the imx565 sensor (4128 x 3008), the noise begins at column 4096. For the imx567 (2472 x 2064) it is less clear where noise begins, but it seems to be at column 2416.
What’s tools to check the raw file?
Could you attached the raw file here to check.
Thanks
LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libgomp.so.1 gst-launch-1.0 -v v4l2src device=/dev/video0 num-buffers=1 ! avenc_pgm ! filesink location=/tmp/by-v4l2src-0.pgm
LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libgomp.so.1 gst-launch-1.0 -v v4l2src device=/dev/video1 num-buffers=1 ! avenc_pgm ! filesink location=/tmp/by-v4l2src-1.pgm
cjpeg /tmp/by-v4l2src-0.pgm > /tmp/by-v4l2src-0.jpg
cjpeg /tmp/by-v4l2src-1.pgm > /tmp/by-v4l2src-1.jpg
The sensors and the driver produce 12 bit pixels (MEDIA_BUS_FMT_Y12_1X12), but I need only 8, so I have added the following entry to vi5_video_formats :
TEGRA_VIDEO_FORMAT(RAW12, 12, Y12_1X12, 1, 1, T_R8,
RAW12, GREY, "GRAY8"),
and this entry to camera_common_color_fmts
{
MEDIA_BUS_FMT_Y12_1X12,
V4L2_COLORSPACE_SRGB,
V4L2_PIX_FMT_GREY,
},
by-v4l2src-1.pgm.gz (7.4 KB)
by-v4l2src-0.pgm.gz (14.2 KB)
Please using v4l2-ctl to get the raw image to confirm it.
Thanks
v4l2-ctl --set-fmt-video=width=4128,height=3008 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=1 --stream-to=test.raw -d /dev/video0
Here you are
test.raw.gz (14.1 KB)
Looks like it’s sensor output problem. The raw dump looks like less than 4128 pixels.
Raw dumps of RG12 mode show no problem. All pixels in the raw dump get the value I have put in the pattern generator. But if I ask to get the same image in GREY8 mode using the conversion provided by the VI, the last pixels are replaced by some sort of noise. It is certainly a bug in the RAW12 to RAW8 conversion provided by the vi5 driver (or hardware).
Could you try T_R16 instead of T_R8
I have replaced T_R8 by T_R16 in my 'GREY, “GRAY8” entry in vi5_video_formats, then run
v4l2-ctl --set-fmt-video=width=4128,height=3008,pixelformat=GREY --set-ctrl bypass_mode=0 --stream-mmap --stream-count=1 --stream-to=test-GREY.raw -d /dev/video0
The kernel complains with :
[ 309.255579] arm-smmu 10000000.iommu: Unhandled context fault: fsr=0x80000402, iova=0x7fffbd8000, fsynr=0x50013, cbfrsynra=0x402, cb=0
[ 309.256010] mc-err: vpr base=0:0, size=0, ctrl=1, override:(201803c6, b9ee11c1, 1, 0)
[ 309.256249] mc-err: (255) csw_viw: MC request violates VPR requirements
[ 309.256438] mc-err: status = 0x0ff74072; hi_addr_reg = 0x00000000 addr = 0xffffffff00
[ 309.256675] mc-err: secure: yes, access-type: write
[ 309.289820] arm-smmu 10000000.iommu: Unhandled context fault: fsr=0x80000402, iova=0x7ffebd8000, fsynr=0x50013, cbfrsynra=0xc02, cb=0
[ 309.290206] mc-err: Too many MC errors; throttling prints
[ 309.324061] arm-smmu 10000000.iommu: Unhandled context fault: fsr=0x80000402, iova=0x7ffdbd8000, fsynr=0x50013, cbfrsynra=0x802, cb=0
and I get the following raw image :
nvidia@ubuntu:~$ hexdump test-GREY.raw
0000000 e81e e81e e81e e81e e81e e81e e81e e81e
*
0bd7800
nvidia@ubuntu:~$
‘0xe81’ is the value I have put in the pattern generator.
My GREY image should be full of ‘0xe8’ bytes.
How about default config?
TEGRA_VIDEO_FORMAT(RAW12, 12, SRGGB12_1X12, 2, 1, T_R16,
111 RAW12, SRGGB12, "RGRG.. GBGB.."),
112 TEGRA_VIDEO_FORMAT(RAW12, 12, SGRBG12_1X12, 2, 1, T_R16,
113 RAW12, SGRBG12, "GRGR.. BGBG.."),
114 TEGRA_VIDEO_FORMAT(RAW12, 12, SGBRG12_1X12, 2, 1, T_R16,
115 RAW12, SGBRG12, "GBGB.. RGRG.."),
116 TEGRA_VIDEO_FORMAT(RAW12, 12, SBGGR12_1X12, 2, 1, T_R16,
117 RAW12, SBGGR12, "BGBG.. GRGR.."),
Also reference to the pixel layout.
I understand that converting RAW12 to T_R16 replicates the most significant nibble of the RAW12 value into the least significant nibble of the T_R16 value. That happens above when trying T_R16 for 8 bits GREY, and that happens correctly for the four 12-bit Bayer variants : RG12, BA12, GB12 and BG12 :
nvidia@ubuntu:~$ for f in RG12 BA12 GB12 BG12
do
v4l2-ctl --set-fmt-video=width=4128,height=3008,pixelformat=$f --set-ctrl bypass_mode=0 --stream-mmap --stream-count=1 --stream-to=test-$f.raw -d /dev/video0
hexdump test-$f.raw
done
<
0000000 e81e e81e e81e e81e e81e e81e e81e e81e
*
17af000
<
0000000 e81e e81e e81e e81e e81e e81e e81e e81e
*
17af000
<
0000000 e81e e81e e81e e81e e81e e81e e81e e81e
*
17af000
<
0000000 e81e e81e e81e e81e e81e e81e e81e e81e
*
17af000
nvidia@ubuntu:~$
So the problem only appears when converting RAW12 to T_R8
My test pattern emits RAW12 0xe81 pixels. They are correctly transformed into T_R8 0xe8 pixels by throwing away the LSB’s (the four bits of the nibble with value 1) for the biggest part of the image
v4l2-ctl --set-fmt-video=width=4128,height=3008,pixelformat=GREY --set-ctrl bypass_mode=0 --stream-mmap --stream-count=1 --stream-to=test-GREY.raw -d /dev/video0
<
hexdump -C test-GREY.raw
00000000 e8 e8 e8 e8 e8 e8 e8 e8 e8 e8 e8 e8 e8 e8 e8 e8 |................|
*
00002020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00002040 e8 e8 e8 e8 e8 e8 e8 e8 e8 e8 e8 e8 e8 e8 e8 e8 |................|
*
00004060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00004080 e8 e8 e8 e8 e8 e8 e8 e8 e8 e8 e8 e8 e8 e8 e8 e8 |................|
*
000060a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000060c0 e8 e8 e8 e8 e8 e8 e8 e8 e8 e8 e8 e8 e8 e8 e8 e8 |................|
*
000080e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00008100 e8 e8 e8 e8 e8 e8 e8 e8 e8 e8 e8 e8 e8 e8 e8 e8 |................|
*
Knowing that 4128 is 1020 hex, one sees that the first line (from 0 to 1020) is correctly transformed, but in the second line the pixels between 2020 and 2040 are replaced by 0 values. The same thing happens for every pair of following lines : the first one is correct (e.g. from 2040 to 3060), but the next one is corrupted at the end (3060 to 4060 is correct, 4060 to 4080 is corrupted)
Hello Shane,
do you need more info ?
have you got support from second line ?
Meanwhile, some more info for what I believe is the same problem.
Images below are captured using the following command :
gst-launch-1.0 -v v4l2src device=${device} num-buffers=1 ! video/x-raw,format=GRAY8 ! jpegenc ! filesink location=test.jpg
with a imx567 sensor programmed with the mode 3 of the pattern generator.
One image (the good one) is obtained on a jetson nano running jetpack-4.6.
The other image (the bad one) is obtained on a orin nx running jetpack-5.1.1.
For comparison and proof that the sensor and the physical transmission are good on the orin, I have added an image created by the following command :
nvargus_nvraw --format jpg --c 1 --file orin-nx-jp511-nvargus_raw.jpg
As you can see, corrupted pixels happen even in the middle of the image, in the bright zone.
Note : my images are all 2472x2064. The bad orin one has been scaled down by the upload to the forum.
I don’t have idea. Could it be the lost the d0-d3 cause the problem?
For me it is clearly a bug either in the vi5 driver (but on the orin-nx we have vi6) or in the hardware itself. Could you report it to your upper level ?
And with the imx567 sensor (line width = 2472, not 4128), the problem arises also with the simple conversion of my solid 0xe81 image to T_R16, but is more difficult to describe. Maybe the distance between a power of 2 and the width of the line amplifies the problem. 4128 = 4096 + 32, but 2472 = 2048 + 424. Do you need examples of garbled RAW12 to T_R16 conversion with line width = 2472 ?
Please conclude the problem and give much detail for the issue for involve developer to check.
Thanks
Hello Shane,
I need to process on a Orin NX with Jetpack-5.1.1 in GREY8 form, images coming from RAW12 sensors. Therefore my driver produces always MEDIA_BUS_FMT_Y12_1X12, and I have added the following entriy to vi5_video_formats
TEGRA_VIDEO_FORMAT(RAW12, 12, Y12_1X12, 1, 1, T_R8,
RAW12, GREY, "GRAY8"),
and
{
MEDIA_BUS_FMT_Y12_1X12,
V4L2_COLORSPACE_SRGB,
V4L2_PIX_FMT_GREY,
},
to camera_common_color_fmts. This works perfectly with TX1’s TX2’s and TX1 Nano.
But, from what I’ve seen, with the ORIN NX and jetpack-5.1.1, there are bugs in the conversion from MIPI RAW12 to Nvidia T_R8 and T_R16 in memory formats. T_R16 format was added after your suggestion. The bug has different results depending on the T_R8 or T_R16 conversion and depends also on the width of the image. The bug seems to be a replacement of block of pixels by pixels with value 0x00 (black). This happens often at end of lines, but also in the lines in the images with the vertical columns (103).
Evidence images are provided. These are images coming from two different sensors (IMX565 4128x3008 and IMX567 2472x2064), with 3 different patterns generated by the sensors, and processed by 3 different capture commands. Here are the capture commands :
NAME=${BASENAME}-GREY8
v4l2-ctl --set-fmt-video=width=${W},height=${H},pixelformat=GREY --set-ctrl bypass_mode=0 --stream-mmap --stream-count=1 -d${d} --stream-to=- | \
rawtopgm -bpp 1 -maxval 255 -littleendian ${W} ${H} > /tmp/${NAME}.pgm
pnmtojpeg /tmp/${NAME}.pgm > /tmp/${NAME}.jpg
NAME=${BASENAME}-GREY16
v4l2-ctl --set-fmt-video=width=${W},height=${H},pixelformat='Y16 ' --set-ctrl bypass_mode=0 --stream-mmap --stream-count=1 -d${d} --stream-to=- | \
rawtopgm -bpp 2 -maxval 65535 -littleendian ${W} ${H} > /tmp/${NAME}.pgm
pnmtojpeg /tmp/${NAME}.pgm > /tmp/${NAME}.jpg
NAME=${BASENAME}-nvargus
nvargus_nvraw --format jpg --c ${d} --file /tmp/${NAME}.jpg
The argus images are provided to show that the image coming from the sensor is not garbled. The sensor is programmed identically in all cases. The intermediate .pgm files are also provided for closer inspection.
I would be happy to test patches that you would provide.
pgms.tar.gz (427.7 KB)
Hello Shane,
have you transmitted my last post to the developers ?
do they need more information ?
best regards
Yes, I am waiting response too.
Thanks
