Hello, we are using Orin NX (bsp version 35.5.0 ) with monochrome sensor. We can capture video with the gstreamer plugin “nvarguscamerasrc” exactly one time. Subsequent attempts do not work, a message appears in the kernel log - “[RCE] ERROR: camera-ip/vi5/vi5.c:745 [vi5_handle_eof] “General error queue is out of sync with frame queue. ts=3050756518688 sof_ts=3050822999008 gerror_code=2 gerror_data=400060 notify_bits=20000”. Only device reboot helps.
Capturing with standard v4l2 command works as expected:
“v4l2-ctl -d 0 -set-ctrl=bypass_mode=0 --stream-mmap --verbose”
Same bayer sensor works without these issues with gstreamer plugin and v4l2-ctl command.
Both sensors use the same driver (b/w or Bayer 12-bit data). The file camera_overrides.isp is configured to produce a monochrome image, as indicated in this forum as well.
Furthermore, the histogram of the image is shifted to the right (there is no black color, the black_level is set to a minimum value- 0, otherwise, it becomes even “grayer”). Is this related to the ISP operation with the camera_overrides.isp config? Can the contrast of the image be improved through the Arsgus interface?
hello sy.abramov,
we may need to check error logs for details,
could you please setup a serial console to obtain instant UART logs.
or.. you may running with $ dmesg --follow to keep gathering the logs.
In the kernel logs, there is only one message that I mentioned earlier. Just in case, I am attaching 3 logs - normal operation (exactly once after the system reboot), desynchronization error, and normal operation through the v4l2-ctl utility. I will repeat, the same driver with the same dtb file, but with a Bayer sensor works without errors.
The contrast of the image on the black-and-white sensor is low (the histogram starts from values of 64). When capturing a raw frame (16-bit or 8-bit after conversion), the contrast is normal, and the black color in the image is normal.
v4l2_kernel_normal_log.txt (985 Bytes)
gstreamer_kernel_log_error.txt (1.6 KB)
gstreamer_kernel_log_normal.txt (4.4 KB)
hello sy.abramov,
may I have more details, what’s the difference between them?
The gmax3405 sensors are monochrome and Bayer, the control is the same, the only difference is the otp - data programmed into the sensors during production. Accordingly, the driver and dts are the same. Only for the monochrome sensor, the file camera_overrides.isp has been modified to obtain a black and white image. The command for video capture via gstreamer is trivial:
gst-launch-1.0 nvarguscamerasrc sensor_mode=1 silent=false ! fakesink
This command works one time only after reboot. Subsequent launches of gstreamer result in an error in the kernel log.
hello sy.abramov,
please share the gmax3405/monochrome device tree, and also modified camera_overrides.isp for checking.
BTW, may I know which Jetpack release version you’re working with?
Hello, sorry for the delay, we are using bsp 35.5.0 (which corresponds to version 5.1.3 jetpack). The contrast issue on the color sensor is the same (the histograms of all colors and brightness start from a value of 64). What exactly does the ISP do when receiving 12-bit data from the sensor (I remember reading somewhere that it shifts the data to 16 bits). At least the following command outputs 16 bits per pixel:
v4l2-ctl -d 0 —all
It’s output in the screenshot, attached to this message. I also attach the text of the dtsi file for the sensor.
Here is the content of the file camera_overrides.isp, these values are taken from this forum.
demosaic.v5.family.use=CCCC;
defaults.yuvMatrix[0] = {0.3333, 0.3333, 0.3333, 0};
defaults.yuvMatrix[1] = {0, 0, 0, 0};
defaults.yuvMatrix[2] = {0, 0, 0, 0};
defaults.yuvMatrix[3] = {0, 0, 0, 1};
Histograms for color and monochrome images are built based on the video obtained by running the nvarguscamerasrc plugin with default settings.
gmax3405.dtsi.txt (22.0 KB)
hello sy.abramov,
there’re something wrong,
it looks you’ve two camera at CSI-A and CSI-C for 4-lane configuration.
vi_port0: port@0 {
reg = <0>;
iv_gmax3405_vi_in0: endpoint {
port-index = <0>;
bus-width = <4>;
remote-endpoint = <&iv_gmax3405_csi_out0>;
};
};
vi_port1: port@1 {
reg = <1>;
iv_gmax3405_vi_in1: endpoint {
port-index = <2>;
bus-width = <4>;
remote-endpoint = <&iv_gmax3405_csi_out1>;
};
however, it doesn’t match with the settings from sensor side,
it should be bus-width = <4>; for the 4-lane settings.
for instance,
gmax3405_cam0: iv_gmax3405_a@10 {
compatible = "gpixel,gmax3405";
ports {
iv_gmax3405_out0: endpoint {
port-index = <0>;
bus-width = <2>; <== it should be <4>.
remote-endpoint = <&iv_gmax3405_csi_in0>;
on the other hand,
the tegra_sinterface looks incorrect for your iv_gmax3405_c@10 camera node.
please revise the camera node settings accordingly.
gmax3405_cam1: iv_gmax3405_c@10 {
compatible = "gpixel,gmax3405";
num_lanes = "4";
tegra_sinterface = "serial_a"; <== should be serial_c
ports {
port@0 {
iv_gmax3405_out1: endpoint {
status = "okay";
port-index = <2>;
bus-width = <2>; <== should be <4>.
remote-endpoint = <&iv_gmax3405_csi_in1>;
Okay, there may be inaccuracies with these parameters (post-index, bus-width) in the dts file. But with this file, the color sensor works fine and it definitely cannot affect the histogram. In reality, one sensor is connected via 4 mipi wires, and there are no messages in the kernel log during boot, which means the configuration is recognized correctly..
Here is output from command media-ctl -d 0 -p
There is no another sensor in the output (it’s actually not connectd).
hello sy.abramov,
there’s only single camera but you’re having two in the device tree definition?
please modify the DT to have identical setup, and, please share the snapshot of your abnormal capture results.
Ok, I’ve changed dts file, there is only one entity for nvcsi controller node
media-ctl -d 0 -p
But nothing else has changed. Unfortunately i have no monochrome sensor available right now to check the stability of operation via gstreamer (color sensor works normally).
The brightness histogram is shown in the screenshot - it is clearly shifted to the right (starting after 60).
The same image on a monochrome sensor, why does the ISP output such a range of values?
gmax3405.dtsi.txt (14.1 KB)
hello sy.abramov,
it looks like you’ve RTSP stream? if yes.. it looks like the issue that stream has clamp to 16-235.
you may see-also Argus NvRaw Tool to capture the raw file to have cross check.
Hello, I’ve made some screenshots for color sensor.
Exposure value - 2.5 ms
nvargus_nvraw --c 0 --mode 1 --skipframes 10 --format “raw, jpg” --file nv_bayer16_2448x2048
Unfortunately, I can’t attach all the images to the message, so all 6 frames are included in the archive.
Exposure value - 10 ms (without gain)
nvargus_nvraw --c 0 --mode 1 --skipframes 10 --format “raw, jpg” --file nv_bayer16_2448x2048 --exp0 “0.01, 18.0”
Raw frame pixel values from raw frames exceed 4096, meaning the 12-bit range has been bit-shifted right to 16 bits.
The GStreamer video output matches the JPG frames captured via the nvargus_nvraw utility at exposure times of 2500 and 10000 microseconds.
In other words, GStreamer does not change the color range of the frame in the stream in any way.
Our application produces the same color results.
gmax3405_screenshots.zip (3.7 MB)
hello sy.abramov,
please refer to Orin TRM for [2.4.5 RAW Memory Formats]. the data is in MSBs and LSBs are replicated from MSBs.
Orin series were using T_R16 to handle Raw10, it is a 16-bit unsigned word format, packing two per 32-bit word, there’re some MSBs replicated.
for instance,
RAW12, data is in bits-15:4 and bits-15:12 are replicated in bits-3:0.
that’s something like this packing formats… d11 d10 d9 d8 d7 d6 d5 d4 d3 d2 d1 d0 x x x x
Hi, I changed the value of black_level to 0. (For some reason, it was set to 128 in the driver). The picture became more contrasted, but it is still darker through Argus (now the pixel values in the ‘raw’ frames are around 0). The difference in the histogram is especially noticeable with the aperture closed. In the archive, there are 4 photos with an exposure of 5ms and 200µs with the aperture closed. Why is there still a difference in brightness (the exposure and gain parameters are the same)?
Perhaps agrus (or the nvarguscamerasrc plugin) changes the contrast somehow, is the black_level parameter adjusted through the ISP or something else that could affect this?
new_screenshots.zip (2.1 MB)
hello sy.abramov,
it’s expected with different exposure settings.
FYI, it’s AE to evaluate the scene lux and then estimate the gain/exp for next frame.
you may configure AE lock if you need to bypass that.
Actually, you’ve misunderstood. The exposure and gain settings are identical for both GStreamer and our app’s video capture. Parameter aelock is set to true, here is gstreamer command:
gst-launch-1.0 nvarguscamerasrc sensor_mode=1 aelock=true silent=false exposuretimerange=“200000 200000” gainrange=“18 18” ispdigitalgainrange=“1 1” ! ‘video/x-raw(memory:NVMM),width=2448, height=2048, framerate=10/1, format=NV12’ ! nvvidconv ! ‘video/x-raw,width=1920, height=1280, format=I420’ ! avenc_mpeg4 bitrate=10000000 ! mpegtsmux ! tcpserversink host=192.168.6.190 port=8080
Encoding doesn’t change the image at all. One of the screenshots shows that with the aperture closed, the histogram is near zero. However, with the same gain (minimum value of 18) and exposure (200µs), the histogram of the frame captured by our app (which uses the same Argus interface) is shifted to the right.
The settings in our app are identical (aelock=true, set programmatically via Argus). In the other two screenshots (at 5ms exposure), the images still differ—the GStreamer one has higher contrast. Could the nvarguscamerasrc plugin be modifying other parameters, or is there something we’ve missed in our app? Perhaps GStreamer adjusts the black level or other ISP parameters (to reiterate, the pixels in raw frames are near zero when the aperture is closed).
Is it possible to check the nvargus-daemon logs to see which ISP settings are applied at the start of capture? We want to compare the parameters used by our application versus the nvarguscamerasrc plugin.
Hi, any ideas why the app’s histogram is right-shifted, as shown in the previous screenshots? Also, how can we check the parameters nvarguscamerasrc passes to nvargus-daemon during initialization?
Is it possible to hardware-shift the image brightness to the left using the Argus interface (perhaps the nvarguscamerasrc plugin is doing exactly that)?
hello sy.abramov,
let’s save the image directly for checking.
you may use jpeg encoder to save the frame.
for instance,
gst-launch-1.0 nvarguscamerasrc sensor_mode=1 aelock=true silent=false exposuretimerange=“200000 200000” gainrange=“18 18” ispdigitalgainrange=“1 1” ! 'video/x-raw(memory:NVMM),width=2448, height=2048, framerate=10/1, format=NV12' ! nvvidconv ! 'video/x-raw,width=1920, height=1280, format=I420' ! jpegenc ! multifilesink location=capture%d.jpeg
Hi! The issue is that the histogram in our application is shifted to the right. However, the histogram of the video captured via gstreamer looks normal — it stays at zero when the aperture is closed. We’d like to understand what might be wrong with our app. Is it possible that the nvarguscamerasrc plugin is affecting the ISP settings? I’ve already shared the screenshots of histograms from gstreamer and our application.
Is there a way to set the black level for the ISP using camera_overrides.isp?
