Problems with imx412 (auto)exposure when using nvarguscamerasrc

Trying the pipeline from the topic More problems with nvarguscamerasrc trying 10bit, with a imx412 camera, and streaming rtp to a vlc player.

The video is hugely overexposed, and there seems to be no autoexposure or any way to change the exposure, either by using the exposuretimerange property from nvarguscamerasrc, or by issuing v4l2-ctl -c exposure=xxx commands after the stream is running.

Funnily enough, if I set aelock=true, then the video seems hugely underexposed, as if the autoexposure stops just at the beginning.

The pipeline is:

gst-launch-1.0 -v -e nvarguscamerasrc sensor-id=0 sensor-mode=3 timeout=200 ! 'video/x-raw(memory:NVMM),format=(string)P010_10LE,framerate=(fraction)30/1' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=(string)I420_10LE' ! omxh265enc insert-vui=true insert-aud=true ! h265parse ! 'video/x-h265, stream-format=(string)byte-stream, framerate=30/1' ! queue ! mpegtsmux ! rtpmp2tpay ! udpsink host=192.168.55.100 port=5000

On the other hand, if I use the camera_recording app from jetson_multimedia_api samples as the first part of the pipeline, then the autoexposure works and the video is fine:

mkfifo fifo.h265
/usr/src/jetson_multimedia_api/samples/10_camera_recording/camera_recording -r 1920x1080 -t H265 -d 3600 -f fifo.h265 & gst-launch-1.0 filesrc location=fifo.h265 do-timestamp=true ! h265parse ! 'video/x-h265, stream-format=(string)byte-stream, framerate=30/1' ! queue ! mpegtsmux ! rtpmp2tpay ! udpsink host=192.168.55.100 port=5000

How do I get the gstreamer nvarguscamerasrc to set the exposure or autoexposure right?

Please review the sensor driver and device tree configure for the exposure/gain config.

Thanks

Could you indicate any specific bits that need looking at?

Maybe clarify the problem by gst-launch-1.0 without network maybe encoder to file or check by preview on display.

Let’s try again.

I use streaming to VLC because it is the easiest way to get video out of the hardware… I could get some stills in files if that helps you, but in essence:

  1. Streaming with:
    gst-launch-1.0 -v -e nvarguscamerasrc sensor-id=0 sensor-mode=3 timeout=200 aelock=false ! 'video/x-raw(memory:NVMM),format=(string)P010_10LE,framerate=(fraction)30/1' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=(string)I420_10LE' ! omxh265enc insert-vui=true insert-aud=true ! h265parse ! 'video/x-h265, stream-format=(string)byte-stream, framerate=30/1' ! queue ! mpegtsmux ! rtpmp2tpay ! udpsink host=192.168.55.100 port=5000
    I get the autoexposure overexposing:
  2. Streaming with:
    gst-launch-1.0 -v -e nvarguscamerasrc sensor-id=0 sensor-mode=3 timeout=200 aelock=true ! 'video/x-raw(memory:NVMM),format=(string)P010_10LE,framerate=(fraction)30/1' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=(string)I420_10LE' ! omxh265enc insert-vui=true insert-aud=true ! h265parse ! 'video/x-h265, stream-format=(string)byte-stream, framerate=30/1' ! queue ! mpegtsmux ! rtpmp2tpay ! udpsink host=192.168.55.100 port=5000
    I got the autoexposure underexposing (most likely because it is stopped at the beginning):
  3. Using camera_recording, I can see the autoexposure working fine:

Now let me know what information you need to help me fix the autoexposure when using nvarguscamerasrc.

Please confirm by preivew on HDMI display connect on Xavier.

gst-launch-1.0 nvarguscamerasrc ! "video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080,format=(string)NV12, framerate=(fraction)30/1" ! nvvidconv ! xvimagesink sync=false

Our hardware does not have an HDMI display.

That is why I use streaming.

On the other hand, trying it with streaming, just using the NV12 format instead of 10-bit ones, seems to work and autoexpose properly:
gst-launch-1.0 -v -e nvarguscamerasrc sensor-id=0 sensor-mode=3 timeout=200 aelock=false ! 'video/x-raw(memory:NVMM),format=(string)NV12,framerate=(fraction)30/1' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=(string)NV12' ! omxh265enc insert-vui=true insert-aud=true ! h265parse ! 'video/x-h265, stream-format=(string)byte-stream, framerate=30/1' ! queue ! mpegtsmux ! rtpmp2tpay ! udpsink host=192.168.55.100 port=5000

So why would the 10-bit format make it to overexpose?

Then I tried this, just using NV12 to I420_10LE, and that exposes correctly:
gst-launch-1.0 -v -e nvarguscamerasrc sensor-id=0 sensor-mode=3 timeout=200 ! 'video/x-raw(memory:NVMM),format=(string)NV12,framerate=(fraction)30/1' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=(string)I420_10LE' ! omxh265enc insert-vui=true insert-aud=true ! h265parse ! 'video/x-h265, stream-format=(string)byte-stream, framerate=30/1' ! queue ! mpegtsmux ! rtpmp2tpay ! udpsink host=192.168.55.100 port=5000

But using P010_10LE to NV12 also overexposes.

So it seems that the P010_10LE causes that. Any ideas on a fix? We would like to use 10-bit across the pipeline.

By the way, it also works fine omitting the nvvidconv and using NV12 straight into h265.

You can try download the source code of nvarguscamerasrc to modify to output PIXEL_FMT_YCbCr_444_888 instead of NV12.

Thanks

Is PIXEL_FMT_YCbCr_444_888 the right output? seems that that is for 8 bit instead of 10 bit.

It appears from the sources that PIXEL_FMT_YCbCr_444_888 is the stream pixel format correspondingg to NV12 capability pixel color format, while PIXEL_FMT_P016 corresponds to the 10LE (called NV12_10LE) and those appear to correspond to GST_VIDEO_FORMAT_NV12 or GST_VIDEO_FORMAT_P010_10LE…

Could you provide your suggested patch?

(Just tried changing the pixel color format in the initialization to NV12_10LE, but that does nothing, probably overriden by whatever format is requested in the pipeline)

Hi,
Please get frame data in NV12(YUV420 8-bit) from Argus. P010_10LE(YUV420 10-bit) is experimental and it may not work properly.

Right, In an attempt to make it work, it might be good to disable auto-exposure, and be able to set it manually, as it seems that auto-exposure is all that is wrong with it at the moment.

Is there a way to disable autoexposure and set it manually in nvarguscamerasrc?

I don’t think cause by auto exposure.
However you can use set the exposuretimerange and gainrange for fixed AE for testing.

Thanks

Tried that, but seems that it goes all the way up no matter what I set in the exposuretimerange, but when setting aelock to true, it stays at the minimum, so it appears that being able to set auto-exposure manually might provide decent images…

Running the pipeline in the background, with aelock=true, I am able to slightly increase the exposure, but it appears to be capped somehow:

$ v4l2-ctl -c exposure=600000
$ v4l2-ctl -C exposure
exposure: 33253

I have increased the defines in nvarguscamerasrc for the minimum and maximum to match the range in the sensor (7000 ns to 660000000 ns), but it appears capped to 33253000 ns for some reason (it is a bit confusing that some comments mention that exposure is in 100ths of us, but drivers manage it as they whish).

Suppose exposure would limited by the frame rate. Maybe set gain to try.

Riught, seems that, if I use aelock=true, then set exposure, gain and black_level as for the working case (when using camera_recording), I can get the P010_10LE working fine, no autoexposure/autogain, but we could implement some slow external autoexposure/autogain loop for the time being, until you guys fix the P010_10LE mode.