Max Resolution for 4:3 camera sensor Jetson Nano and Cropping Digital Zoom Strategy

[xtianus]

Arducam imx477 sensor with 4-lane mipi csi interface

hi everybody i have an arducam with imx477 sensor that work with csi interface and I attached it to jetson nano original nvidia carrier board but there is a problem here in FPS . in arducam documentation about this camera module stated that this module support 4-lane mipi csi and could reach 30fps in 4k resolution ( 4056(H) x 3040(V) ) but in jetson nano we only have 2-lane mipi csi so because of this difference I cant reach 30 fps in 4k resolution??? or this is a software problem something…

The previous topic listed above helped a little but I have further questions regarding how we can utilize output of the camera sensor in our jetson nano SDK GStreamer pipelines.

With a camera sensor that is 4:3 full 4056 (H) x 3040 (V) can you utilize that output as your src. Side off comment: It seems that resolution isn’t really the point but rather the resolution aspect ratio and DPI.

Here are my questions.

1. Can I use the full 4:3 output (above) or can I use only the described 4k 3840 x 2160?

The reason for my interest in the 4:3 resolution is because I need more of a 1:1 aspect ratio rather than a 16:9 aspect ratio that is the 3840 x 2160 ~ 4k.

4:3 isn’t exactly 1:1 but I have to follow the sensor or some distortion would arise per se from my understanding.

Potentially I could treat the 2160 (v) as the basis of the 4:3 which would then make the (H) 2880.

Either way, 2880 x 2160 or potentially 4053 x 3040 is more than enough.

So the cropping related questions.

2. Because I want to crop from the source of a 4:3 can I set the src to 1 of the above width’s and heights and use that as my source?

3. If the above resolution is my source can I digitally zoom by using the crop on Gstreamer or Nvidia and then resolve the width and height back to the base level source width and height?

4. If I keep the 4:3 aspect ratio will that give me all my factors of zoom let’s say 1 - 3 times and proper resolution aspect ratio’s?

[JerryChang]

hello christian.holes,

you may examine all the sensor supported resolutions by running this v4l format dumps, $ v4l2-ctl -d /dev/video0 --list-formats-ext
and… you may choose the closest sensor mode table as your source format to setup the camera stream. later, you could adding video convert to crop/scale the source, for creating the new output images.

here’s an example for your reference,
it’s enable camera stream with 3280x2464 sensor mode, and adding video converter, nvvidconv downscale to 1920x1080 to render to display monitor.
$ gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=3280, height=2464, framerate=21/1, format=NV12' ! nvvidconv ! 'video/x-raw(memory:NVMM), format=(string)NV12, width=1920, height=1080' ! nvoverlaysink -ev

please see-also developer guide, Accelerated GStreamer.

[xtianus]

Here is our output.

v4l2-ctl -d /dev/video0 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
Index : 0
Type : Video Capture
Pixel Format: ‘RG10’
Name : 10-bit Bayer RGRG/GBGB
Size: Discrete 3840x2160
Interval: Discrete 0.033s (30.000 fps)
Size: Discrete 1920x1080
Interval: Discrete 0.017s (60.000 fps)

to obtain 4:3 I am thinking we could go with the 2880 x 2160. Would you agree? Or am I thinking of this incorrectly?

[xtianus]

In the docs here you show a 16:9 video getting scaled down to a 4:3.

st-launch-1.0 videotestsrc ! \
  'video/x-raw, format=(string)I420, width=(int)1280, \
  height=(int)720' ! nvvidconv ! \
  'video/x-raw(memory:NVMM), width=(int)640, height=(int)480, \
  format=(string)I420' ! nvv4l2h264enc ! \
  'video/x-h264, stream-format=(string)byte-stream' ! h264parse ! \
  qtmux ! filesink location=test.mp4 -e

This is similar to what I am thinking. Can you explain the scale integrity here. scaling down to 4:3 one would assume there would be distortion going from 16:9 to 4:3.

The way I see it. I want to source the image as 2880 x 2160. Crop it down [top, bottom, left, right] and then scale it back to the 2880 x 2160. What do you think? Btw the output above doesn’t make a lot of sense. as it seems like there are only 2 available sizes.

[JerryChang]

@xtianus

Pixel Format: ‘RG10’
Name : 10-bit Bayer RGRG/GBGB
Size: Discrete 3840x2160
Interval: Discrete 0.033s (30.000 fps)
Size: Discrete 1920x1080
Interval: Discrete 0.017s (60.000 fps)

hello xtianus,

as you can see,
here’re only two sources available, 3840x2160, and 1920x1080. all these two resolutions were 16:9 aspect ratio.

so,
what’s your expectation of the output resolution.
are you care the horizontal view angles? how about cropping left/right to obtain 4:3 results.
please specify crop region (left, right, top, bottom) behind video converter,
for example, $ gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=3840, height=2160, framerate=21/1, format=NV12' ! nvvidconv left=480 right=3360 top=0 bottom=2160 ! nvoverlaysink -ev

[xtianus]

@JerryChang

hind video converter,
for example, $ gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=3840, height=2160, framerate=21/1, for

Yes, i want to crop to 4:3 and then resize(scale) to a larger 4:3. I want to set to 4:3 and then scale to a larger 4:3 for scaling. I want the source to be 4:3 but it sounds like what you’re saying is that I can only get it to crop because of the outputs.

however, where do these outputs come from? Are they from the Jetson Nano sdk firmware or are they from the camera sensor?

[JerryChang]

hello xtianus,

I don’t understand your question clearly. the sources is camera frames, and output is the cropping results.

[xtianus]

Let’s start with this question.

This output

Pixel Format: ‘RG10’
Name : 10-bit Bayer RGRG/GBGB
Size: Discrete 3840x2160
Interval: Discrete 0.033s (30.000 fps)
Size: Discrete 1920x1080
Interval: Discrete 0.017s (60.000 fps)

Where is that coming from? It’s not the raw sensor. The raw sensor has more modes than that.

The next question from that is I am looking for at the least a 4:3 camera output. There’s a difference of scaling and cropping and picking a mode for the source. Right?

If I look at the above 2 choices it seems as if I only have those 2 outputs. perhaps I am missing something.

[JerryChang]

hello xtianus,

this is the v4l to parse sensor capability and report all available sensor modes.
so, which camera sensor you’re used? did you have driver implementation? please review the camera_common_frmfmt you’ve define in driver layer.

[xtianus]

It is a sony 477.

Oddly, when we run the below command it comes back with nothing detected. We’re currently using the jetson nano dev kit. For the driver implementation we may not have done that. Is there documentation on that I can refer to?

gst-device-monitor-1.0 Video/Source

Expected in the basic drive mode

Drive mode Number of active pixels Maximum frame rate
[frame/s] Output interface ADC [bit]
Full (4:3)
(Normal)
4056 (H) × 3040 (V)
approx. 12.33 M pixels
60 CSI-2 10
40 CSI-2 12
Full (4:3)
(DOL-HDR)
4056 (H) × 3040 (V)
approx. 12.33 M pixels
DOL 2 frame : 30
DOL 3 frame : 15 CSI-2 10
Full (16:9) 4K2K
(Normal)
4056 (H) × 2288 (V)
approx. 9.28 M pixels 79 CSI-2 10
Full (16:9) 4K2K
(DOL-HDR)
4056 (H) × 2288 (V)
approx. 9.28 M pixels
DOL 2 frame : 39
DOL 3 frame : 19 CSI-2 10
Full (4:3) Binning
(Normal)
2028 (H) × 1520 (V)
approx. 3.08 M pixels 179 CSI-2 10
Full (16:9) Binning
1080P (Normal)
2028 (H) × 1128 (V)
approx. 2.29 M pixels 240 CSI-2 10
Full (16:9) Binning
720P (Normal)
1348 (H) × 750 (V)
approx. 1.01 M pixels 240 CSI-2 10
Full (16:9) Scaling
1080P (Normal)
2024 (H) × 1142 (V)
approx. 2.31 M pixels 79 CSI-2 10
Full (16:9) Scaling
720P (Normal)
1348 (H) × 762 (V)
approx. 1.03 M pixels 79 CSI-2 10

[JerryChang]

@xtianus

For the driver implementation we may not have done that.

hello xtianus,

it should be rbpcv3-imx477, right?
there’s built-in driver, you may check below for default defined modes.
for example, $public_sources/kenrel_src/kernel/nvidia/drivers/media/i2c/imx477_mode_tbls.h

please bring-up other sensor modes by yourself.
please see-also developer guide, Sensor Software Driver Programming.
thanks

[xtianus]

You are correct. Thank you @JerryChang We now have the information needed. If we have trouble implemented the software driver I will setup another ticket.

I will say the 3840 x 2160 works nicely but I wonder how much quality and accuracy we’re missing by not having the native sensor resolution?

Also, I would ask that you guys include a 4:3 resolution because for many of applications 4:3 would be the preferred resolution as outputting to 16:9 only has a very limited use case. Cinematic view.

[JerryChang]

hello xtianus,

actually, you may bring-up 4:3 sensor modes by yourself.

[xtianus]

Hi @JerryChang yes, we would have to bring up the modes but that would mean we have to install the driver correct? The engineering team was making it seem like it is a big deal to do that?

Is it as simple as applying the driver and then applying the modes available to the driver?

or is there a lot of work to do in order to keep the driver the same as the default and procure modes and other properties associated to the driver? I went through the documentation and it seems like it’s not the most simple task but perhaps it’s not very complicated either.

[JerryChang]

hello xtianus,

are you using rbpcv3-imx477? if yes, the default sensor modes should works, right?
you may contact with sensor vendor for init register settings, adding that to imx477 sensor driver.
and, you shall also port the necessary device tree settings.