Hello all,
I am developing a middleware that takes RAW frames and sends them to a ROS topic for each camera available based on the sample_camera provided with the dev SDK.
I am using DW version 6.0.10.0.
I am receiving DW_CAMERA_OUTPUT_CUDA_RGBA_UINT8 frames and the sensor params are:
camera-name=IMX728_RGGB, interface=csi-ab, CPHY-mode=1, link=0, output-format=yuv, async-record=1, file-buffer-size=16777216, nito-file=/usr/share/camera/IMX728_RGGB.nito, isp0-res=3840X2160, type=user, format=h265, bitrate=8000000, gop-length=15, quality=24, fifo-size=4, encoder-instance=0.
Can I directly encode the frames to h265 right after receiving them from the isp0 stream? From the test I have done, the message “CameraClient: h264 and h265 video format encoding is deprecated. Please switch to a different format“ also appears, but may be because of the way I am trying to convert the images.
From the samples I have read, it seems the new codec stack also doesn’t support h265, would the performance difference of using 8x 4k30fps camera in the previous codec stack be significant agains, say a mp4 codec in the new stack?
Thank you very much,
P. Ramos
mp4 is a container over H264 encoded stream. We recommend to use mp4 format.
I have sucessfully encoded the image coming from the isp0 to h265 (or h264 if I want) and sent it along to a ROS2 network.
My problem now is that I am using 4x 4k30fps IMX728 cameras and even with the isp0 output at 1080p, I cannot get over 20fps on each camera. I could get 1080p30fps without the encoding.
I know the bottleneck is at the encoder since I pipe the images to it straight from the isp0 and about 1/3 of my frames are dropped from the fifo queue with the message (all 4 pipeline indexes appear):
SIPLNotifier: Notification received from pipeline index:0 of type: NOTIF_WARN_ICP_FRAME_DROP
If I increase the fifo-size, predictably, it takes longer for these message to start appearing.
My question is, can more encoder instances fix this bottleneck? Or some other setting I haven’t mentioned from my original rig config? I have tried encoder-instance 1 and 2.
Encoder instance 1:
Sensor::onDataRead, Health critical for sensor camera[0] DW_SENSOR_STATE_DELTA_CURRENT_AND_HOST_TIME no data received for 500000 us currentDiff 1772625613291520 Frequency of DI: 200
CameraClient: Readframe timeout
CameraClient: Camera client cannot read processed frame. camera=1 Status=DW_TIME_OUT
Encoder instance 2:
Driveworks exception thrown: DW_INVALID_ARGUMENT: CameraClient: invalid encoder instance
Is it possible to scale the encoder instances? I am also going to add 4 more IMX728 4k30fps cameras (so 8 total) but these wont be encoded. So far the isp0 pipeline handles it well.
can more encoder instances fix this bottleneck?
No. Orin has only one encoder.
The feasible FPS for a camera module can be verified using NvMedia IEP sample. You need to plan encoding in your pipeline accordingly.
Ok thanks, I will try that.
I managed to get encoder 1 working for the second set of four cameras I have connected, leaving encoder 0 for my first set of four.
With some other optimizations, I managed to get 25 fps from all 8x 8mp cameras and I have reached the hardware limit for the ISP input, the rest of the system is not being stressed at all and I can increase the isp resolution of some cameras from 1080p to 4k or even increase encoding quality without any performance drop.
Knowing the max ISP max throughput is about 1.85 gigapixels per second, this makes sense as 6 cameras are 1.5Gpx/s and 8 are 2Gpx/s.
Is there anyway I can configure the cameras to output only 1080p (before downscaling in the ISP)? In my final application some cameras only need a small resolution and this would free up bandwidth for the rest in the ISP input. I know this GMSL2 camera can be configured by an external board, but then I guess my nito file would be innacurate, unless the sensor params in my rig file can compensate for it.
Here is an example of the configurations that can be done to my camera.
Thank you very much for your time.