Nvsiple_camera run multiple camera report resource error

Autonomous Vehicles DRIVE AGX Thor DRIVE AGX Thor General
1

DRIVE OS Version: 7.0.3.0-0010

HW Board Variate: p3960-10-sw03

Issue Description:

Testing nvsipl_camera with multiple camera, get resource error.

when use nvsipl_camera to run with 10 camera at the same time is OK.

when try to add 11 camera, NvSciBufObjAlloc return NvSciError_ResourceError.

Detail log:

nvsipl_camera: ERROR: NvSciBufObjAlloc failed. nvSciStatus: 50
nvsipl_camera: ERROR: FillAttributesAllocateSciBuf failed for output type:0
nvsipl_camera: ERROR: AllocateBuffers failed for sensor pipeline:17

Jul 14 14:01:36 tegra-ubuntu nvsipl_camera[535987]: [TS:95739604313214] Module_id 30 Severity 5 : --------- END PIPELINE CONFIGURATION --------- 17
Jul 14 14:01:36 tegra-ubuntu nvsipl_camera[535987]: [TS:95739648188653] Module_id 39 Severity 2 : NvMapMemAllocInternalTagged failed: error 12
Jul 14 14:01:36 tegra-ubuntu nvsipl_camera[535987]: [TS:95739648210439] Module_id 39 Severity 2 : Failed ioctlCmd: 1075858947
Jul 14 14:01:36 tegra-ubuntu nvsipl_camera[535987]: [TS:95739648212615] Module_id 39 Severity 2 : NvMapMemHandleAlloc failed: error 12
Jul 14 14:01:36 tegra-ubuntu nvsipl_camera[535987]: [TS:95739648224745] Module_id 45 Severity 2 : NvRmMemHandleAllocAttr failed. NvError: 6
Jul 14 14:01:36 tegra-ubuntu nvsipl_camera[535987]: [TS:95739648227763] Module_id 45 Severity 2 : NvSciBufRmAlloc failed
Jul 14 14:01:36 tegra-ubuntu nvsipl_camera[535987]: [TS:95739648229411] Module_id 45 Severity 2 : allocation failed for allocation type: 0
Jul 14 14:01:36 tegra-ubuntu nvsipl_camera[535987]: [TS:95739648231930] Module_id 45 Severity 2 : Could not allocate buffer from allocation interface
Jul 14 14:01:36 tegra-ubuntu nvsipl_camera[535987]: [TS:95739648234106] Module_id 45 Severity 2 : NvSciBufObj Getting attr failed
Jul 14 14:01:40 tegra-ubuntu nvsipl_camera[535987]: [TS:95743608902263] Module_id 30 Severity 2 : CNvFusaISPBlock.cpp 1721

Camera used:

Smartlead BF8S103G-100-00.

Test step:

run cmd with 10 camera ok:

./nvsipl_camera --platform-config “R0SIM728S3RU2120NB2_5Des_CPHY_x4” --link-enable-masks “0x0111 0x00 0x0000 0x1111 0x0111” -s

run cmd with 11 camera report buffer resource error:

./nvsipl_camera --platform-config “R0SIM728S3RU2120NB2_5Des_CPHY_x4” --link-enable-masks “0x0111 0x00 0x0000 0x1111 0x1111” -s

run cmd with 11 camera and ISP1 and ISP2 disabled, still error:

./nvsipl_camera --platform-config “R0SIM728S3RU2120NB2_5Des_CPHY_x4” --link-enable-masks “0x0111 0x00 0x0000 0x1111 0x1111” -s -1 -2

Question:

  1. what is the upper limitation for for this error, how to increac the resource?
  2. why when ISP1/2 channels are disabled, less buffer applied, still report resource error?
5

Dear @snow.yin ,
Did you try bufferAlloc parameter ? Command Line Switches — NVIDIA DriveOS 7.0.3 Linux SDK Developer Guide
Also, could you check sudo cat /sys/kernel/debug/nvmap/vimem/free_size before/after/during running of application

6

Hi @SivaRamaKrishnaNV

I tried the recommended CMD and option:

free_size is 1G, and bufferAlloc option only make the problem worse, with “40 64 64 64”, only one camera would work.

And I being confusing about the upper limit, “40” is the upper limit for all camera VI totally combine right? If it only apply to one camera, the size would be too big?

By default each camera apply ICP buffer 6, then 40/6 = 6.6, but as test result shows, I can bring up 10 cameras.

Jul 14 16:04:11 tegra-ubuntu nvsipl_camera[581916]: [TS:103094727103929] Module_id 30 Severity 5 : maxCaptureBufferCount: 40
Jul 14 16:04:11 tegra-ubuntu nvsipl_camera[581916]: [TS:103094727109299] Module_id 30 Severity 5 : maxIsp0BufferCount: 64
Jul 14 16:04:11 tegra-ubuntu nvsipl_camera[581916]: [TS:103094727111170] Module_id 30 Severity 5 : maxIsp1BufferCount: 64
Jul 14 16:04:11 tegra-ubuntu nvsipl_camera[581916]: [TS:103094727112438] Module_id 30 Severity 5 : maxIsp2BufferCount: 64

please see log below.

drive@tegra-ubuntu:/tmp$ sudo cat /sys/kernel/debug/nvmap/vimem/free_size
[sudo] password for drive:
0x0000000040000000
drive@tegra-ubuntu:/tmp$ ./nvsipl_camera --platform-config “R0SIM728S3RU2120NB2_5Des_CPHY_x4” --link-enable-masks “0x0 0x00 0x0000 0x0001 0x0” -d 1 -s -v 1 -B ‘40 64 64 64’
Pipeline: 10 ISP Output: 0 is using YUV 420 SEMI-PLANAR UINT8 BL REC_709ER
Pipeline: 10 ISP Output: 1 is using YUV 420 SEMI-PLANAR UINT8 BL REC_709ER
Pipeline: 10 ISP Output: 2 is using YUV 420 SEMI-PLANAR UINT8 BL REC_709ER

snowtst CreatePMIC status issup 0
MAX96712: Revision 5 detected
MAX96712 Link 0: PHY optimization was enabled
MAX96717: Revision 6 detected!
Sensor IMX728 Rev 0x34 RGGB detected!
nvsipl_camera: Opened NITO file for module “R0SIM728S3RU2120NB2”, file name: “/usr/share/camera/R0SIM728S3RU2120NB2.nito”

Output
Sensor10_Out1 Frame rate (fps): 28.9991
Sensor10_Out2 Frame rate (fps): 28.9991
Sensor10_Out3 Frame rate (fps): 28.9991

q
Output
Sensor10_Out1 Frame rate (fps): 29.9991
Sensor10_Out2 Frame rate (fps): 29.9991
Sensor10_Out3 Frame rate (fps): 29.9991

Sensor10 Frame captured: 118 Frame drops: 0 Frame discontinuities: 0 Frames authenticated: 0 Frames failed to authenticate: 0 Frames authenticated as Out Of Order: 0
SUCCESS
drive@tegra-ubuntu:/tmp$ sudo cat /sys/kernel/debug/nvmap/vimem/free_size
0x0000000040000000
drive@tegra-ubuntu:/tmp$ ./nvsipl_camera --platform-config “R0SIM728S3RU2120NB2_5Des_CPHY_x4” --link-enable-masks “0x0 0x00 0x0000 0x0011 0x0” -d 1 -s -v 1 -B ‘40 64 64 64’
Pipeline: 10 ISP Output: 0 is using YUV 420 SEMI-PLANAR UINT8 BL REC_709ER
Pipeline: 10 ISP Output: 1 is using YUV 420 SEMI-PLANAR UINT8 BL REC_709ER
Pipeline: 10 ISP Output: 2 is using YUV 420 SEMI-PLANAR UINT8 BL REC_709ER
nvsipl_camera: ERROR: NvSciBufObjAlloc failed. nvSciStatus: 50
nvsipl_camera: ERROR: FillAttributesAllocateSciBuf failed for output type:0
nvsipl_camera: ERROR: AllocateBuffers failed for sensor pipeline:11
nvsipl_camera: ERROR: GetSyncObj failed. status: 10
nvsipl_camera: ERROR: Compositor: UnregisterNvSciSyncObjs failed
drive@tegra-ubuntu:/tmp$ sudo cat /sys/kernel/debug/nvmap/vimem/free_size
0x0000000040000000

7

I did more test

run one camera, the free size become 0x0000000039f40000, consuming exactly 6 buffer 96MB.

run two camera, the free size become 0x0000000033e80000, consuming exactly 12 buffer 193MB.

after run 10 cameras, the free size left is 0x0000000033e80000, so it’s not enough.

8

May I know how many cameras and which ISP outputs needed in the use case.
Also, do you have any dependency on DOS 7.0.3 release or can be upgraded to DOS 7.2.5 release?

9

Hi @SivaRamaKrishnaNV

our use case requires 11 camera running at the same time.

we need all three ISP output from 0/1/2. one for compositor display, one for encoding, one for scaling and crops.

we do have plans to upgraded to 7.2.5 but perhaps several month later, we find camera drivers are not really compatible and need some rework.

with your help I realize that an alternative is to reduce that the ICP buffer used by one camera from 6 to 4, but not sure about the side effect yet.

10

Is it working well after reducing ICP buffers?

11

yes, I did not observed any frame drop when change ICP to 4~5

tested inter-frame diff always 33ms