SIPL and Valgrind Issue

Please provide the following info (check/uncheck the boxes after creating this topic):
Software Version
DRIVE OS Linux 5.2.6
DRIVE OS Linux 5.2.0
DRIVE OS Linux 5.2.0 and DriveWorks 3.5
NVIDIA DRIVE™ Software 10.0 (Linux)
NVIDIA DRIVE™ Software 9.0 (Linux)
other DRIVE OS version
other

Target Operating System
Linux
QNX
other

Hardware Platform
NVIDIA DRIVE™ AGX Xavier DevKit (E3550)
NVIDIA DRIVE™ AGX Pegasus DevKit (E3550)
other

SDK Manager Version
1.6.1.8175
1.6.0.8170
other

Host Machine Version
native Ubuntu 18.04
other


Hi,

we are having an issue while profiling programs based on SIPL with valgrind. SIPL wont start and It can be reproduced with the SIPL Camera (nvsipl_camera) Sample

It happens with valgrind installed with apt and this command:

/usr/bin/valgrind ./nvsipl_camera --platform-config "IMX390RGGBE3_200FOV_RGGB_CPHY_x2" --link-enable-masks "0x0001 0x0000 0x0000 0x0000" --enableRawOutput --showfps --nvsci

If I remove the --nvsci option, I can reproduce the issue if I install valgrind from source.

And finally the log is:

Module_id 30 Severity 2 : src/devblk/nonfusa/devices/MAX9295ASerializerDriver/cdi_max9295.c 1030
Module_id 30 Severity 2 : MAX9295: Device ID mismatch (expected, returned) 145 0
Module_id 30 Severity 2 : src/devblk/nonfusa/cameramodule/MAX96712cameramodule/CNvMTransportLink_Max96712_9295.cpp 100
Module_id 30 Severity 2 : MAX9295: MAX9295CheckPresence(brdcstSerCDI) failed with NvMedia error 6
Module_id 30 Severity 2 : src/devblk/nonfusa/cameramodule/MAX96712cameramodule/CNvMTransportLink_Max96712_9295.cpp 215
Module_id 30 Severity 2 : MAX96712: SetupAddressTranslations failed with SIPL error 2
Module_id 30 Severity 2 : src/devblk/nonfusa/cameramodule/MAX96712cameramodule/CNvMMAX96712CameraModule.cpp 46
Module_id 30 Severity 2 : MAX96712: TransportLink Init failed with SIPL error 2
Module_id 30 Severity 2 : src/devblk/common/core/CNvMDeviceBlock.cpp 711
Module_id 30 Severity 2 : Init failed for module on link 0
DEVBLK_WORKER_0: /dvs/git/dirty/git-master_linux/camera/fusa/sipl/src/core/CNvMDeviceBlockMgr.cpp: 435: ThreadFunc: Device block init failed(2)
Module_id 30 Severity 2 : src/core/CNvMDeviceBlockMgr.cpp 144
Module_id 30 Severity 2 : Device block action failed for block (action, block) 10
Module_id 30 Severity 2 : src/core/CNvMDeviceBlockMgr.cpp 180
Module_id 30 Severity 2 : Failed to initialize device blocks

Main: /dvs/git/dirty/git-master_linux/camera/fusa/sipl/src/core/CNvMCamera.cpp: 436: Init: DeviceBlock manager init failed
nvsipl_camera: ERROR: NvSIPLCamera Init failed

nvsipl_camera: ERROR: Master initialization failed. status: 2

Main: /dvs/git/dirty/git-master_linux/camera/fusa/sipl/src/core/pipelineMgr/spmgr/pipeline/CNvMSensorPipeline.cpp: 705: Deinit: Deinit() call in wrong state
Main: /dvs/git/dirty/git-master_linux/camera/fusa/sipl/src/core/CNvMCamera.cpp: 994: DeinitPipelines: Pipeline deinit failed for pipeline:0
Module_id 30 Severity 2 : src/devblk/fusa/cameramodule/common/utils/pwr_utils.cpp 464
Module_id 30 Severity 2 : nvccp_set_aggreg_pwr_off failed with status 2
Module_id 30 Severity 2 : src/devblk/nonfusa/devices/MAX96712DeserializerDriver/CNvMMax96712.cpp 566
Module_id 30 Severity 2 : MAX96712: CNvMMax96712::DoSetPower failed with SIPL error 127
Module_id 30 Severity 2 : src/devblk/common/ddi/DeserializerIF/CNvMDeserializer.cpp 42
Module_id 30 Severity 2 : CNvMDeserializer::SetPower failed with SIPL error 127
Module_id 30 Severity 2 : src/devblk/common/core/CNvMDeviceBlock.cpp 845
Module_id 30 Severity 2 : Deserializer SetPower failed with SIPL error 127
DEVBLK_WORKER_0: /dvs/git/dirty/git-master_linux/camera/fusa/sipl/src/core/CNvMDeviceBlockMgr.cpp: 456: ThreadFunc: Device block deinit failed(7f)
Module_id 30 Severity 2 : src/core/CNvMDeviceBlockMgr.cpp 144
Module_id 30 Severity 2 : Device block action failed for block (action, block) 40
Module_id 30 Severity 2 : src/core/CNvMDeviceBlockMgr.cpp 244
Module_id 30 Severity 2 : Failed to deinit device blocks

Main: /dvs/git/dirty/git-master_linux/camera/fusa/sipl/src/core/CNvMCamera.cpp: 1016: Deinit: DeviceBlock manager fails to deinit
Main: /dvs/git/dirty/git-master_linux/camera/fusa/sipl/src/core/CNvMCamera.cpp: 49: ~CNvMCamera: CNvMCamera object fails to deinit

NOTE: this happens with DRIVE OS 5.2.6. It used to work in DRIVE OS 5.2.0.

Hi @alejandro.solis,

Did you mean with valgrind installed with apt you can reproduce the issue with the below command?

/usr/bin/valgrind ./nvsipl_camera --platform-config “IMX390RGGBE3_200FOV_RGGB_CPHY_x2” --link-enable-masks “0x0001 0x0000 0x0000 0x0000” --enableRawOutput --showfps --nvsci

With valgrind installed from source, you can only reproduce with the below command (without “–nvsci option”)?

/usr/bin/valgrind ./nvsipl_camera --platform-config “IMX390RGGBE3_200FOV_RGGB_CPHY_x2” --link-enable-masks “0x0001 0x0000 0x0000 0x0000” --enableRawOutput --showfps

Does either command work without valgrind?

With apt’s valgrind I’ve only been able to reproduce the issue while enabling --nvsci (if I disable the option, I don’t see the issue)

With valgrind installed from source, I can reproduce the issue with and without the option.

And yes, both commands work without valgrind. But we need valgrind to ensure that the code has no leaks.

It looks the device id (read out from DevBlkCDII2CPgmrReadUint8() called in MAX9295CheckPresence() in ~/nvidia_sdk/DRIVE_OS_5.2.6_SDK_Linux_OSWithSamples_DRIVE_AGX_XAVIER/DRIVEOS/drive-t186ref-linux/samples/nvmedia/nvsipl/devblk/devices/MAX9295ASerializerDriver/cdi_max9295.c) isn’t as expected 0x19 but 0.

Is there any output from valgrind?

Thats right. I’m attaching valgrind’s output.

==32379== Memcheck, a memory error detector
==32379== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==32379== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==32379== Command: ./nvsipl_camera --platform-config IMX390RGGBE3_200FOV_RGGB_CPHY_x2 --link-enable-masks 0x0001\ 0x0000\ 0x0000\ 0x0000 --enableRawOutput --showfps --nvsci
==32379== Parent PID: 13207
==32379== 
==32379== Thread 4 DEVBLK_WORKER_0:
==32379== Invalid read of size 4
==32379==    at 0x53F97AC: DevBlkCDII2CPgmrCreate (in /usr/lib/libnvsipl_devblk_cdi.so)
==32379==    by 0x75E2D9B: ??? (in /usr/lib/nvsipl_drv/libnvsipl_devblk_drv_ar0820.so)
==32379==    by 0x53F8F97: DevBlkCDIDeviceCreate (in /usr/lib/libnvsipl_devblk_cdi.so)
==32379==    by 0x5412C17: nvsipl::CNvMDevice::CreateCDIDevice(void*, unsigned char) (in /usr/lib/libnvsipl_devblk_ddi.so)
==32379==    by 0x50082C7: ??? (in /usr/lib/libnvsipl_devblk.so)
==32379==    by 0x48FF4FB: ??? (in /usr/lib/libnvsipl.so)
==32379==    by 0x491336F: ??? (in /usr/lib/libnvsipl.so)
==32379==    by 0x4CECE93: ??? (in /usr/lib/aarch64-linux-gnu/libstdc++.so.6.0.25)
==32379==    by 0x4BE0087: start_thread (pthread_create.c:463)
==32379==    by 0x4F71FFB: thread_start (clone.S:78)
==32379==  Address 0x705c7d4 is 0 bytes after a block of size 4 alloc'd
==32379==    at 0x4847B0C: calloc (in /usr/lib/valgrind/vgpreload_memcheck-arm64-linux.so)
==32379== 
==32379== Invalid read of size 4
==32379==    at 0x53F97AC: DevBlkCDII2CPgmrCreate (in /usr/lib/libnvsipl_devblk_cdi.so)
==32379==    by 0x795461B: ??? (in /usr/lib/nvsipl_drv/libnvsipl_devblk_drv_imx390.so)
==32379==    by 0x53F8F97: DevBlkCDIDeviceCreate (in /usr/lib/libnvsipl_devblk_cdi.so)
==32379==    by 0x5412C17: nvsipl::CNvMDevice::CreateCDIDevice(void*, unsigned char) (in /usr/lib/libnvsipl_devblk_ddi.so)
==32379==    by 0x79399F7: ??? (in /usr/lib/nvsipl_drv/libnvsipl_devblk_drv_imx390.so)
==32379==    by 0x541383B: nvsipl::CNvMCameraModule::CreateCDIDevice(void*, unsigned char) (in /usr/lib/libnvsipl_devblk_ddi.so)
==32379==    by 0x50083B3: ??? (in /usr/lib/libnvsipl_devblk.so)
==32379==    by 0x48FF4FB: ??? (in /usr/lib/libnvsipl.so)
==32379==    by 0x491336F: ??? (in /usr/lib/libnvsipl.so)
==32379==    by 0x4CECE93: ??? (in /usr/lib/aarch64-linux-gnu/libstdc++.so.6.0.25)
==32379==    by 0x4BE0087: start_thread (pthread_create.c:463)
==32379==    by 0x4F71FFB: thread_start (clone.S:78)
==32379==  Address 0x70aa034 is 0 bytes after a block of size 4 alloc'd
==32379==    at 0x4847B0C: calloc (in /usr/lib/valgrind/vgpreload_memcheck-arm64-linux.so)
==32379== 
==32379== Invalid read of size 4
==32379==    at 0x53F97AC: DevBlkCDII2CPgmrCreate (in /usr/lib/libnvsipl_devblk_cdi.so)
==32379==    by 0x795D0E7: ??? (in /usr/lib/nvsipl_drv/libnvsipl_devblk_drv_imx390.so)
==32379==    by 0x53F8F97: DevBlkCDIDeviceCreate (in /usr/lib/libnvsipl_devblk_cdi.so)
==32379==    by 0x5412C17: nvsipl::CNvMDevice::CreateCDIDevice(void*, unsigned char) (in /usr/lib/libnvsipl_devblk_ddi.so)
==32379==    by 0x7939A0B: ??? (in /usr/lib/nvsipl_drv/libnvsipl_devblk_drv_imx390.so)
==32379==    by 0x541383B: nvsipl::CNvMCameraModule::CreateCDIDevice(void*, unsigned char) (in /usr/lib/libnvsipl_devblk_ddi.so)
==32379==    by 0x50083B3: ??? (in /usr/lib/libnvsipl_devblk.so)
==32379==    by 0x48FF4FB: ??? (in /usr/lib/libnvsipl.so)
==32379==    by 0x491336F: ??? (in /usr/lib/libnvsipl.so)
==32379==    by 0x4CECE93: ??? (in /usr/lib/aarch64-linux-gnu/libstdc++.so.6.0.25)
==32379==    by 0x4BE0087: start_thread (pthread_create.c:463)
==32379==    by 0x4F71FFB: thread_start (clone.S:78)
==32379==  Address 0x70a86d4 is 0 bytes after a block of size 4 alloc'd
==32379==    at 0x4847B0C: calloc (in /usr/lib/valgrind/vgpreload_memcheck-arm64-linux.so)
==32379== 
ARM64 front end: load_store
disInstr(arm64): unhandled instruction 0xB820003E
disInstr(arm64): 1011'1000 0010'0000 0000'0000 0011'1110
==32379== valgrind: Unrecognised instruction at address 0x4b4b484.
==32379==    at 0x4B4B484: ??? (in /usr/lib/aarch64-linux-gnu/libEGL_nvidia.so.0)
==32379==    by 0x4AE96A7: ??? (in /usr/lib/aarch64-linux-gnu/libEGL_nvidia.so.0)
==32379==    by 0x4B4C28F: ??? (in /usr/lib/aarch64-linux-gnu/libEGL_nvidia.so.0)
==32379==    by 0x4BE7C77: __pthread_once_slow (pthread_once.c:116)
==32379==    by 0x4B4C053: ??? (in /usr/lib/aarch64-linux-gnu/libEGL_nvidia.so.0)
==32379==    by 0x4AE7F37: ??? (in /usr/lib/aarch64-linux-gnu/libEGL_nvidia.so.0)
==32379==    by 0x400DDD7: _dl_fini (dl-fini.c:138)
==32379==    by 0x4ED6E53: __run_exit_handlers (exit.c:108)
==32379==    by 0x4ED6F8B: exit (exit.c:139)
==32379==    by 0x4EC2723: (below main) (libc-start.c:344)
==32379== Your program just tried to execute an instruction that Valgrind
==32379== did not recognise.  There are two possible reasons for this.
==32379== 1. Your program has a bug and erroneously jumped to a non-code
==32379==    location.  If you are running Memcheck and you just saw a
==32379==    warning about a bad jump, it's probably your program's fault.
==32379== 2. The instruction is legitimate but Valgrind doesn't handle it,
==32379==    i.e. it's Valgrind's fault.  If you think this is the case or
==32379==    you are not sure, please let us know and we'll try to fix it.
==32379== Either way, Valgrind will now raise a SIGILL signal which will
==32379== probably kill your program.
==32379== 
==32379== Process terminating with default action of signal 4 (SIGILL)
==32379==  Illegal opcode at address 0x4B4B484
==32379==    at 0x4B4B484: ??? (in /usr/lib/aarch64-linux-gnu/libEGL_nvidia.so.0)
==32379==    by 0x4AE96A7: ??? (in /usr/lib/aarch64-linux-gnu/libEGL_nvidia.so.0)
==32379==    by 0x4B4C28F: ??? (in /usr/lib/aarch64-linux-gnu/libEGL_nvidia.so.0)
==32379==    by 0x4BE7C77: __pthread_once_slow (pthread_once.c:116)
==32379==    by 0x4B4C053: ??? (in /usr/lib/aarch64-linux-gnu/libEGL_nvidia.so.0)
==32379==    by 0x4AE7F37: ??? (in /usr/lib/aarch64-linux-gnu/libEGL_nvidia.so.0)
==32379==    by 0x400DDD7: _dl_fini (dl-fini.c:138)
==32379==    by 0x4ED6E53: __run_exit_handlers (exit.c:108)
==32379==    by 0x4ED6F8B: exit (exit.c:139)
==32379==    by 0x4EC2723: (below main) (libc-start.c:344)
==32379== 
==32379== HEAP SUMMARY:
==32379==     in use at exit: 13,318 bytes in 72 blocks
==32379==   total heap usage: 210,184 allocs, 210,112 frees, 80,054,785 bytes allocated
==32379== 
==32379== LEAK SUMMARY:
==32379==    definitely lost: 208 bytes in 4 blocks
==32379==    indirectly lost: 6,322 bytes in 60 blocks
==32379==      possibly lost: 0 bytes in 0 blocks
==32379==    still reachable: 6,788 bytes in 8 blocks
==32379==         suppressed: 0 bytes in 0 blocks
==32379== Rerun with --leak-check=full to see details of leaked memory
==32379== 
==32379== For counts of detected and suppressed errors, rerun with: -v
==32379== ERROR SUMMARY: 5 errors from 3 contexts (suppressed: 0 from 0)

Above is only valgrind output. I also can get both logs together:

==22433== Memcheck, a memory error detector
==22433== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==22433== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==22433== Command: ./nvsipl_camera --platform-config IMX390RGGBE3_200FOV_RGGB_CPHY_x2 --link-enable-masks 0x0001\ 0x0000\ 0x0000\ 0x0000 --enableRawOutput --showfps --nvsci
==22433== 
==22433== Thread 4 DEVBLK_WORKER_0:
==22433== Invalid read of size 4
==22433==    at 0x53F97AC: DevBlkCDII2CPgmrCreate (in /usr/lib/libnvsipl_devblk_cdi.so)
==22433==    by 0x75E2D9B: ??? (in /usr/lib/nvsipl_drv/libnvsipl_devblk_drv_ar0820.so)
==22433==    by 0x53F8F97: DevBlkCDIDeviceCreate (in /usr/lib/libnvsipl_devblk_cdi.so)
==22433==    by 0x5412C17: nvsipl::CNvMDevice::CreateCDIDevice(void*, unsigned char) (in /usr/lib/libnvsipl_devblk_ddi.so)
==22433==    by 0x50082C7: ??? (in /usr/lib/libnvsipl_devblk.so)
==22433==    by 0x48FF4FB: ??? (in /usr/lib/libnvsipl.so)
==22433==    by 0x491336F: ??? (in /usr/lib/libnvsipl.so)
==22433==    by 0x4CECE93: ??? (in /usr/lib/aarch64-linux-gnu/libstdc++.so.6.0.25)
==22433==    by 0x4BE0087: start_thread (pthread_create.c:463)
==22433==    by 0x4F71FFB: thread_start (clone.S:78)
==22433==  Address 0x705c7d4 is 0 bytes after a block of size 4 alloc'd
==22433==    at 0x4847B0C: calloc (in /usr/lib/valgrind/vgpreload_memcheck-arm64-linux.so)
==22433== 
==22433== Invalid read of size 4
==22433==    at 0x53F97AC: DevBlkCDII2CPgmrCreate (in /usr/lib/libnvsipl_devblk_cdi.so)
==22433==    by 0x795461B: ??? (in /usr/lib/nvsipl_drv/libnvsipl_devblk_drv_imx390.so)
==22433==    by 0x53F8F97: DevBlkCDIDeviceCreate (in /usr/lib/libnvsipl_devblk_cdi.so)
==22433==    by 0x5412C17: nvsipl::CNvMDevice::CreateCDIDevice(void*, unsigned char) (in /usr/lib/libnvsipl_devblk_ddi.so)
==22433==    by 0x79399F7: ??? (in /usr/lib/nvsipl_drv/libnvsipl_devblk_drv_imx390.so)
==22433==    by 0x541383B: nvsipl::CNvMCameraModule::CreateCDIDevice(void*, unsigned char) (in /usr/lib/libnvsipl_devblk_ddi.so)
==22433==    by 0x50083B3: ??? (in /usr/lib/libnvsipl_devblk.so)
==22433==    by 0x48FF4FB: ??? (in /usr/lib/libnvsipl.so)
==22433==    by 0x491336F: ??? (in /usr/lib/libnvsipl.so)
==22433==    by 0x4CECE93: ??? (in /usr/lib/aarch64-linux-gnu/libstdc++.so.6.0.25)
==22433==    by 0x4BE0087: start_thread (pthread_create.c:463)
==22433==    by 0x4F71FFB: thread_start (clone.S:78)
==22433==  Address 0x70aa034 is 0 bytes after a block of size 4 alloc'd
==22433==    at 0x4847B0C: calloc (in /usr/lib/valgrind/vgpreload_memcheck-arm64-linux.so)
==22433== 
==22433== Invalid read of size 4
==22433==    at 0x53F97AC: DevBlkCDII2CPgmrCreate (in /usr/lib/libnvsipl_devblk_cdi.so)
==22433==    by 0x795D0E7: ??? (in /usr/lib/nvsipl_drv/libnvsipl_devblk_drv_imx390.so)
==22433==    by 0x53F8F97: DevBlkCDIDeviceCreate (in /usr/lib/libnvsipl_devblk_cdi.so)
==22433==    by 0x5412C17: nvsipl::CNvMDevice::CreateCDIDevice(void*, unsigned char) (in /usr/lib/libnvsipl_devblk_ddi.so)
==22433==    by 0x7939A0B: ??? (in /usr/lib/nvsipl_drv/libnvsipl_devblk_drv_imx390.so)
==22433==    by 0x541383B: nvsipl::CNvMCameraModule::CreateCDIDevice(void*, unsigned char) (in /usr/lib/libnvsipl_devblk_ddi.so)
==22433==    by 0x50083B3: ??? (in /usr/lib/libnvsipl_devblk.so)
==22433==    by 0x48FF4FB: ??? (in /usr/lib/libnvsipl.so)
==22433==    by 0x491336F: ??? (in /usr/lib/libnvsipl.so)
==22433==    by 0x4CECE93: ??? (in /usr/lib/aarch64-linux-gnu/libstdc++.so.6.0.25)
==22433==    by 0x4BE0087: start_thread (pthread_create.c:463)
==22433==    by 0x4F71FFB: thread_start (clone.S:78)
==22433==  Address 0x70a86d4 is 0 bytes after a block of size 4 alloc'd
==22433==    at 0x4847B0C: calloc (in /usr/lib/valgrind/vgpreload_memcheck-arm64-linux.so)
==22433== 
MAX96712: Revision 2 detected
MAX96712 Rev 2: Link 0: Tx amplitude 0x28, 0x68, 0xb3, 0x72, 0xd2
MAX96712 Rev 2 manual adaptation on the link 0 (31)
Module_id 30 Severity 2 : src/devblk/nonfusa/devices/MAX9295ASerializerDriver/cdi_max9295.c 1030
Module_id 30 Severity 2 : MAX9295: Device ID mismatch (expected, returned) 145 0
Module_id 30 Severity 2 : src/devblk/nonfusa/cameramodule/MAX96712cameramodule/CNvMTransportLink_Max96712_9295.cpp 100
Module_id 30 Severity 2 : MAX9295: MAX9295CheckPresence(brdcstSerCDI) failed with NvMedia error 6
Module_id 30 Severity 2 : src/devblk/nonfusa/cameramodule/MAX96712cameramodule/CNvMTransportLink_Max96712_9295.cpp 215
Module_id 30 Severity 2 : MAX96712: SetupAddressTranslations failed with SIPL error 2
Module_id 30 Severity 2 : src/devblk/nonfusa/cameramodule/MAX96712cameramodule/CNvMMAX96712CameraModule.cpp 46
Module_id 30 Severity 2 : MAX96712: TransportLink Init failed with SIPL error 2
Module_id 30 Severity 2 : src/devblk/common/core/CNvMDeviceBlock.cpp 711
Module_id 30 Severity 2 : Init failed for module on link 0
DEVBLK_WORKER_0: /dvs/git/dirty/git-master_linux/camera/fusa/sipl/src/core/CNvMDeviceBlockMgr.cpp: 435: ThreadFunc: Device block init failed(2)
Module_id 30 Severity 2 : src/core/CNvMDeviceBlockMgr.cpp 144
Module_id 30 Severity 2 : Device block action failed for block (action, block) 10
Module_id 30 Severity 2 : src/core/CNvMDeviceBlockMgr.cpp 180
Module_id 30 Severity 2 : Failed to initialize device blocks

Main: /dvs/git/dirty/git-master_linux/camera/fusa/sipl/src/core/CNvMCamera.cpp: 436: Init: DeviceBlock manager init failed
nvsipl_camera: ERROR: NvSIPLCamera Init failed

nvsipl_camera: ERROR: Master initialization failed. status: 2

Main: /dvs/git/dirty/git-master_linux/camera/fusa/sipl/src/core/pipelineMgr/spmgr/pipeline/CNvMSensorPipeline.cpp: 705: Deinit: Deinit() call in wrong state
Main: /dvs/git/dirty/git-master_linux/camera/fusa/sipl/src/core/CNvMCamera.cpp: 994: DeinitPipelines: Pipeline deinit failed for pipeline:0
Module_id 30 Severity 2 : src/devblk/fusa/cameramodule/common/utils/pwr_utils.cpp 464
Module_id 30 Severity 2 : nvccp_set_aggreg_pwr_off failed with status 2
Module_id 30 Severity 2 : src/devblk/nonfusa/devices/MAX96712DeserializerDriver/CNvMMax96712.cpp 566
Module_id 30 Severity 2 : MAX96712: CNvMMax96712::DoSetPower failed with SIPL error 127
Module_id 30 Severity 2 : src/devblk/common/ddi/DeserializerIF/CNvMDeserializer.cpp 42
Module_id 30 Severity 2 : CNvMDeserializer::SetPower failed with SIPL error 127
Module_id 30 Severity 2 : src/devblk/common/core/CNvMDeviceBlock.cpp 845
Module_id 30 Severity 2 : Deserializer SetPower failed with SIPL error 127
DEVBLK_WORKER_0: /dvs/git/dirty/git-master_linux/camera/fusa/sipl/src/core/CNvMDeviceBlockMgr.cpp: 456: ThreadFunc: Device block deinit failed(7f)
Module_id 30 Severity 2 : src/core/CNvMDeviceBlockMgr.cpp 144
Module_id 30 Severity 2 : Device block action failed for block (action, block) 40
Module_id 30 Severity 2 : src/core/CNvMDeviceBlockMgr.cpp 244
Module_id 30 Severity 2 : Failed to deinit device blocks

Main: /dvs/git/dirty/git-master_linux/camera/fusa/sipl/src/core/CNvMCamera.cpp: 1016: Deinit: DeviceBlock manager fails to deinit
Main: /dvs/git/dirty/git-master_linux/camera/fusa/sipl/src/core/CNvMCamera.cpp: 49: ~CNvMCamera: CNvMCamera object fails to deinit
ARM64 front end: load_store
disInstr(arm64): unhandled instruction 0xB820003E
disInstr(arm64): 1011'1000 0010'0000 0000'0000 0011'1110
==22433== valgrind: Unrecognised instruction at address 0x4b4b484.
==22433==    at 0x4B4B484: ??? (in /usr/lib/aarch64-linux-gnu/libEGL_nvidia.so.0)
==22433==    by 0x4AE96A7: ??? (in /usr/lib/aarch64-linux-gnu/libEGL_nvidia.so.0)
==22433==    by 0x4B4C28F: ??? (in /usr/lib/aarch64-linux-gnu/libEGL_nvidia.so.0)
==22433==    by 0x4BE7C77: __pthread_once_slow (pthread_once.c:116)
==22433==    by 0x4B4C053: ??? (in /usr/lib/aarch64-linux-gnu/libEGL_nvidia.so.0)
==22433==    by 0x4AE7F37: ??? (in /usr/lib/aarch64-linux-gnu/libEGL_nvidia.so.0)
==22433==    by 0x400DDD7: _dl_fini (dl-fini.c:138)
==22433==    by 0x4ED6E53: __run_exit_handlers (exit.c:108)
==22433==    by 0x4ED6F8B: exit (exit.c:139)
==22433==    by 0x4EC2723: (below main) (libc-start.c:344)
==22433== Your program just tried to execute an instruction that Valgrind
==22433== did not recognise.  There are two possible reasons for this.
==22433== 1. Your program has a bug and erroneously jumped to a non-code
==22433==    location.  If you are running Memcheck and you just saw a
==22433==    warning about a bad jump, it's probably your program's fault.
==22433== 2. The instruction is legitimate but Valgrind doesn't handle it,
==22433==    i.e. it's Valgrind's fault.  If you think this is the case or
==22433==    you are not sure, please let us know and we'll try to fix it.
==22433== Either way, Valgrind will now raise a SIGILL signal which will
==22433== probably kill your program.
==22433== 
==22433== Process terminating with default action of signal 4 (SIGILL)
==22433==  Illegal opcode at address 0x4B4B484
==22433==    at 0x4B4B484: ??? (in /usr/lib/aarch64-linux-gnu/libEGL_nvidia.so.0)
==22433==    by 0x4AE96A7: ??? (in /usr/lib/aarch64-linux-gnu/libEGL_nvidia.so.0)
==22433==    by 0x4B4C28F: ??? (in /usr/lib/aarch64-linux-gnu/libEGL_nvidia.so.0)
==22433==    by 0x4BE7C77: __pthread_once_slow (pthread_once.c:116)
==22433==    by 0x4B4C053: ??? (in /usr/lib/aarch64-linux-gnu/libEGL_nvidia.so.0)
==22433==    by 0x4AE7F37: ??? (in /usr/lib/aarch64-linux-gnu/libEGL_nvidia.so.0)
==22433==    by 0x400DDD7: _dl_fini (dl-fini.c:138)
==22433==    by 0x4ED6E53: __run_exit_handlers (exit.c:108)
==22433==    by 0x4ED6F8B: exit (exit.c:139)
==22433==    by 0x4EC2723: (below main) (libc-start.c:344)
==22433== 
==22433== HEAP SUMMARY:
==22433==     in use at exit: 13,318 bytes in 72 blocks
==22433==   total heap usage: 210,184 allocs, 210,112 frees, 80,054,785 bytes allocated
==22433== 
==22433== LEAK SUMMARY:
==22433==    definitely lost: 208 bytes in 4 blocks
==22433==    indirectly lost: 6,322 bytes in 60 blocks
==22433==      possibly lost: 0 bytes in 0 blocks
==22433==    still reachable: 6,788 bytes in 8 blocks
==22433==         suppressed: 0 bytes in 0 blocks
==22433== Rerun with --leak-check=full to see details of leaked memory
==22433== 
==22433== For counts of detected and suppressed errors, rerun with: -v
==22433== ERROR SUMMARY: 5 errors from 3 contexts (suppressed: 0 from 0)
Illegal instruction

We can see a similar issue with a Sekonix camera module.
We will check this with the internal team and get back to you.
Thanks.

1 Like

Hi @VickNV,

Thanks!

Do you think a fix could be shared for this release (5.2.6)?

Usually, we don’t provide a fix in released versions. Sorry for any inconvenience.

@VickNV

Could you recommend another tool to profile memory consumption? Maybe there’s an Nvidia tool that will work with the SIPL framework? We just need to be sure that there are no leaks in any running code.

Have you observed any memory leakage? even after stop/kill the process?

I think there might be a leak in some SIPL library. I say that because I noticed a leak while starting the camera multiple times within the same process. So probably the memory will be freed when the process is over.

And I say it’s not our code because I tested this on 5.2.0 and there was no leak, so I think it’s a leak in the SIPL framework of this new DRIVE OS (5.2.6). Anyway, I cannot use valgrind to address the leak.

Please, if you can enable valgrind for the next release, it would be nice, so we can profile our code easier. Thank you!