Unable to add custom lidar node to minipepeline example

Please provide the following info (tick the boxes after creating this topic):
Software Version
DRIVE OS 6.0.8.1
DRIVE OS 6.0.6
DRIVE OS 6.0.5
DRIVE OS 6.0.4 (rev. 1)
DRIVE OS 6.0.4 SDK
other

Target Operating System
Linux
QNX
other

Hardware Platform
DRIVE AGX Orin Developer Kit (940-63710-0010-300)
DRIVE AGX Orin Developer Kit (940-63710-0010-200)
DRIVE AGX Orin Developer Kit (940-63710-0010-100)
DRIVE AGX Orin Developer Kit (940-63710-0010-D00)
DRIVE AGX Orin Developer Kit (940-63710-0010-C00)
DRIVE AGX Orin Developer Kit (not sure its number)
other

SDK Manager Version
1.9.3.10904
other

Host Machine Version
native Ubuntu Linux 20.04 Host installed with SDK Manager
native Ubuntu Linux 20.04 Host installed with DRIVE OS Docker Containers
native Ubuntu Linux 18.04 Host installed with DRIVE OS Docker Containers
other

Hello,
I was trying to apply what discussed in this topic How to link custom plugin library for Radar in CGF to my LIDAR, by using a custom plugin lib which has been previously and succesfully tested with the lidar_sample_replay and linking to the minipepeline example provvided by the SDK.

However i did notice that minipeline example was implementing Camera, Imu and Radar Node but no Lidar (or PointCloud) was mentioned at all.
I then implemnted the Lidar Graphlet as shown in picture below:

with dwLidarBatchNode taken from usr/local/driveworks-5.14/src/dwframework/dwnodes/sensors/dwlidarnode folder.

I added it at my minipepeline as in picture:


and properly modified the app.json as:

{
    "name": "MiniPipelineApp",
    "logSpec": "file/rfc5424:{{logpath}}/{{appname}}.log",
    "parameters": {
     ...
    },
    "requiredSensors": "./MiniPipeline.required-sensors.json",
    "sensorMappingLookups": [
        "../../../../../usr/local/driveworks-5.14/apps/roadrunner-2.0/config/sensor_mappings"
    ],
    "subcomponents": {
        "miniPipeline": {
            "componentType": "../graphlets/MiniPipeline.graphlet.json",
            "parameters": {}
        }
    },
    "connections": [],
    "states": {
        "STANDARD": {
            "stmScheduleKey": "standardSchedule",
            "default": true
        }
    },
    "stmSchedules": {
        "standardSchedule": {
            "wcet": "./MiniPipeline_wcet.yaml",
            "hyperepochs": {
                "mainHyperepoch": {
                    "period": 33000000,
                    "epochs": {
                        "mainEpoch": {
                            "period": 33000000,
                            "frames": 1,
                            "passes": [
                                [   

                                    "miniPipeline.lidarSensor",
                                    "miniPipeline.imuSensor",
                                    "miniPipeline.radarSensor0",
                                    "miniPipeline.radarSensor1"
                                ]
                            ]
                        }
                    },
                    "resources": {
                        "machine0.CPU0": []
                    }
                },
                "renderHyperepoch": {
                    "period": 33000000,
                    "epochs": {
                        "renderEpoch": {
                            "period": 33000000,
                            "frames": 1,
                            "passes": [
                                [
                                    "miniPipeline.visualization"
                                ]
                            ]
                        },
                        "camEpoch": {
                            "period": 33000000,
                            "frames": 1,
                            "passes": [
                                [
                                    "miniPipeline.camera0",
                                    "miniPipeline.camera1",
                                    "miniPipeline.camera2",
                                    "miniPipeline.camera3",
                                    "miniPipeline.camera4",
                                    "miniPipeline.camera5",
                                    "miniPipeline.camera6"
                                ]
                            ]
                        }
                    },
                    "resources": {
                        "machine0.CPU1": [],
                        "machine0.CPU2": [],
                        "machine0.CPU3": [],
                        "machine0.CPU4": [],
                        "machine0.CPU5": [],
                        "machine0.CPU6": [],
                        "machine0.CPU7": [],
                        "machine0.CPU8": [],
                        "machine0.dGPU": [],
                        "mainClient.CUDA_STREAM0:dGPU": [],
                        "mainClient.CUDA_MUTEX_LOCK": []
                    }
                }
            }
        }
    },
    "processes": {
        "ssm": {
            "runOn": "machine0",
            "executable": "SSM/vanillassm"
        },
        "schedule_manager": {
            "runOn": "machine0",
            "executable": "ScheduleManager",
            "argv": {
                "--enableScheduleSwitching": "true",
                "--scheduleManagerHostIP": "127.0.0.1",
                "--scheduleManagerHostPort": "4010",
                "--scheduleManagerNumClients": "1"
            }
        },
        "stm_master": {
            "runOn": "machine0",
            "executable": "stm_master",
            "argv": {
                "--timeout-us": "80000000",
                "--log": "stm_sreport.log",
                "--soc": "TegraA",
                "-m": true
            }
        },
        "mainClient": {
            "runOn": "machine0",
            "executable": "LoaderLite",
            "subcomponents": [
                "miniPipeline.camera0",
                "miniPipeline.camera1",
                "miniPipeline.camera2",
                "miniPipeline.camera3",
                "miniPipeline.camera4",
                "miniPipeline.camera5",
                "miniPipeline.camera6",
                "miniPipeline.lidarSensor",
                "miniPipeline.radarSensor0",
                "miniPipeline.radarSensor1",
                "miniPipeline.imuSensor",
                "miniPipeline.visualization"
            ]
        }
    },
    "extraInfo": "./MiniPipelineExtraInfo.json"
}

and modified the sensor section of the rig file by adding:

{
                "name": "lidar:front:center",
                "parameter": "file=irisSimShort_2217.bin,lidar_port=7502,imu_port=7503,session_id=test_log,decoder-path=/path/to/libouster_lidar_plugin_arm.so, transport=someip, device=CUSTOM_EX, ip=169.254.229.245 ,scan-frequency=10,fw-protocol-ver=2217,service-id=41632",
                "properties": null,
                "protocol": "lidar.custom",
                "nominalSensor2Rig_FLU": {
                    "roll-pitch-yaw": [
                        0.0,
                        0.0,
                        -135.0
                    ],
                    "t": [
                        -0.937,
                        -0.701,
                        0.499
                    ]
                }
            }

Once generated both yaml and stm, i tried to lunch it on Orin board but i got the following messages in the launcher.log file.


> [STM ERROR]:[av/stm/runtime/src/client/commands/call.c][opInitCall] [123]: CPU runnable miniPipeline_lidarSensor_dwLidarBatchNode_pass_0 not found
[STM ERROR]:[av/stm/runtime/src/client/thread.c][threadAppendCommands] [323]: Unable to init instruction
[STM ERROR]:[av/stm/runtime/src/client/stm_internal.c][stmClientReceiveCoreCommands] [1078]: Could not append commands to thread.
[STM ERROR]:[av/stm/runtime/src/client/stm.c][stmEnterScheduler] [911]: Could not receive core commands. stmErrorCode=8 (STM_ERROR_GENERIC)
av/stm/runtime/src/client/stm.c:912 assertion failure, errno=11 (Resource temporarily unavailable)

***** Backtrace disabled *****
[2024-01-23T11:45:26.201727Z][INFO][tid:0][Launcher.cpp:1198][Launcher] Got signal: 17 (Child exited), si_pid: 3140
[2024-01-23T11:45:26.201812Z][DEBUG][tid:0][Launcher.cpp:1270][Launcher] Sending SIGINT: /usr/local/driveworks/bin//ScheduleManager, name: schedule_manager
[2024-01-23T11:45:26.201937Z][FATAL][tid:0][Launcher.cpp:1039][Launcher] Process mainClient:3140 terminated by signal: 6 (Aborted)
[2024-01-23T11:45:26.201954Z][ERROR][tid:0][Launcher.cpp:1299][Launcher] There is child process exited abnormally.
[2024-01-23T11:45:26.201968Z][ERROR][tid:0][Launcher.cpp:1392][Launcher] Running processes:
[2024-01-23T11:45:26.201976Z][ERROR][tid:0][Launcher.cpp:1397][Launcher] ssm: 3139
[2024-01-23T11:45:26.201983Z][ERROR][tid:0][Launcher.cpp:1397][Launcher] schedule_manager: 3141
[2024-01-23T11:45:26.201990Z][ERROR][tid:0][Launcher.cpp:1397][Launcher] stm_master: 3142
[2024-01-23T11:45:26.201998Z][ERROR][tid:0][Launcher.cpp:1356][Launcher] Killing all live child processes with SIGTERM…
[2024-01-23T11:45:26.251863Z][FATAL][tid:0][Launcher.cpp:1039][Launcher] Process ssm:3139 terminated by signal: 15 (Terminated)
[2024-01-23T11:45:56.252030Z][ERROR][tid:0][Launcher.cpp:1327][Launcher] sigtimedwait() timeout.
[2024-01-23T11:45:56.252196Z][ERROR][tid:0][Launcher.cpp:1384][Launcher] Killing all live child processes with SIGKILL…
[2024-01-23T11:45:56.254426Z][FATAL][tid:0][Launcher.cpp:1039][Launcher] Process stm_master:3142 terminated by signal: 9 (Killed)
[2024-01-23T11:45:56.258222Z][FATAL][tid:0][Launcher.cpp:1039][Launcher] Process schedule_manager:3141 terminated by signal: 9 (Killed)
[2024-01-23T11:45:56.258260Z][INFO][tid:0][Launcher.cpp:1095][Launcher] waitForChildExit: No more child process!
[2024-01-23T11:45:56.258272Z][ERROR][tid:0][Launcher.cpp:1345][Launcher] All child processes has been killed successfully.
[2024-01-23T11:45:56.258286Z][FATAL][tid:0][Launcher.cpp:1565][Launcher] launcher exit status: 33

The error in bold made me think the .so library which lidar node was recalling, has no implementation for the passes that are described in the json.

Hence the questions:

  • Is there an example of minipepeline which is currently working with LIDAR devices?

  • Is there anything missing in the recalling of the decoder path of the lidar, as shown above?

Thanks in advance for the clarification.

Most of the details look good. However, as per the error log, it seems that you should re-generate the STM scheduling table?

You would need to use descriptionScheduleYamlGenerator.py to generate a new MiniPipeline__standardSchedule.yaml, and then use stmcompiler to generate a new MiniPipeline__standardSchedule.stm based on your changes.

It’s a MUST step, if you haven’t re-generated the STM binary, the application won’t work.

Hi, thanks for this suggestion.
Apparently the yaml generated was not over-written correctly but i made sure to have it fixed now.

I run the example again but i got a new error in the launcher.log:

[2024-01-25T12:25:04.151299Z][DEBUG][tid:0][Launcher.cpp:927][Launcher] All apps are launched.

DefaultLogger: [25-01-2024 13:25:04] TopExecutor: set TopExecutor to cpuset /sys/fs/cgroup/cpuset/rr2init/tasks
DefaultLogger: [25-01-2024 13:25:04] Failed to open cpuset /sys/fs/cgroup/cpuset/rr2init/tasks, errno: No such file or directory (2)
DefaultLogger: [25-01-2024 13:25:04] Failed to set TopExecutor to RR2 init cpuset: /sys/fs/cgroup/cpuset/rr2init/tasks
terminate called after throwing an instance of 'dw::core::ExceptionWithStatusCode<int, dw::core::detail::IntToStr>'
  what():  25: apps/roadrunner-2.0/module/SwcConfigLoader.cpp:1390 DriveWorks Error DW_SAL_CANNOT_CREATE_SENSOR: Failed to create sensor lidar:front:center

***** Backtrace disabled *****
[2024-01-25T12:25:16.25580Z][INFO][tid:0][Launcher.cpp:1198][Launcher] Got signal: 17 (Child exited), si_pid: 8989
[2024-01-25T12:25:16.25679Z][DEBUG][tid:0][Launcher.cpp:1270][Launcher] Sending SIGINT: /usr/local/driveworks/bin//ScheduleManager, name: schedule_manager
[2024-01-25T12:25:16.25791Z][FATAL][tid:0][Launcher.cpp:1033][Launcher] Process mainClient:8989 failed with exit code: 207
[2024-01-25T12:25:16.25817Z][ERROR][tid:0][Launcher.cpp:1299][Launcher] There is child process exited abnormally.
[2024-01-25T12:25:16.25836Z][ERROR][tid:0][Launcher.cpp:1392][Launcher] Running processes:
[2024-01-25T12:25:16.25846Z][ERROR][tid:0][Launcher.cpp:1397][Launcher] 	ssm: 8988
[2024-01-25T12:25:16.25854Z][ERROR][tid:0][Launcher.cpp:1397][Launcher] 	schedule_manager: 8990
[2024-01-25T12:25:16.25861Z][ERROR][tid:0][Launcher.cpp:1397][Launcher] 	stm_master: 8991
[2024-01-25T12:25:16.25871Z][ERROR][tid:0][Launcher.cpp:1356][Launcher] Killing all live child processes with SIGTERM...

DefaultLogger: [25-01-2024 13:25:16] [ScheduleManager] received exit
[2024-01-25T12:25:16.82342Z][FATAL][tid:0][Launcher.cpp:1039][Launcher] Process ssm:8988 terminated by signal: 15 (Terminated)
[2024-01-25T12:26:13.596298Z][ERROR][tid:0][Launcher.cpp:1327][Launcher] sigtimedwait() timeout.
[2024-01-25T12:26:13.609143Z][ERROR][tid:0][Launcher.cpp:1384][Launcher] Killing all live child processes with SIGKILL...
[2024-01-25T12:26:13.610829Z][FATAL][tid:0][Launcher.cpp:1039][Launcher] Process stm_master:8991 terminated by signal: 9 (Killed)
[2024-01-25T12:26:13.625366Z][FATAL][tid:0][Launcher.cpp:1039][Launcher] Process schedule_manager:8990 terminated by signal: 9 (Killed)
[2024-01-25T12:26:13.625433Z][INFO][tid:0][Launcher.cpp:1095][Launcher] waitForChildExit: No more child process!
[2024-01-25T12:26:13.625448Z][ERROR][tid:0][Launcher.cpp:1345][Launcher] All child processes has been killed successfully.
[2024-01-25T12:26:13.625518Z][FATAL][tid:0][Launcher.cpp:1565][Launcher] launcher exit status: 207
[2024-01-25T12:26:13.657441Z][DEBUG][tid:0][Launcher.cpp:1589][Launcher] swc_list.txt content:

The same error is reported in the mainClient0.log :

<13>1 2024-01-25T13:25:14.699911+01:00 - mainClient 8989 - - [0us][DEBUG][tid:rr2_main][SwcConfigLoader.cpp:563][RigSensorInfo] Loaded requiredSensors: {
    ... other sensors ... ,
    "lidar:front:center": [
        {
            "connectedTo": [
                "machine0"
            ],
            "typeIndex": 0
        }
    ]
}
...
...
...
<13>1 2024-01-25T13:25:15.955612+01:00 - mainClient 8989 - - [0us][DEBUG][tid:rr2_main][SwcConfigLoader.cpp:974][RigSensorInfo] initResources name pair: name lidar:front:center sensorId: 24 sensorIdx: 0
<13>1 2024-01-25T13:25:15.955642+01:00 - mainClient 8989 - - [0us][DEBUG][tid:rr2_main][SwcConfigLoader.cpp:1023][RigSensorInfo] initResources: lidar:front:center lidarCount: 0 sensorId : 24
<13>1 2024-01-25T13:25:15.955680+01:00 - mainClient 8989 - - [0us][DEBUG][tid:rr2_main][SensorFactory.cpp:321][NO_TAG] SensorFactory::createSensor() -> lidar.custom, device=CUSTOM_EX,log_level=WARN,ip=169.254.229.245,lidar_port=7502,imu_port=7503,decoder-path=/path/to/lib/libouster_lidar_plugin_arm.so
<13>1 2024-01-25T13:25:15.956020+01:00 - mainClient 8989 - - [0us][DEBUG][tid:rr2_main][SensorFactory.cpp:333][NO_TAG] Found driver: lidar.custom
<11>1 2024-01-25T13:25:15.963394+01:00 - mainClient 8989 - - [0us][ERROR][tid:rr2_main][:0][] [25-01-2024 13:25:15] <11>1 2024-01-25T13:25:15.963435+01:00 - mainClient 8989 - - [0us][ERROR][tid:rr2_main][:0][] Parameter "dip" not provided.<11>1 2024-01-25T13:25:15.966559+01:00 - mainClient 8989 - - [0us][ERROR][tid:rr2_main][Exception.cpp:387][NO_TAG] [25-01-2024 13:25:15] Driveworks exception thrown: DW_SAL_CANNOT_CREATE_SENSOR: LidarCustom: could not create sensor
<11>1 2024-01-25T13:25:15.966584+01:00 - mainClient 8989 - - [0us][ERROR][tid:rr2_main][Exception.cpp:387][NO_TAG] 
<11>1 2024-01-25T13:25:15.967582+01:00 - mainClient 8989 - - [0us][ERROR][tid:rr2_main][TopExecutor.hpp:694][TopExecutor] [25-01-2024 13:25:15] [FaultHandler] Caught signal: 6 sent by pid 8989
<8>1 2024-01-25T13:25:15.967637+01:00 - mainClient 8989 - - [0us][FATAL][tid:rr2_main][FaultHandler.cpp:94][FaultHandler] [25-01-2024 13:25:15] 

***** Backtrace disabled *****
<8>1 2024-01-25T13:25:15.967684+01:00 - mainClient 8989 - - [0us][FATAL][tid:rr2_main][TopExecutor.hpp:634][TopExecutor] [25-01-2024 13:25:15] Caught signal 6 sent by pid 8989 during initialization step 207: Initializing resources
...
...

Recalling that such library is already been proved to work correctly with the lidar_sample_replay.

Do you have any clue about what could be the problem?

Could you review why the dip (Destination IP Address) issue did not occur when running sample_lidar_replay?

Is this still an issue to support? Any result can be shared?

Dear @kayccc
can you please track this issue in Interface between Custom Lidar.so and Drive Orin failed

these are same.

Best regards,
Sushant