CGF stuck when executing dwFeatureHistoryArray_copyAsync from GPU to CPU

Autonomous Vehicles DRIVE AGX Orin DRIVE AGX Orin General
1

Please provide the following info (tick the boxes after creating this topic):
Software Version
DRIVE OS 6.0.8.1

Target Operating System
Linux

Hardware Platform
DRIVE AGX Orin Developer Kit (940-63710-0010-300)

SDK Manager Version
1.9.3.10904

Host Machine Version
native Ubuntu Linux 20.04 Host installed with DRIVE OS Docker Containers

Hi, I am trying to create a CGF node that performs object detection and tracking. I called this node detectAndTrack.

Since /usr/local/driveworks-5.14/samples/src/dnn/sample_object_detector_tracker/main.cpp already does this, I decide to copy the its core parts into a CGF node. So, onInitialize method has become part of the detectAndTrack constructor, onProcess has become the detectAndTrack::processPass and so no.

Since I implemented this node step by step I am sure that the detection part works correctly, but as soon as I added the tracking part the application gets stuck. By adding several logs to the code I realized that the line where the app gets stuck is this one:

CHECK_DW_ERROR(dwFeatureHistoryArray_copyAsync(&m_featureHistoryCPU, &m_featureHistoryGPU, 0));

This is how I initialized and use m_featureHistoryCPU and m_featureHistoryGPU, which is the same way as sample_object_detector_tracker/main.cpp from above does:

From detectAndTrackImpl.hpp

dwContextHandle_t m_sdk = DW_NULL_HANDLE;
uint32_t m_maxFeatureCount;
uint32_t m_historyCapacity;
dwFeatureHistoryArray m_featureHistoryCPU = {};
dwFeatureHistoryArray m_featureHistoryGPU = {};

From detectAndTrackImpl.cpp

// detectAndTrackImpl::initialize() {
dwContextParameters sdkParams = {};
CHECK_DW_ERROR(dwInitialize(&m_sdk, DW_VERSION, &sdkParams));
...
m_historyCapacity = 3;
m_maxFeatureCount = 2000;
CHECK_DW_ERROR(dwFeatureHistoryArray_createNew(&m_featureHistoryCPU, m_maxFeatureCount, m_historyCapacity, DW_MEMORY_TYPE_CPU, nullptr, m_sdk));
CHECK_DW_ERROR(dwFeatureHistoryArray_createNew(&m_featureHistoryGPU, m_maxFeatureCount, m_historyCapacity, DW_MEMORY_TYPE_CUDA, nullptr, m_sdk));
// } // end detectAndTrackImpl::initialize()

// detectAndTrackImpl::trackFeatures() { // called from runTracker(), called from processPass()
// **This is the line where the code gets stuck:**
CHECK_DW_ERROR(dwFeatureHistoryArray_copyAsync(&m_featureHistoryCPU, &m_featureHistoryGPU, 0));
// } // end detectAndTrackImpl::trackFeatures()

Here there are CGF logs:

launcher.log

...
[22-01-2024 12:20:40] [INFO] Current State of schedule_manager: PRE_READY

[22-01-2024 12:20:40] [INFO] Current State of schedule_manager: READY

MAX96712: Revision 5 detected

MAX96712: Revision 5 detected

MAX96712 Link 0: PHY optimization was enabled

MAX96712 Link 0: PHY optimization was enabled

MAX96712 Link 1: PHY optimization was enabled

MAX96712 Link 1: PHY optimization was enabled

MAX96712 Link 2: PHY optimization was enabled

MAX96712 Link 3: PHY optimization was enabled

MAX96712 Link 3: PHY optimization was enabled

MAX9295: Revision 8 detected!

MAX9295: Revision 8 detected!

Sensor AR0820 GRBG Rev 2.1 detected!

Sensor AR0820 GRBG Rev 2.1 detected!

MAX9295: Revision 8 detected!

MAX9295: Revision 8 detected!

Sensor AR0820 GRBG Rev 2.1 detected!

Sensor AR0820 GRBG Rev 2.1 detected!

MAX9295: Revision 8 detected!

MAX9295: Revision 8 detected!

Sensor AR0820 GRBG Rev 2.1 detected!

Sensor AR0820 GRBG Rev 2.1 detected!

MAX9295: Revision 8 detected!

Sensor AR0820 GRBG Rev 2.1 detected!

SSM_LATENCY::4 mainClient

SSM_LATENCY::0 1849ms

SSM_LATENCY::9 mainClient_runStateMachine

SSM_LATENCY::0 1850ms

[2024-01-22T11:21:06.114649Z][INFO][tid:0][Launcher.cpp:1198][Launcher] Got signal: 15 (Terminated), si_pid: 5569

[2024-01-22T11:21:06.114766Z][INFO][tid:0][Launcher.cpp:1205][Launcher] This is user exit signal: 15. Wait for gracefully exit done.

[STM ERROR]:[av/stm/runtime/src/core/synchronization.c][stmCondSemTimedWaitForValue] [122]: CondSem Wait Timed Out. err=5

[STM ERROR]:[av/stm/runtime/src/client/sync.c][stmFenceWait] [72]: Could not wait on stmOsFence f47 in iteration: 1 stmError=5

[STM ERROR]:[av/stm/runtime/src/client/commands/wof.c][opExecuteWof] [28]: [Thread pool6] [Time 12:22:02.640778320] Fence timeout: f47. Iteration: 1

[STM ERROR]:[av/stm/runtime/src/core/synchronization.c][stmCondSemTimedWaitForValue] [122]: CondSem Wait Timed Out. err=5

[STM ERROR]:[av/stm/runtime/src/client/stm.c][stmEnterScheduler] [1105]: Thread 0 [pool1] did not terminate (currently at command #86)

[STM ERROR]:[av/stm/runtime/src/client/sync.c][stmFenceWait] [72]: Could not wait on stmOsFence f47 in iteration: 1 stmError=5

[STM] Starting thread pool0

[STM] Starting thread pool1

[STM] Starting thread pool2

[STM] Starting thread pool3

[STM] Starting thread pool4

[STM] Starting thread pool5

[STM] Starting thread pool6

[STM] Starting thread pool7

[STM] Starting thread pool8

[STM] Current thread stopCount = 1, time: 12:21:06.134758756

[STM] Thread pool0 exiting, thread exit count 2, time: 12:21:06.134781508

[STM] Thread pool6 exiting, thread exit count 6, time: 12:22:02.640794064

[STM INFO]:[av/stm/runtime/src/client/commands/cusub.c][opPrintCusub] [CORE 1] [Time 12:22:02.640874833] CUSUB Stuck at: f43, CUDA_STREAM0, miniPipeline_detectAndTrack_pass_1

[STM ERROR]:[av/stm/runtime/src/client/commands/wof.c][opExecuteWof] [28]: [Thread pool7] [Time 12:22:02.640911890] Fence timeout: f47. Iteration: 1

[STM ERROR]:[av/stm/runtime/src/core/synchronization.c][stmCondSemTimedWaitForValue] [122]: CondSem Wait Timed Out. err=5

[STM ERROR]:[av/stm/runtime/src/client/sync.c][stmFenceWait] [72]: Could not wait on stmOsFence f47 in iteration: 1 stmError=5

[STM ERROR]:[av/stm/runtime/src/client/commands/wof.c][opExecuteWof] [28]: [Thread pool8] [Time 12:22:02.641022259] Fence timeout: f47. Iteration: 1

[STM ERROR]:[av/stm/runtime/src/core/synchronization.c][stmCondSemTimedWaitForValue] [122]: CondSem Wait Timed Out. err=5

[STM ERROR]:[av/stm/runtime/src/client/sync.c][stmFenceWait] [72]: Could not wait on stmOsFence f47 in iteration: 1 stmError=5

[STM ERROR]:[av/stm/runtime/src/client/commands/wof.c][opExecuteWof] [28]: [Thread pool4] [Time 12:22:02.641105396] Fence timeout: f47. Iteration: 1

[STM ERROR]:[av/stm/runtime/src/client/stm.c][stmEnterScheduler] [1105]: Thread 0 [pool2] did not terminate (currently at command #13)

[STM] Thread pool7 exiting, thread exit count 7, time: 12:22:02.640952594

[STM] Thread pool8 exiting, thread exit count 8, time: 12:22:02.641042355

[STM] Thread pool4 exiting, thread exit count 9, time: 12:22:02.641125076

[STM INFO]:[av/stm/runtime/src/client/commands/wof.c][opPrintWof] [CORE 2] [Time 12:22:02.641175605] WOF Stuck at: f17

[STM ERROR]:[av/stm/runtime/src/client/stm.c][stmEnterScheduler] [1105]: Thread 0 [pool3] did not terminate (currently at command #3)

[STM INFO]:[av/stm/runtime/src/client/commands/wof.c][opPrintWof] [CORE 3] [Time 12:22:02.641254582] WOF Stuck at: f16

[STM ERROR]:[av/stm/runtime/src/client/stm.c][stmEnterScheduler] [1105]: Thread 0 [pool5] did not terminate (currently at command #0)

[STM INFO]:[av/stm/runtime/src/client/commands/wof.c][opPrintWof] [CORE 5] [Time 12:22:02.641299062] WOF Stuck at: f47

FATAL: exception not rethrown

FATAL: exception not rethrown

***** Backtrace disabled *****

***** Backtrace disabled *****

[2024-01-22T11:22:03.247126Z][INFO][tid:0][Launcher.cpp:1198][Launcher] Got signal: 17 (Child exited), si_pid: 5596

[2024-01-22T11:22:03.247219Z][DEBUG][tid:0][Launcher.cpp:1270][Launcher] Sending SIGINT: /usr/local/driveworks/bin//ScheduleManager, name: schedule_manager

[2024-01-22T11:22:03.247367Z][FATAL][tid:0][Launcher.cpp:1039][Launcher] Process mainClient:5596 terminated by signal: 6 (Aborted)

[2024-01-22T11:22:03.247393Z][ERROR][tid:0][Launcher.cpp:1299][Launcher] There is child process exited abnormally.

[2024-01-22T11:22:03.247415Z][ERROR][tid:0][Launcher.cpp:1392][Launcher] Running processes:

[2024-01-22T11:22:03.247426Z][ERROR][tid:0][Launcher.cpp:1397][Launcher] ssm: 5595

[2024-01-22T11:22:03.247436Z][ERROR][tid:0][Launcher.cpp:1397][Launcher] schedule_manager: 5597

[2024-01-22T11:22:03.247445Z][ERROR][tid:0][Launcher.cpp:1397][Launcher] stm_master: 5598

[2024-01-22T11:22:03.247456Z][ERROR][tid:0][Launcher.cpp:1356][Launcher] Killing all live child processes with SIGTERM...

[STM][ERROR] sem_timedwait failed; errno: 4 (Interrupted system call)

[STM ERROR]:[av/stm/runtime/src/core/comm.c][stmReceiveStateData] [63]: recv failed to receive state data

[STM ERROR]:[av/stm/runtime/src/core/state.c][recvGlobalScheduleState] [701]: Could not receive updated schedule state from Master

av/stm/runtime/src/core/state.c:702 assertion failure, errno=4 (Interrupted system call)

[2024-01-22T11:22:03.257316Z][FATAL][tid:0][Launcher.cpp:1039][Launcher] Process schedule_manager:5597 terminated by signal: 6 (Aborted)

[2024-01-22T11:22:03.298195Z][FATAL][tid:0][Launcher.cpp:1039][Launcher] Process ssm:5595 terminated by signal: 15 (Terminated)

[2024-01-22T11:22:33.298373Z][ERROR][tid:0][Launcher.cpp:1327][Launcher] sigtimedwait() timeout.

[2024-01-22T11:22:33.298499Z][ERROR][tid:0][Launcher.cpp:1384][Launcher] Killing all live child processes with SIGKILL...

[2024-01-22T11:22:33.300750Z][FATAL][tid:0][Launcher.cpp:1039][Launcher] Process stm_master:5598 terminated by signal: 9 (Killed)

[2024-01-22T11:22:33.300803Z][INFO][tid:0][Launcher.cpp:1095][Launcher] waitForChildExit: No more child process!

[2024-01-22T11:22:33.300859Z][ERROR][tid:0][Launcher.cpp:1345][Launcher] All child processes has been killed successfully.

[2024-01-22T11:22:33.300910Z][FATAL][tid:0][Launcher.cpp:1565][Launcher] launcher exit status: 33

[2024-01-22T11:22:33.300985Z][DEBUG][tid:0][Launcher.cpp:1589][Launcher] swc_list.txt content:

line 1 : mainClient,127.0.0.1

line 2 : schedule_manager,127.0.0.1

line 3 :

[2024-01-22T11:22:33.301222Z][ERROR][tid:0][Launcher.cpp:1601][Launcher] Failed to remove SWC/SSM temp files: Inappropriate ioctl for device

mainClient_0.log
mainClient_0.log (4.5 MB)

ssm.log

...
Finding port: schedule_manager

FROM: SSM

FROM: mainClient

FROM: schedule_manager

<13>1 2024-01-22T12:20:40.438044+01:00 - ssm 5595 - - [0us][DEBUG][tid:0][vanilla_main.cpp:180][SSM] [SSM] clients ready

<13>1 2024-01-22T12:20:40.438107+01:00 - ssm 5595 - - [0us][DEBUG][tid:0][vanilla_main.cpp:90][SSM] [PERF] SSMStart: [1705922440438]

<13>1 2024-01-22T12:20:40.438205+01:00 - ssm 5595 - - [0us][DEBUG][tid:0][vanilla_main.cpp:104][SSM] [SSM] run pre init

<13>1 2024-01-22T12:20:40.438258+01:00 - ssm 5595 - - [0us][DEBUG][tid:0][vanilla_main.cpp:98][SSM] Current State of SSM: PRE_INIT

<13>1 2024-01-22T12:20:40.438306+01:00 - ssm 5595 - - [0us][DEBUG][tid:0][vanilla_main.cpp:90][SSM] [PERF] InitPhaseComplete: SSM => PRE_INIT [1705922440438]

<13>1 2024-01-22T12:20:40.441962+01:00 - ssm 5595 - - [0us][DEBUG][tid:1][vanilla_main.cpp:98][SSM] SSM Server Init Signalling: 2 2

<13>1 2024-01-22T12:20:40.442114+01:00 - ssm 5595 - - [0us][DEBUG][tid:2][vanilla_main.cpp:90][SSM] [PERF] InitPhaseComplete: mainClient => PRE_INIT [1705922440441979]

<13>1 2024-01-22T12:20:40.448514+01:00 - ssm 5595 - - [0us][DEBUG][tid:2][vanilla_main.cpp:90][SSM] [PERF] InitPhaseComplete: schedule_manager => PRE_INIT [1705922440448456]

<13>1 2024-01-22T12:20:40.452248+01:00 - ssm 5595 - - [0us][DEBUG][tid:0][vanilla_main.cpp:106][SSM] [SSM] pre init completed

<13>1 2024-01-22T12:20:40.452269+01:00 - ssm 5595 - - [0us][DEBUG][tid:0][vanilla_main.cpp:108][SSM] [SSM] run init

<13>1 2024-01-22T12:20:40.452294+01:00 - ssm 5595 - - [0us][DEBUG][tid:0][vanilla_main.cpp:90][SSM] [PERF] InitBroadcast: SSM => INIT [1705922440452]

<13>1 2024-01-22T12:20:40.452312+01:00 - ssm 5595 - - [0us][DEBUG][tid:0][vanilla_main.cpp:98][SSM] InitBroadcast: SSM => INIT [1705922440452] [mainClient,schedule_manager,]

<13>1 2024-01-22T12:20:40.468919+01:00 - ssm 5595 - - [0us][DEBUG][tid:2][vanilla_main.cpp:90][SSM] [PERF] InitPhaseComplete: schedule_manager => INIT [1705922440468843]

FROM: SSM

<13>1 2024-01-22T12:20:40.502534+01:00 - ssm 5595 - - [0us][DEBUG][tid:0][vanilla_main.cpp:98][SSM] Current State of SSM: INIT

<13>1 2024-01-22T12:20:40.502652+01:00 - ssm 5595 - - [0us][DEBUG][tid:0][vanilla_main.cpp:90][SSM] [PERF] InitPhaseComplete: SSM => INIT [1705922440502]

<13>1 2024-01-22T12:20:40.502737+01:00 - ssm 5595 - - [0us][DEBUG][tid:1][vanilla_main.cpp:98][SSM] SSM Server Init Signalling: 2 2

<13>1 2024-01-22T12:20:40.561680+01:00 - ssm 5595 - - [0us][DEBUG][tid:2][vanilla_main.cpp:90][SSM] [PERF] InitPhaseComplete: mainClient => INIT [1705922440561591]

<13>1 2024-01-22T12:20:40.563378+01:00 - ssm 5595 - - [0us][DEBUG][tid:0][vanilla_main.cpp:110][SSM] [SSM] init completed

<13>1 2024-01-22T12:20:40.563401+01:00 - ssm 5595 - - [0us][DEBUG][tid:0][vanilla_main.cpp:112][SSM] [SSM] run post init

<13>1 2024-01-22T12:20:40.563431+01:00 - ssm 5595 - - [0us][DEBUG][tid:0][vanilla_main.cpp:90][SSM] [PERF] InitBroadcast: SSM => POST_INIT [1705922440563]

<13>1 2024-01-22T12:20:40.563450+01:00 - ssm 5595 - - [0us][DEBUG][tid:0][vanilla_main.cpp:98][SSM] InitBroadcast: SSM => POST_INIT [1705922440563] [mainClient,schedule_manager,]

FROM: schedule_manager

FROM: SSM

FROM: schedule_manager

FROM: mainClient

<13>1 2024-01-22T12:20:40.580311+01:00 - ssm 5595 - - [0us][DEBUG][tid:2][vanilla_main.cpp:90][SSM] [PERF] InitPhaseComplete: schedule_manager => POST_INIT [1705922440580229]

<13>1 2024-01-22T12:20:40.613735+01:00 - ssm 5595 - - [0us][DEBUG][tid:0][vanilla_main.cpp:98][SSM] Current State of SSM: POST_INIT

<13>1 2024-01-22T12:20:40.613836+01:00 - ssm 5595 - - [0us][DEBUG][tid:0][vanilla_main.cpp:90][SSM] [PERF] InitPhaseComplete: SSM => POST_INIT [1705922440613]

<13>1 2024-01-22T12:20:40.613910+01:00 - ssm 5595 - - [0us][DEBUG][tid:1][vanilla_main.cpp:98][SSM] SSM Server Init Signalling: 2 2

<13>1 2024-01-22T12:20:40.641722+01:00 - ssm 5595 - - [0us][DEBUG][tid:2][vanilla_main.cpp:90][SSM] [PERF] InitPhaseComplete: mainClient => POST_INIT [1705922440641640]

<13>1 2024-01-22T12:20:40.644187+01:00 - ssm 5595 - - [0us][DEBUG][tid:0][vanilla_main.cpp:114][SSM] [SSM] post init completed

<13>1 2024-01-22T12:20:40.644212+01:00 - ssm 5595 - - [0us][DEBUG][tid:0][vanilla_main.cpp:116][SSM] [SSM] run pre ready

<13>1 2024-01-22T12:20:40.644237+01:00 - ssm 5595 - - [0us][DEBUG][tid:0][vanilla_main.cpp:90][SSM] [PERF] InitBroadcast: SSM => PRE_READY [1705922440644]

<13>1 2024-01-22T12:20:40.644254+01:00 - ssm 5595 - - [0us][DEBUG][tid:0][vanilla_main.cpp:98][SSM] InitBroadcast: SSM => PRE_READY [1705922440644] [mainClient,schedule_manager,]

<13>1 2024-01-22T12:20:40.661532+01:00 - ssm 5595 - - [0us][DEBUG][tid:2][vanilla_main.cpp:90][SSM] [PERF] InitPhaseComplete: schedule_manager => PRE_READY [1705922440661452]

<13>1 2024-01-22T12:20:40.662404+01:00 - ssm 5595 - - [0us][DEBUG][tid:2][vanilla_main.cpp:90][SSM] [PERF] InitPhaseComplete: mainClient => PRE_READY [1705922440662328]

<13>1 2024-01-22T12:20:40.662448+01:00 - ssm 5595 - - [0us][DEBUG][tid:1][vanilla_main.cpp:98][SSM] SSM Server Init Signalling: 2 2

<13>1 2024-01-22T12:20:40.662509+01:00 - ssm 5595 - - [0us][DEBUG][tid:0][vanilla_main.cpp:98][SSM] Current State of SSM: PRE_READY

<13>1 2024-01-22T12:20:40.662573+01:00 - ssm 5595 - - [0us][DEBUG][tid:0][vanilla_main.cpp:90][SSM] [PERF] InitPhaseComplete: SSM => PRE_READY [1705922440662]

<13>1 2024-01-22T12:20:40.662587+01:00 - ssm 5595 - - [0us][DEBUG][tid:0][vanilla_main.cpp:118][SSM] [SSM] pre ready completed

<13>1 2024-01-22T12:20:40.662595+01:00 - ssm 5595 - - [0us][DEBUG][tid:0][vanilla_main.cpp:120][SSM] [SSM] run ready

<13>1 2024-01-22T12:20:40.662611+01:00 - ssm 5595 - - [0us][DEBUG][tid:0][vanilla_main.cpp:90][SSM] [PERF] InitBroadcast: SSM => READY [1705922440662]

<13>1 2024-01-22T12:20:40.662626+01:00 - ssm 5595 - - [0us][DEBUG][tid:0][vanilla_main.cpp:98][SSM] InitBroadcast: SSM => READY [1705922440662] [mainClient,schedule_manager,]

FROM: schedule_manager

FROM: SSM

<13>1 2024-01-22T12:20:40.681854+01:00 - ssm 5595 - - [0us][DEBUG][tid:2][vanilla_main.cpp:90][SSM] [PERF] InitPhaseComplete: schedule_manager => READY [1705922440681779]

<13>1 2024-01-22T12:20:40.712816+01:00 - ssm 5595 - - [0us][DEBUG][tid:0][vanilla_main.cpp:98][SSM] Current State of SSM: READY

<13>1 2024-01-22T12:20:40.712899+01:00 - ssm 5595 - - [0us][DEBUG][tid:0][vanilla_main.cpp:90][SSM] [PERF] InitPhaseComplete: SSM => READY [1705922440712]

<13>1 2024-01-22T12:20:40.712984+01:00 - ssm 5595 - - [0us][DEBUG][tid:1][vanilla_main.cpp:98][SSM] SSM Server Init Signalling: 2 2

<13>1 2024-01-22T12:20:42.532238+01:00 - ssm 5595 - - [0us][DEBUG][tid:2][vanilla_main.cpp:90][SSM] [PERF] InitPhaseComplete: mainClient => READY [1705922442532144]

<13>1 2024-01-22T12:20:42.536820+01:00 - ssm 5595 - - [0us][DEBUG][tid:0][vanilla_main.cpp:122][SSM] [SSM] ready completed

<13>1 2024-01-22T12:20:42.536867+01:00 - ssm 5595 - - [0us][DEBUG][tid:0][vanilla_main.cpp:90][SSM] [PERF] InitBroadcast: SSM => ENTER_STATE_MACHINE [1705922442536]

<13>1 2024-01-22T12:20:42.536886+01:00 - ssm 5595 - - [0us][DEBUG][tid:0][vanilla_main.cpp:98][SSM] InitBroadcast: SSM => ENTER_STATE_MACHINE [1705922442536] [mainClient,schedule_manager,]

<13>1 2024-01-22T12:20:42.537026+01:00 - ssm 5595 - - [0us][DEBUG][tid:0][vanilla_main.cpp:189][SSM] [SSM] Starting SSM...

<13>1 2024-01-22T12:20:42.537105+01:00 - ssm 5595 - - [0us][DEBUG][tid:0][vanilla_main.cpp:90][SSM] [PERF] StateChange: SSM => Manual [1705922442537]

<13>1 2024-01-22T12:20:42.537186+01:00 - ssm 5595 - - [0us][DEBUG][tid:0][vanilla_main.cpp:98][SSM] Broadcasting activate: SSM ==> mainClient

<13>1 2024-01-22T12:20:42.537228+01:00 - ssm 5595 - - [0us][DEBUG][tid:0][vanilla_main.cpp:98][SSM] Broadcasting activate: SSM ==> schedule_manager

<13>1 2024-01-22T12:20:42.542367+01:00 - ssm 5595 - - [0us][DEBUG][tid:2][vanilla_main.cpp:90][SSM] [PERF] StateChange: mainClient => INIT_CLONE [1705922442542296]

<13>1 2024-01-22T12:20:42.542420+01:00 - ssm 5595 - - [0us][DEBUG][tid:2][vanilla_main.cpp:90][SSM] [PERF] StateChange: mainClient => Manual [1705922442542374]

stm_master.log

[STM ERROR]:[av/stm/runtime/src/core/state.c][clientBarrierWait] [180]: 1 clients have already exited
[STM ERROR]:[av/stm/runtime/src/core/state.c][setExitState] [237]: Error during clients exit sync, time: 12:22:03.248164689
[STM ERROR]:[av/stm/runtime/src/core/state.c][stmSetState] [634]: Incorrect state transition request., time: 12:22:03.302991867

detectAndTrack.node.json

{
    "comment": "Generated by the nodedescriptor tool based on data provided by the C++ API of the node class",
    "generated": true,
    "library": "libminipipeline_nodes.so",
    "name": "dw::framework::detectAndTrack",
    "inputPorts": {
        "IN_IMG": {
            "type": "dwImageHandle_t",
            "bindingRequired": true
        }
    },
    "outputPorts": {
        "BOX_ARR": {
            "type": "YoloScoreRectArray",
            "bindingRequired": true
        },
        "BOX_NUM": {
            "type": "uint32_t"
        }
    },
    "parameters": {},
    "passes": [
        {
            "name": "SETUP",
            "processorTypes": [
                "CPU"
            ]
        },
        {
            "name": "PROCESS",
            "processorTypes": [
                "GPU"
            ]
        },
        {
            "name": "TEARDOWN",
            "processorTypes": [
                "CPU"
            ]
        }
    ]
}

If you need further information, let me know.

Thank you.

3

Dear @vincenzo.longobardi,
Thanks for trying object detection and tracking .
So you don’t see any issue and object detection happening in the displaying on GUI window?

Could you share your code. I will repro and debug further.

4

Dear @SivaRamaKrishnaNV ,

thank you for your help.
You can find the code on my repo.

When I run the application:

  • If the object detection is enabled and the tracking is disabled, I see the image from the camera and the proper bounding box on it;
  • If the both object detection and tracking are enabled, I see a black screen and the application does not die, but it gets stuck. The application only closes if I kill it.
5

Dear @vincenzo.longobardi,
Thank you for sharing the repo. I will check and provide you update.

6

Dear @SivaRamaKrishnaNV,
do you have any update? This problem is blocking for me.

Thank you.

8

Dear @vincenzo.longobardi,
Per cgf_driveworks/README.txt at main · alexshade15/cgf_driveworks · GitHub , I just need to copy irisSimShort_2217.bin minipipeline dataset folder. I don’t need any additional files like video* from there?

10

Dear @SivaRamaKrishnaNV,

Yes, you just need to copy irisSimShort_2217.bin. No additional files are needed. Thank you.

11

Dear @vincenzo.longobardi,
cgf_driveworks/PerceptionPipeV0_10/dataset/rig.json at main · alexshade15/cgf_driveworks · GitHub seems to be using live camera? It is different from /usr/local/driveworks-5.14/data/samples/minipipeline/dataset/rig.json ? I hit input resolution related issue when trying to use /usr/local/driveworks-5.14/data/samples/minipipeline/dataset/rig.json . Minisample uses 1920x1208 where as your sample seems to 3848 * 2168 (from cgf_driveworks/PerceptionPipeV0_10/applications/PipelineV0_3.app.json at main · alexshade15/cgf_driveworks · GitHub) .

12

Hi @SivaRamaKrishnaNV

I think yes, this example does use live camera, I can see JSON in cgf_driveworks/PerceptionPipeV0_10/dataset/rig.json at main · alexshade15/cgf_driveworks · GitHub
F008A120RM0AV2 this camera is used with “height”: “2168”, “width”: “3848”.

13

Dear @sushant.bahadure,
Is it possible to update the JSON with recorded video for debugging purpose on our side?

14

Dear @SivaRamaKrishnaNV @vincenzo.longobardi ,
I think this could be possible to work with prerecorded video with JSON update.

15

Hi @SivaRamaKrishnaNV ,

gentle reminder, do let us know if you need anything.

this is bit blocking.

best regards,
Sushant Bahadure

16

Dear @sushant.bahadure,
As asked please provide recorded videos(as resolution used different from minipipeline config), updated config and verify if the issue to debug it on ourside.

17

Dear @SivaRamaKrishnaNV ,

You can use this video. It has the same resolution we are using.
/usr/local/driveworks-5.14/data/samples/cgf/trafficlightturning-hyperion8/camera_front_wide_120fov.mp4

18

Dear @vincenzo.longobardi,
Ok. I will check using that video and update you the status

19

@SivaRamaKrishnaNV were you able to reproduce this unexpected behavior? A gentle reminder.

20

Hi @sushant.bahadure ,
I used the /usr/local/driveworks-5.14/data/samples/cgf/trafficlightturning-hyperion8/camera_front_wide_120fov.mp4 for all the camera instance in the rig.json. When Launched the app to reproduce the issue, I notice issues related to X11. I am checking with engineering team to fix it repro the issue.

Could you confirm, if using /usr/local/driveworks-5.14/data/samples/cgf/trafficlightturning-hyperion8/camera_front_wide_120fov.mp4 worked for you? If so, could you update the repo with corresponding changes?

21

Hi @SivaRamaKrishnaNV,
I’m sorry for the delay.

I confirm that by using /usr/local/driveworks-5.14/data/samples/cgf/trafficlightturning-hyperion8/camera_front_wide_120fov.mp4 I get the same error.

I updated the repo with the following main changes:

  • Removed unused camera and sensor (lidar, radar, etc) from app, graphlets and rig.json. They are not the root cause of this problem;
  • Update the rig.json as requested.

Since I did the test on the real device, in case you are running this example on docker, you have to make sure that X11 forwarding is correctly done.

22

Dear @SivaRamaKrishnaNV ,

a friendly reminder.

Best Regards,
Sushant Bahadure

23

Dear @sushant.bahadure,
I replaced MiniPipeline.graphlet.json , PipelineV0_3.app.json , PipelineV0_3__standardSchedule.yaml, MiniPipeline.required-sensors.json , PipelineV0_3__standardSchedule.stm rig.json
Is it enough? Is your local repo in sync with shared repo? I hit a new issue now. Not sure why I see radar sensor in this use case. Do you see miniPipeline_radarSensor1_radarNode_pass_0 is called in your logs?

[STM ERROR]:[av/stm/runtime/src/client/commands/call.c][opInitCall] [123]: CPU runnable miniPipeline_radarSensor1_radarNode_pass_0 not found
[STM ERROR]:[av/stm/runtime/src/client/thread.c][threadAppendCommands] [323]: Unable to init instruction
[STM ERROR]:[av/stm/runtime/src/client/stm_internal.c][stmClientReceiveCoreCommands] [1078]: Could not append commands to thread.
[STM ERROR]:[av/stm/runtime/src/client/stm.c][stmEnterScheduler] [911]: Could not receive core commands. stmErrorCode=8 (STM_ERROR_GENERIC)
av/stm/runtime/src/client/stm.c:912 assertion failure, errno=11 (Resource temporarily unavailable)