[BUG] cgf helloworld sample nvsci+interprocess communication hangs with exception error

Required Info:

  • Software Version
    [ x ] DRIVE OS 6.0.8.1
  • Target OS
    [x ] Linux
  • SDK Manager Version
    [x ] 1.9.2.10884
  • Host Machine Version
    [x ] native Ubuntu Linux 20.04 Host installed with DRIVE OS DOCKER Containers

Describe the bug

with helloworld and sum source code provided by driveworks-5.14 release sample, nv_driveworks/driveworks-5.14/samples/src/cgf_nodes/HelloWorldNode.hpp at main · ZhenshengLee/nv_driveworks · GitHub , nv_driveworks/driveworks-5.14/samples/src/cgf_nodes/SumNodeImpl.cpp at main · ZhenshengLee/nv_driveworks · GitHub
and the official cgf helloworld demo guide nv_driveworks/drive-agx-orin-doc/3-drive-works/CGF-presentation.pdf at main · ZhenshengLee/nv_driveworks · GitHub , we run the minimal test for the connection of nvsci, and found the unexpected behavior.

To Reproduce

  1. follow the nvsci doc and add channel config in /etc/nvsciipc.cfg
  2. follow the official pdf and build a cgf app with helloworld and sum process.
  3. edit the json, change the connection from default to nvsci+interprocess, with 3 cases
    caseA, 1 connection with default streamName
    caseB: 2 connections with default streamName
    caseC: 1 connection with custom streamName
  4. compile and run the cgf app, check the log and data consistency

for step0, add the following channel config to /etc/nvsciipc.cfg, use the shell if you want

for i in {0..3..1}; do sudo echo "INTER_PROCESS   cgf_$(($i+0))_0               cgf_$(($i+0))_1               16       4096" | sudo tee -a /etc/nvsciipc.cfg; done
sudo sed -i '$ a INTER_PROCESS   cgf_hello_0               cgf_hello_1               16       4096' /etc/nvsciipc.cfg
sudo sed -i '$ a INTER_PROCESS   cgf_hello_2               cgf_hello_3               16       4096' /etc/nvsciipc.cfg
sudo sed -i '$ a INTER_PROCESS   cgf_hello_4               cgf_hello_5               16       4096' /etc/nvsciipc.cfg
INTER_PROCESS   cgf_0_0               cgf_0_1               16       4096
INTER_PROCESS   cgf_1_0               cgf_1_1               16       4096
INTER_PROCESS   cgf_2_0               cgf_2_1               16       4096
INTER_PROCESS   cgf_3_0               cgf_3_1               16       4096
INTER_PROCESS   cgf_hello_0               cgf_hello_1               16       4096
INTER_PROCESS   cgf_hello_2               cgf_hello_3               16       4096
INTER_PROCESS   cgf_hello_4               cgf_hello_5               16       4096

for step2, caseA, 1 connection with default streamName
graphlet.json

        {
            "src": "helloWorldNode.VALUE_0",
            "dests": {
                "sumNode.VALUE_0": {
                    "mailbox": true,
                    "reuse": true
                }
            },
            "params": {
                "type": "nvsci"
            }
        },
        {
            "src": "helloWorldNode.VALUE_1",
            "dests": {
                "sumNode.VALUE_1": {
                    "mailbox": true,
                    "reuse": true
                }
            }
        },

for step2, caseB, 2 connections with default streamName
graphlet.json

        {
            "src": "helloWorldNode.VALUE_0",
            "dests": {
                "sumNode.VALUE_0": {
                    "mailbox": true,
                    "reuse": true
                }
            },
            "params": {
                "type": "nvsci"
            }
        },
        {
            "src": "helloWorldNode.VALUE_1",
            "dests": {
                "sumNode.VALUE_1": {
                    "mailbox": true,
                    "reuse": true
                }
            },
            "params": {
                "type": "nvsci"
            }
        },

for step2, caseC: 1 connection with custom streamName
graphlet.json

        {
            "src": "helloWorldNode.VALUE_0",
            "dests": {
                "sumNode.VALUE_0": {
                    "mailbox": true,
                    "reuse": true
                }
            },
           "params": {
                "type": "nvsci",
                "srcEndpoint": "cgf_hello_0",
                "id": "cgf_hello_0",
                "destEndpoint": "cgf_hello_1"
            }
        },
        {
            "src": "helloWorldNode.VALUE_1",
            "dests": {
                "sumNode.VALUE_1": {
                    "mailbox": true,
                    "reuse": true
                }
            }
        },

Expected behavior

with helloworld sample, the nvsci+interprocess communication should be available in cgf, in all cases.

Actual behavior

For CaseA, 1 connection with default streamName, the cgf app works well.
For CaseB and CaseC, the cgf app hangs with exception report in launcher.log with DW_INTERNAL_ERROR

DefaultLogger: [14-05-2024 16:21:16] TopExecutor: set TopExecutor to cpuset /sys/fs/cgroup/cpuset/rr2init/tasks
DefaultLogger: [14-05-2024 16:21:16] Failed to open cpuset /sys/fs/cgroup/cpuset/rr2init/tasks, errno: No such file or directory (2)
DefaultLogger: [14-05-2024 16:21:16] Failed to set TopExecutor to RR2 init cpuset: /sys/fs/cgroup/cpuset/rr2init/tasks
terminate called after throwing an instance of 'dw::core::ExceptionWithStatusCode<int, dw::core::detail::IntToStr>'
  what():  14: apps/roadrunner-2.0/framework/runtime/Runtime.cpp:1466 DriveWorks Error DW_INTERNAL_ERROR: node name: top.helloWorldNode output portID: 0output portName: VALUE_0
[STM] Waiting for STM master to start...
[STM] Master launched
[STM] Performing Client Global Shm Init ...
[STM] Waiting for all clients to start...
_ctx=0xffffadc34e10
Stack trace (most recent call last):
#16   Object "/usr/local/driveworks-5.14/bin/LoaderLite", at 0xaaaab8b78d6f, in $x

Additional context

  1. Please the forum reproduce the issue with the official minimal helloworld sample.
  2. Please provide the correct way to use the nvsci channel in dwcgf, and all needed infomation to help us run dwcgf well.

step2, caseD: connection 1:1 with predefined channelName in nvsciipc.cfg
caseD works well.

        {
            "src": "helloWorldNode.VALUE_0",
            "dests": {
                "sumNode.VALUE_0": {
                    "mailbox": true,
                    "reuse": true
                }
            },
            "params": {
                "type": "nvsci",
                "srcEndpoint": "nvscistream_0",
                "destEndpoint": "nvscistream_1"
            }
        },
        {
            "src": "helloWorldNode.VALUE_1",
            "dests": {
                "sumNode.VALUE_1": {
                    "mailbox": true,
                    "reuse": true
                }
            },
            "params": {
                "type": "nvsci",
                "srcEndpoint": "nvscistream_2",
                "destEndpoint": "nvscistream_3"
            }
        },

@SivaRamaKrishnaNV Friendly ping for response.

Dear @lizhensheng,
I will reproduce the issue and update tomorrow EOD

step2, caseE: connection 1:N with predefined channelName in nvsciipc.cfg
caseE hangs too, with different exception.

NOTE: the multipleNode is another instance of sumNode

{
            "src": "helloWorldNode.VALUE_1",
            "dests": {
                "sumNode.VALUE_1": {
                    "mailbox": true,
                    "reuse": true
                },
                "multipleNode.VALUE_1": {
                    "mailbox": true,
                    "reuse": true
                }
            },
            "params": {
                "type": "nvsci",
                "limits": -1,
                "srcEndpoint": "nvscistream_4:nvscistream_6",
                "destEndpoint": "nvscistream_5:nvscistream_7"
            }
        },

the log shows the limits parameter is fault.

- - [1715844068087636us][DEBUG][tid:20][ChannelNvSciStream.hpp:1066][ChannelNvSciStreamProducer]  Producer: Pool creation 
<13>1 2024-05-16T15:21:39.211685+08:00 - helloworld_process0 77951 - - [1715844068087681us][DEBUG][tid:20][ChannelNvSciStream.hpp:1069][ChannelNvSciStreamProducer]  Producer: Creating producer block 
<13>1 2024-05-16T15:21:39.211720+08:00 - helloworld_process0 77951 - - [1715844068087716us][DEBUG][tid:20][ChannelNvSciStream.hpp:1074][ChannelNvSciStreamProducer]  Source (UP) stream creation with ipcEndpoints, scisync, scibuf and ipcblocks
<13>1 2024-05-16T15:21:39.211735+08:00 - helloworld_process0 77951 - - [1715844068087731us][DEBUG][tid:20][ChannelNvSciStream.hpp:1248][ChannelNvSciStreamProducer] Producer: Opening endpoint with name nvscistream_4
<13>1 2024-05-16T15:21:39.211814+08:00 - helloworld_process0 77951 - - [1715844068087809us][DEBUG][tid:20][ChannelNvSciStream.hpp:1251][ChannelNvSciStreamProducer]  Resetting Producer IPC endpoint
<13>1 2024-05-16T15:21:39.211882+08:00 - helloworld_process0 77951 - - [1715844068087877us][DEBUG][tid:20][ChannelNvSciStream.hpp:1272][ChannelNvSciStreamProducer] Connecting ipc/limiter and multicast
<13>1 2024-05-16T15:21:39.212377+08:00 - helloworld_process0 77951 - - [1715844068088372us][DEBUG][tid:20][Exception.cpp:403][NO_TAG] Driveworks exception thrown: DW_INVALID_ARGUMENT: ChannelNvSciStreamParams: limiter maxPackets index out of range
<15>1 2024-05-16T15:21:39.220662+08:00 - helloworld_process0 77951 - - [1715844068096656us][VERBOSE][tid:25][ChannelNvSciStreamUtil.hpp:665][ChannelNvSciStreamBlockWithEventNotifier] Notify start 10
<15>1 2024-05-16T15:21:39.220719+08:00 - helloworld_process0 77951 - - [1715844068096715us][VERBOSE][tid:25][ChannelNvSciStream.cpp:148][NvSciStreamBlockEventQuery] [pid.tid=77951.281461689116928][ID=1000][Block=10][Type=IPCSRC] result NvSciStreamEventType_Disconnected
<11>1 2024-05-16T15:21:39.220748+08:00 - helloworld_process0 77951 - - [1715844068096744us][ERROR][tid:25][ChannelFactory.cpp:259][ChannelFactoryImpl] ChannelFactory::handleEventLoopError: uid: 0, blockType: IPCSRC, status: DW_END_OF_STREAM, sciError: NvSciError_Success, endpoint: nvscistream_4, what: NvSciEventService notify
<13>1 2024-05-16T15:21:39.220771+08:00 - helloworld_process0 77951 - - [1715844068096767us][DEBUG][tid:25][ChannelNvSciStreamUtil.hpp:849][ChannelNvSciStreamEventServiceLoop] Removing block 10

QST: How to properly config the 1:N nvsci channel communication?

Dear @lizhensheng,
Does your repo (GitHub - ZhenshengLee/nv_driveworks: learning materials of driveworks from nvidia drive sdk.) has tested helloWorld CGF sample files to repro the directly. If not, can you push the files? I could not find it in repo.

For security reasons, I cannot push the project files to the github.

nv_driveworks/driveworks-5.14/samples/src/cgf_nodes at main · ZhenshengLee/nv_driveworks (github.com)

NOTE: The files is the same in the sdk /usr/local/driveworks

the buildup tutorial is in nv_driveworks/drive-agx-orin-doc/3-drive-works/CGF-presentation.pdf at main · ZhenshengLee/nv_driveworks (github.com)

NOTE: The pdf is provided by the nvidia developer forum.

Please refer to this topic to get all src files.

As far as I can see from the comment.

Although all src files are provided, the forum still cannot compile and run the app, nveer say reproducing the issue.

Dear @lizhensheng,
Thanks for sharing the src path. Let me relook into it and get back to you if I see any issue.

Dear @lizhensheng,
I see below snippet in the thread you have pointed

subcomponents": {
        "helloWorldNode": {
            "componentType": "../../../nodes/example/helloworld/HelloWorldNode.node.json",
            "parameters": {
                "name": "$paraName"
            }
        },
        "sumNode": {
            "componentType": "../../../nodes/example/helloworld/SumNode.node.json"
        }
    }

I just want to confirm if these node.json files are same as /usr/local/driveworks/samples/src/cgf_nodes/ or you have modified adding any additional parameters/settings to generate new lib with HelloworldNode and SumNode? Last time, I was seeing issue when using the app.json from that thread.
Note that, forum team does not much experiencing in writing CGF apps on own and hence we are asking to share the files and steps. We expect to have minimal changes on our side. The recommendation is to put the files under /usr/local/driveworks/apps as a seperate folder similar to other CGF applications so that we can manage to repro across different machines easily. This is helps to setup the application and reproduce the issue to get insights form core team further.

The same.

The issue cannot repro in my code repo.

As far as I know, there is no helloworld app in the /usr/local/driveworks/apps, so it’s better that there is a helloworld demo in the apps folder.

As far as I know, there is no any complete example of cgf app that include graphs and node srcs.

Dear @lizhensheng,
I have tested your code on freshly flashed target with DRIVE OS 6.0.8.1 and notice the same issue like last time

root@6.0.8.1-0006-build-linux-sdk:/usr/local/driveworks/apps/helloworldDemo# /usr/local/driveworks/tools/descriptionScheduleYamlGenerator/descriptionScheduleYamlGenerator.py --app /usr/local/driveworks/apps/helloworldDemo/DWCGFHelloworld.app.json --output ./DWCGFHelloworld__standardSchedule.yaml
Input Description file path: /usr/local/driveworks/apps/helloworldDemo/DWCGFHelloworld.app.json
Output schedule directory path: /usr/local/driveworks-5.14/apps/helloworldDemo/
=============================
Analysis...
Traceback (most recent call last):
  File "/usr/local/driveworks/tools/descriptionScheduleYamlGenerator/descriptionScheduleYamlGenerator.py", line 3123, in <module>
    main()
  File "/usr/local/driveworks/tools/descriptionScheduleYamlGenerator/descriptionScheduleYamlGenerator.py", line 3081, in main
    systemDescription = SystemDescription(appFile)
  File "/usr/local/driveworks/tools/descriptionScheduleYamlGenerator/descriptionScheduleYamlGenerator.py", line 2999, in __init__
    self.schedules = [
  File "/usr/local/driveworks/tools/descriptionScheduleYamlGenerator/descriptionScheduleYamlGenerator.py", line 3000, in <listcomp>
    ScheduleDescription(self.baseDir, stmSchedule, stmScheduleKey, identifier, stmScheduleKey == defaultStmScheduleKey, self)
  File "/usr/local/driveworks/tools/descriptionScheduleYamlGenerator/descriptionScheduleYamlGenerator.py", line 1437, in __init__
    self.subcomponents[graphletId] = GraphletDescription(graphletId, grapheletFile, self)
  File "/usr/local/driveworks/tools/descriptionScheduleYamlGenerator/descriptionScheduleYamlGenerator.py", line 2780, in __init__
    graphletJson = json.load(gf)
  File "/usr/lib/python3.8/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.8/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 26 column 17 (char 693)
root@6.0.8.1-0006-build-linux-sdk:/usr/local/driveworks/apps/helloworldDemo#
root@6.0.8.1-0006-build-linux-sdk:/usr/local/driveworks/apps/helloworldDemo# vim DWCGFHelloworld.app.json                  root@6.0.8.1-0006-build-linux-sdk:/usr/local/driveworks/apps/helloworldDemo# ls
DWCGFHelloworld.app.json       DWCGFHelloworld_wcet.yaml     HelloWorldNode.node.json  sensor_mappings
DWCGFHelloworld.graphlet.json  DWCGFImagePipeExtraInfo.json  SumNode.node.json

Could you please put the unzip the file(
helloworldDemo.zip (3.8 KB)

) under /usr/local/driveworks/apps folder and test.

app.json and other files are copied from [BUG] cgf helloworld sample socket communication inconsistency behavior in mailbox+reuse connection.

1 Like

OK, I will prepare a repo that is in the /usr/local/driveworks/apps and give a detailed repro steps to you, EOD.

@SivaRamaKrishnaNV

The dwcgf helloworld sample is uploaded in the /usr/local/driveworks/sample, please check the doc to reproduce this cgf-nvsci bug.

git clone -b bug/nvsci https://github.com/ZhenshengLee/nv_driveworks.git

the doc is in nv_driveworks/README_en.md at bug/nvsci · ZhenshengLee/nv_driveworks (github.com)

Please ask any question when you encounter problem reproduce the cgf issue.

Dear @lizhensheng ,
I cloned your repo to 6.0.8.1 docker and followed the steps to compile the samples from nv_driveworks/driveworks-5.14. I enabled only src/framework , src/dwchannel and src/cgf_nodes in nv_driveworks/driveworks-5.14/samples/CMakeListss.txt and notice below issue when running make command.

But If I use /usr/local/driveworks/samples instead of nv_driveworks/driveworks-5.14/samples with cmake, I could cross compile the samples.

root@6.0.8.1-0006-build-linux-sdk:/home/nvidia/GAC/nv_driveworks# cmake -B ./target/dw514/ -DCMAKE_TOOLCHAIN_FILE=/usr/local/driveworks/samples/cmake/Toolchain-V5L.cmake -DVIBRANTE_PDK=/drive/drive-linux -S ./driveworks-5.14/samples
-- VIBRANTE_PDK = /drive/drive-linux
-- Found PDK version 6.0.8.1 (6000801)
-- VIBRANTE_PDK = /drive/drive-linux
-- Found PDK version 6.0.8.1 (6000801)
-- The C compiler identification is GNU 9.3.0
-- The CXX compiler identification is GNU 9.3.0
-- Check for working C compiler: /drive/toolchains/aarch64--glibc--stable-2022.03-1/bin/aarch64-linux-gcc
-- Check for working C compiler: /drive/toolchains/aarch64--glibc--stable-2022.03-1/bin/aarch64-linux-gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /drive/toolchains/aarch64--glibc--stable-2022.03-1/bin/aarch64-linux-g++
-- Check for working CXX compiler: /drive/toolchains/aarch64--glibc--stable-2022.03-1/bin/aarch64-linux-g++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- The CUDA compiler identification is NVIDIA 11.4.409
-- Check for working CUDA compiler: /usr/local/cuda/bin/nvcc
-- Check for working CUDA compiler: /usr/local/cuda/bin/nvcc -- works
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1")
-- Found EGL: /drive/drive-linux/lib-target/libEGL.so
-- Found /drive/drive-linux/lib-target/libEGL.so:
--  - Includes: [/drive/drive-linux/include]
--  - Libraries: [/drive/drive-linux/lib-target/libEGL.so]
-- Found: /drive/drive-linux/lib-target/libdrm.so
-- Header at: /drive/drive-linux/include
-- DW_EXPERIMENTAL_FORCE_EGL set and EGL Support Enabled
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Performing Test C_COMPILER_FLAG_FNO_OMIT_FRAME_POINTER
-- Performing Test C_COMPILER_FLAG_FNO_OMIT_FRAME_POINTER - Success
-- Performing Test CXX_COMPILER_FLAG_FNO_OMIT_FRAME_POINTER
-- Performing Test CXX_COMPILER_FLAG_FNO_OMIT_FRAME_POINTER - Success
-- Performing Test C_COMPILER_FLAG_FNO_TREE_VECTORIZE
-- Performing Test C_COMPILER_FLAG_FNO_TREE_VECTORIZE - Success
-- Performing Test CXX_COMPILER_FLAG_FNO_TREE_VECTORIZE
-- Performing Test CXX_COMPILER_FLAG_FNO_TREE_VECTORIZE - Success
-- Performing Test C_COMPILER_FLAG_FSTACK_PROTECTOR_STRONG
-- Performing Test C_COMPILER_FLAG_FSTACK_PROTECTOR_STRONG - Success
-- Performing Test CXX_COMPILER_FLAG_FSTACK_PROTECTOR_STRONG
-- Performing Test CXX_COMPILER_FLAG_FSTACK_PROTECTOR_STRONG - Success
-- Performing Test CXX_COMPILER_FLAG_WERROR_ALL
-- Performing Test CXX_COMPILER_FLAG_WERROR_ALL - Success
-- Building GLFW for X11 (static)
-- Found X11: /drive/drive-linux/include
-- Looking for XOpenDisplay in /drive/drive-linux/lib-target/libX11.so;/drive/drive-linux/lib-target/libXext.so
-- Looking for XOpenDisplay in /drive/drive-linux/lib-target/libX11.so;/drive/drive-linux/lib-target/libXext.so - found
-- Looking for gethostbyname
-- Looking for gethostbyname - found
-- Looking for connect
-- Looking for connect - found
-- Looking for remove
-- Looking for remove - found
-- Looking for shmat
-- Looking for shmat - found
-- VIBRANTE_PDK_BRANCH = 6.0.8.1
-- **** Please copy the contents of `/home/nvidia/GAC/nv_driveworks/target/dw514/install/usr/local/driveworks/samples/bin' on the host filesystem to `/usr/local/driveworks/samples/bin' on the target filesystem. ****
-- Found CUDART: /usr/local/cuda/targets/aarch64-linux/include
-- Found cuBLAS: /usr/local/cuda/targets/aarch64-linux/include
-- Found NvSCI: /drive/drive-linux/include
-- Found cuDNN: /usr/include/aarch64-linux-gnu (found suitable version "8.9.2", minimum required is "8.6.0")
-- Found TensorRT: /usr/include/aarch64-linux-gnu (found suitable version "8.6.11.4", minimum required is "8.5.10.4")
-- Found cuDLA: /usr/local/cuda/targets/aarch64-linux/include
-- Found cuPVA: /usr/local/driveworks/targets/aarch64-Linux/include/cupva
-- Could NOT find NvMedia (missing: NvMedia_INCLUDE_DIR NvMedia_LIBRARY)
-- Found 'dw/core/base/Version.h' in /usr/local/driveworks-5.14/targets/aarch64-Linux/include
-- Found dwtrace library in /usr/local/driveworks-5.14/targets/aarch64-Linux/lib/libdwtrace.so
-- Found 'dw/core/base/Version.h' in /usr/local/driveworks-5.14/targets/aarch64-Linux/include
-- Found dw_base library in /usr/local/driveworks-5.14/targets/aarch64-Linux/lib/libdw_base.so
-- Found 'dw/core/base/Version.h' in /usr/local/driveworks-5.14/targets/aarch64-Linux/include
-- Found dw_calibration library in /usr/local/driveworks-5.14/targets/aarch64-Linux/lib/libdw_calibration.so
-- Found 'dw/core/base/Version.h' in /usr/local/driveworks-5.14/targets/aarch64-Linux/include
-- Found dw_egomotion library in /usr/local/driveworks-5.14/targets/aarch64-Linux/lib/libdw_egomotion.so
-- Found 'dw/core/base/Version.h' in /usr/local/driveworks-5.14/targets/aarch64-Linux/include
-- Found dw_imageprocessing library in /usr/local/driveworks-5.14/targets/aarch64-Linux/lib/libdw_imageprocessing.so
-- Found 'dw/core/base/Version.h' in /usr/local/driveworks-5.14/targets/aarch64-Linux/include
-- Found dw_pointcloudprocessing library in /usr/local/driveworks-5.14/targets/aarch64-Linux/lib/libdw_pointcloudprocessing.so
-- Found 'dw/core/base/Version.h' in /usr/local/driveworks-5.14/targets/aarch64-Linux/include
-- Found dw_sensors library in /usr/local/driveworks-5.14/targets/aarch64-Linux/lib/libdw_sensors.so
-- Found 'dw/core/base/Version.h' in /usr/local/driveworks-5.14/targets/aarch64-Linux/include
-- Found dw_vehicleio library in /usr/local/driveworks-5.14/targets/aarch64-Linux/lib/libdw_vehicleio.so
-- Found 'dw/core/base/Version.h' in /usr/local/driveworks-5.14/targets/aarch64-Linux/include
-- Found dw_dnn_base library in /usr/local/driveworks-5.14/targets/aarch64-Linux/lib/libdw_dnn_base.so
-- Found 'dwvisualization/core/Visualization.h' in /usr/local/driveworks-5.14/targets/aarch64-Linux/include
-- Found driveworks_visualization library in /usr/local/driveworks-5.14/targets/aarch64-Linux/lib/libdriveworks_visualization.so
-- Found 'dwvisualization/core/Visualization.h' in /usr/local/driveworks-5.14/targets/aarch64-Linux/include
-- Found 'dw/core/base/Version.h' in /usr/local/driveworks-5.14/targets/aarch64-Linux/include
-- Found dwshared library in /usr/local/driveworks-5.14/targets/aarch64-Linux/lib/libdwshared.so
-- Found 'dw/core/DynamicMemory.h' in /usr/local/driveworks-5.14/targets/aarch64-Linux/include
-- Found dwdynamicmemory library in /usr/local/driveworks-5.14/targets/aarch64-Linux/lib/libdwdynamicmemory.so
-- Found 'dwframework/dwnodes/common/factories/DWChannelFactory.hpp' in /usr/local/driveworks-5.14/targets/aarch64-Linux/include
-- Found 'dwcgf/channel/ChannelFactory.hpp' in /usr/local/driveworks-5.14/targets/aarch64-Linux/include
-- Found 'modern-json/json.hpp' in /usr/local/driveworks-5.14/targets/aarch64-Linux/include/3rdparty
-- Found dwpbwire library in /usr/local/driveworks-5.14/targets/aarch64-Linux/lib/libdwpbwire.so
-- Found 'dwframework/dwnodes/common/factories/DWChannelFactory.hpp' in /usr/local/driveworks-5.14/targets/aarch64-Linux/include
-- Found 'dwcgf/channel/ChannelFactory.hpp' in /usr/local/driveworks-5.14/targets/aarch64-Linux/include
-- Found 'modern-json/json.hpp' in /usr/local/driveworks-5.14/targets/aarch64-Linux/include/3rdparty
-- Found dwcgf library in /usr/local/driveworks-5.14/targets/aarch64-Linux/lib/libdwcgf.so
-- Found 'dwframework/dwnodes/common/factories/DWChannelFactory.hpp' in /usr/local/driveworks-5.14/targets/aarch64-Linux/include
-- Found 'dwcgf/channel/ChannelFactory.hpp' in /usr/local/driveworks-5.14/targets/aarch64-Linux/include
-- Found 'modern-json/json.hpp' in /usr/local/driveworks-5.14/targets/aarch64-Linux/include/3rdparty
-- Found dwframework_dwnodes library in /usr/local/driveworks-5.14/targets/aarch64-Linux/lib/libdwframework_dwnodes.so
-- Found 'dwframework/dwnodes/common/factories/DWChannelFactory.hpp' in /usr/local/driveworks-5.14/targets/aarch64-Linux/include
-- Found 'dwcgf/channel/ChannelFactory.hpp' in /usr/local/driveworks-5.14/targets/aarch64-Linux/include
-- Found 'modern-json/json.hpp' in /usr/local/driveworks-5.14/targets/aarch64-Linux/include/3rdparty
-- Found dwframework_dwnodes_channel_packets library in /usr/local/driveworks-5.14/targets/aarch64-Linux/lib/libdwframework_dwnodes_channel_packets.so
-- Found 'dwframework/dwnodes/common/factories/DWChannelFactory.hpp' in /usr/local/driveworks-5.14/targets/aarch64-Linux/include
-- Found 'dwcgf/channel/ChannelFactory.hpp' in /usr/local/driveworks-5.14/targets/aarch64-Linux/include
-- Found 'modern-json/json.hpp' in /usr/local/driveworks-5.14/targets/aarch64-Linux/include/3rdparty
-- Found dwroadcast-frontend library in /usr/local/driveworks-5.14/targets/aarch64-Linux/lib/libdwroadcast-frontend.so
'/usr/local/driveworks/tools/schema/validate_cgfdescriptors.py' '--ignore-order' '--ignore-array-order' '--ignore-indentation' '/home/nvidia/GAC/nv_driveworks/driveworks-5.14/samples/src/cgf_nodes/graphs'
-- ✓ /home/nvidia/GAC/nv_driveworks/driveworks-5.14/samples/src/cgf_nodes/graphs/apps/example/appHelloworld/DWCGFHelloworld.app.json
✓ /home/nvidia/GAC/nv_driveworks/driveworks-5.14/samples/src/cgf_nodes/graphs/apps/example/appHelloworld/DWCGFHelloworld.graphlet.json
✓ /home/nvidia/GAC/nv_driveworks/driveworks-5.14/samples/src/cgf_nodes/graphs/nodes/example/helloworld/HelloWorldNode.node.json
✓ /home/nvidia/GAC/nv_driveworks/driveworks-5.14/samples/src/cgf_nodes/graphs/nodes/example/helloworld/MultipleNode.node.json
✓ /home/nvidia/GAC/nv_driveworks/driveworks-5.14/samples/src/cgf_nodes/graphs/nodes/example/helloworld/SubtractNode.node.json
✓ /home/nvidia/GAC/nv_driveworks/driveworks-5.14/samples/src/cgf_nodes/graphs/nodes/example/helloworld/SumNode.node.json
✓ /home/nvidia/GAC/nv_driveworks/driveworks-5.14/samples/src/cgf_nodes/graphs/extra/appCommon/DWCGFImagePipe.required-sensors.json
✓ /home/nvidia/GAC/nv_driveworks/driveworks-5.14/samples/src/cgf_nodes/graphs/extra/sensor_mappings/pilot.sensor-mappings.json

Validated 8 files against their schema and additional constraints


-- Configuring done
-- Generating done
-- Build files have been written to: /home/nvidia/GAC/nv_driveworks/target/dw514

root@6.0.8.1-0006-build-linux-sdk:/home/nvidia/GAC/nv_driveworks/target/dw514# make -j12
[  3%] Built target lodepng-src
[  5%] Built target cgf_custom_nodes_yamlHelloworld
[  8%] Built target samples_allocator
[ 33%] Built target glfw-x11
[ 37%] Built target cgf_custom_nodes_stmHelloworld
[ 38%] Built target glfw-src
[ 44%] Built target cgf_custom_nodes_htmlHelloworld
[ 79%] Built target samples_framework
[ 81%] Linking CXX executable sample_cgf_dwchannel
[ 83%] Linking CXX shared library libcgf_custom_nodes.so
/drive/toolchains/aarch64--glibc--stable-2022.03-1/bin/../lib/gcc/aarch64-buildroot-linux-gnu/9.3.0/../../../../aarch64-buildroot-linux-gnu/bin/ld: cannot find -ludev
/drive/toolchains/aarch64--glibc--stable-2022.03-1/bin/../lib/gcc/aarch64-buildroot-linux-gnu/9.3.0/../../../../aarch64-buildroot-linux-gnu/bin/ld: cannot find -lusb-1.0
collect2: error: ld returned 1 exit status
make[2]: *** [src/dwchannel/CMakeFiles/sample_cgf_dwchannel.dir/build.make:142: src/dwchannel/sample_cgf_dwchannel] Error 1
make[1]: *** [CMakeFiles/Makefile2:348: src/dwchannel/CMakeFiles/sample_cgf_dwchannel.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
/drive/toolchains/aarch64--glibc--stable-2022.03-1/bin/../lib/gcc/aarch64-buildroot-linux-gnu/9.3.0/../../../../aarch64-buildroot-linux-gnu/bin/ld: cannot find -ludev
/drive/toolchains/aarch64--glibc--stable-2022.03-1/bin/../lib/gcc/aarch64-buildroot-linux-gnu/9.3.0/../../../../aarch64-buildroot-linux-gnu/bin/ld: cannot find -lusb-1.0
collect2: error: ld returned 1 exit status
make[2]: *** [src/cgf_nodes/CMakeFiles/cgf_custom_nodes.dir/build.make:232: src/cgf_nodes/libcgf_custom_nodes.so] Error 1
make[1]: *** [CMakeFiles/Makefile2:460: src/cgf_nodes/CMakeFiles/cgf_custom_nodes.dir/all] Error 2
make: *** [Makefile:130: all] Error 2

I think it is ok to update /usr/local/driveworks/samples/cgf_nodes with your folder nv_driveworks/driveworks-5.14/samples/cgf_nodes . I will check it and update you.