Driveworks Samples unable to run and compile/cross-compile on Host PC

Autonomous Vehicles DRIVE AGX Orin DRIVE AGX Orin General
1

Please provide the following info (tick the boxes after creating this topic):
Software Version
DRIVE OS 6.0.8.1
DRIVE OS 6.0.6
DRIVE OS 6.0.5
DRIVE OS 6.0.4 (rev. 1)
DRIVE OS 6.0.4 SDK
other

Target Operating System
Linux
QNX
other

Hardware Platform
DRIVE AGX Orin Developer Kit (940-63710-0010-300)
DRIVE AGX Orin Developer Kit (940-63710-0010-200)
DRIVE AGX Orin Developer Kit (940-63710-0010-100)
DRIVE AGX Orin Developer Kit (940-63710-0010-D00)
DRIVE AGX Orin Developer Kit (940-63710-0010-C00)
DRIVE AGX Orin Developer Kit (not sure its number)
other

SDK Manager Version
2.1.0.11698
1.9.3.10904
other

Host Machine Version
native Ubuntu Linux 20.04 Host installed with SDK Manager
native Ubuntu Linux 20.04 Host installed with DRIVE OS Docker Containers
native Ubuntu Linux 18.04 Host installed with DRIVE OS Docker Containers
other

Issue Description
Host components and Target components downloaded and visible on Host PC. OS Image creation process is successful. Flashing is successful.
CUDA samples build on Host and run successfully on Target. TensorRT samples build on Host and run successfully on Target.

Followed the instructions under [Host System x86] (DriveWorks SDK Reference: Getting Started Using the NVIDIA SDK Manager) for the precompiled samples and notice issue when running precompiled samples. Logs are attached below.

CUDA is installed on the Host PC according to the SDK Manager.

Then tried the instructions to build from source [Samples Compilation From Source] (DriveWorks SDK Reference: Getting Started Using the NVIDIA SDK Manager). At make install the files get written to /usr/local/bin instead of % $DRIVEWORKS_WORKSPACE/build-x86_64-linux-gnu/install/usr/local/driveworks/samples/bin/.

Unclear what step was missed to cause this issue but when I then run % $DRIVEWORKS_WORKSPACE/build-x86_64-linux-gnu/install/usr/local/driveworks/samples/bin/sample_hello_world, I get

% $DRIVEWORKS_WORKSPACE/build-x86_64-linux-gnu/install/usr/local/driveworks/samples/bin/sample_hello_world: No such file or directory

Error String
Driveworks exception thrown: Platform: cannot acquire CUDA context. Error cudaErrorNoDevice: no CUDA-capable device is detected

Logs

daf@daf-rcp:/usr/local/driveworks/bin$ ./sample_hello_world 
*************************************************
Welcome to Driveworks SDK
[19-08-2024 11:54:42] Platform: Detected Generic x86 Platform
[19-08-2024 11:54:42] TimeSource: monotonic epoch time offset is 1724057195759399
[19-08-2024 11:54:42] Adding variable DW_Base:DW_Version
[19-08-2024 11:54:42] Added variable DW_Base:DW_Version
[19-08-2024 11:54:42] Driveworks exception thrown: Platform: cannot acquire CUDA context. Error cudaErrorNoDevice: no CUDA-capable device is detected

Cannot init SDK
3

Dear @adityen.sudhakaran,
Please follow the template at Announcement from NVIDIA: Guidelines to create a new topic when creating a new topic. I have updated your description.

Does your host has a CUDA GPU? You need ampere based GPU to run DW samples on host.

Also, did you set DRIVEWORKS_WORKSPACE environment variable ?

Is there any reason to use DRIVE OS 6.0.6? We recommend to use to latest release i.e. DRIVE OS 6.010.0. Please see Install DRIVE OS 6.0.10.0 for NVIDIA Developer Users | NVIDIA Docs and Requirements for Your Development Environment | NVIDIA Docs for details.

4

Hi SivaRamaKrishna,

Thanks! And apologies.

Yes, the laptop has an NVIDIA GeForce RTX 4080 which is compatible according to the GPU Compute Capability.

CUDA was installed by the SDK manager. DRIVEWORKS_WORKSPACE was set as ~/driveworks and was set correctly.

We were told to install 6.0.6 by our partners but let me explore that possibility with them and get back to you.

In the mean time, despite CUDA being installed by the SDK manager, why is it still not being detected when DriveWorks samples are run?

5

Dear @adityen.sudhakaran,
Could you share the output of the CUDA device query sample? Please see /usr/local/cuda/samples/1_Utilities/deviceQuery

6

@SivaRamaKrishnaNV

This is what I get. Bear in mind that CUDA was installed by the SDK Manager:

daf-rcp@daf-rcp:/usr/local/cuda/samples/1_Utilities/deviceQuery$ ./deviceQuery 
./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

cudaGetDeviceCount returned 100
-> no CUDA-capable device is detected
Result = FAIL
7

Maybe also good to know this:

daf-rcp@daf-rcp:/usr/local/cuda/samples$ lspci | grep -i nvidia
01:00.0 VGA compatible controller: NVIDIA Corporation Device 27a0 (rev a1)
01:00.1 Audio device: NVIDIA Corporation Device 22bc (rev a1)
8

@adityen.sudhakaran did you install NVIDIA drivers ? Could you check if you find nvidia-smi tool on host to confirm. If so, please run nvidia-smi to know DRIVER version.

9

Yes, after installing the driver and running nvidia-smi, I get

NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

Is it possible to have a live debug session via an online meeting?

10

Dear @adityen.sudhakaran,
May I know which CUDA driver version is installed. Can you check rebooting the host after installation?

11

Hi @SivaRamaKrishnaNV

So I finally got some things working so here are the outputs. The only combination that worked on my Host is Cuda 12.2 and nvidia-535.183.01. Not sure if that causes other problems yet, but here we are. I can boot the laptop and get the following outputs.

daf@daf-rcp:~$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Tue_Aug_15_22:02:13_PDT_2023
Cuda compilation tools, release 12.2, V12.2.140
Build cuda_12.2.r12.2/compiler.33191640_0

daf@daf-rcp:~$ nvidia-smi
Wed Aug 21 06:22:59 2024       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01             Driver Version: 535.183.01   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 4080 ...    Off | 00000000:01:00.0 Off |                  N/A |
| N/A   32C    P4              N/A /  50W |     15MiB / 12282MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      1187      G   /usr/lib/xorg/Xorg                            4MiB |
|    0   N/A  N/A      1703      G   /usr/lib/xorg/Xorg                            4MiB |
+---------------------------------------------------------------------------------------+

And finally, the CUDA folder contains:

daf@daf-rcp:~$ ll -h /usr/local/cuda/
total 136K
drwxr-xr-x 15 root root 4,0K aug 20 17:27 ./
drwxr-xr-x 11 root root 4,0K aug 20 17:27 ../
drwxr-xr-x  3 root root 4,0K aug 20 17:27 bin/
drwxr-xr-x  5 root root 4,0K aug 20 17:27 compute-sanitizer/
drwxr-xr-x  3 root root 4,0K aug 20 17:26 doc/
-rw-r--r--  1 root root  160 aug 16  2023 DOCS
-rw-r--r--  1 root root  61K aug 16  2023 EULA.txt
drwxr-xr-x  5 root root 4,0K aug 20 17:43 extras/
drwxr-xr-x  4 root root 4,0K aug 20 17:27 gds/
lrwxrwxrwx  1 root root   28 aug 16  2023 include -> targets/x86_64-linux/include/
lrwxrwxrwx  1 root root   24 aug 16  2023 lib64 -> targets/x86_64-linux/lib/
drwxr-xr-x  7 root root 4,0K aug 20 17:27 libnvvp/
drwxr-xr-x  2 root root 4,0K aug 20 17:27 nsightee_plugins/
drwxr-xr-x  3 root root 4,0K aug 20 17:27 nvml/
drwxr-xr-x  6 root root 4,0K aug 20 17:26 nvvm/
-rw-r--r--  1 root root  524 aug 16  2023 README
lrwxrwxrwx  1 root root   24 aug 16  2023 res -> targets/x86_64-linux/res/
drwxr-xr-x  3 root root 4,0K aug 20 17:27 share/
drwxr-xr-x  2 root root 4,0K aug 20 17:26 src/
drwxr-xr-x  3 root root 4,0K aug 20 17:25 targets/
drwxr-xr-x  2 root root 4,0K aug 20 17:27 tools/
-rw-r--r--  1 root root 3,0K aug 23  2023 version.json
12

Dear @adityen.sudhakaran,
Did you install CUDA 12.2 toolkit manually on host? The DRIVE OS 6.x comes with CUDA 11.4. Is CUDA 11.4 folder and CUDA 11.4 libs exists on host? If so, please make sure to point /usr/local/cuda symlink to /usr/local/cuda-11.4 and test CUDA 11.4 deviceQuery sample to validate if it outputs NVIDIA GPU details. You need this to test applications on host.

If you are just looking to develop the application on host and cross compile the app to test on target, you can use DRIVE OS 6.0.6 docker container to avoid all system state related issues.

13

Hi @SivaRamaKrishnaNV

Yes, I did install 12.2 manually. No, 11.4 is removed completely. Do I install CUDA 11.4 via the SDK manager?

Meanwhile, I heard back from our suppliers who are ok with upgrading to DRIVE OS 6.0.10. Do you recommend that I just install 6.0.10 using the Docker container? This will install CUDA 11.4 on my Host, correct?
Then I point /usr/local/cuda to /usr/local/11.4 and check if deviceQuery sample works?

14

Yes. we recommend to use latest release.

The docker container will have the needed libs and toolchain setup that are needed for application development. It does not install CUDA 11.4 on your host. But it will be part of docker image . Please check DriveWorks SDK Reference: Getting Started Using the NVIDIA DRIVE OS NGC Docker Container if it helps.

15

Hi @SivaRamaKrishnaNV,

So I was able to setup DRIVEOS 6.0.10 with Docker and get it up and running. DriveWorks also works. I tested the sample helloworld application of DriveWorks and the sample_camera_usb. Both worked fine, including with cross-compilation.

I guess using Docker is the easiest way to get everything up and running but the nvidia-driver installation was the most cumbersome part. I found that installing the driver through the Software & Updates application on Ubuntu 20 to be the only thing that worked. Hence the installation of driver-535.

This is my current output of nvidia-smi:

daf@daf-rcp:~$ nvidia-smi
    Thu Aug 22 08:28:07 2024       
    +---------------------------------------------------------------------------------------+
    | NVIDIA-SMI 535.183.01             Driver Version: 535.183.01   CUDA Version: 12.2     |
    |-----------------------------------------+----------------------+----------------------+
    | GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
    |                                         |                      |               MIG M. |
    |=========================================+======================+======================|
    |   0  NVIDIA GeForce RTX 4080 ...    Off | 00000000:01:00.0 Off |                  N/A |
    | N/A   34C    P0              N/A / 150W |     15MiB / 12282MiB |      0%      Default |
    |                                         |                      |                  N/A |
    +-----------------------------------------+----------------------+----------------------+
                                                                                            
    +---------------------------------------------------------------------------------------+
    | Processes:                                                                            |
    |  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
    |        ID   ID                                                             Usage      |
    |=======================================================================================|
    |    0   N/A  N/A      1218      G   /usr/lib/xorg/Xorg                            4MiB |
    |    0   N/A  N/A      1989      G   /usr/lib/xorg/Xorg                            4MiB |
    +---------------------------------------------------------------------------------------+

This is what I get when I run the DriveWorks “Hello World” sample:

nvidia@tegra-ubuntu:~$ /usr/local/driveworks/bin/sample_hello_world 
    *************************************************
    Welcome to Driveworks SDK
    [22-08-2024 13:01:40] Platform: Detected Drive Orin P3710
    [22-08-2024 13:01:40] Adding variable DW_Base:DW_Version
    [22-08-2024 13:01:40] Added variable DW_Base:DW_Version
    [22-08-2024 13:01:40] Platform: number of GPU devices detected 1
    [22-08-2024 13:01:40] Platform: currently selected GPU device 0, Resource Data Dir: trt_08_06_12_04, Arch: ga10b
    [22-08-2024 13:01:40] Platform: currently selected GPU device integrated ID 0
    [22-08-2024 13:01:40] CUDLAEngine:getDLACount: CUDLA version is = 1004000
    [22-08-2024 13:01:40] CUDLAEngine:getDLACount: Number of DLA devices = 2
    [22-08-2024 13:01:40] Context::mountResourceCandidateDataPath resource FAILED to mount from './resources': VirtualFileSystem: Failed to mount './resources/resources.pak'
    [22-08-2024 13:01:40] Context::mountResourceCandidateDataPath resource FAILED to mount from '/home/nvidia/data': VirtualFileSystem: Failed to mount '/home/nvidia/data/resources.pak'
    [22-08-2024 13:01:40] Context::findDataRootInPathWalk data/DATA_ROOT found at: /usr/local/driveworks/data
    [22-08-2024 13:01:40] Context::mountResourceCandidateDataPath resource FAILED to mount from '/usr/local/driveworks/data': VirtualFileSystem: Failed to mount '/usr/local/driveworks/data/resources.pak'
    [22-08-2024 13:01:40] Context::findDataRootInPathWalk data/DATA_ROOT found at: /usr/local/driveworks-5.20/data
    [22-08-2024 13:01:40] Context::mountResourceCandidateDataPath resource FAILED to mount from '/usr/local/driveworks-5.20/data': VirtualFileSystem: Failed to mount '/usr/local/driveworks-5.20/data/resources.pak'
    [22-08-2024 13:01:40] Context::findResourcesPackageInPathWalk: Could not find ./resources/resources.pak in upto 7 parent directories from /usr/local/driveworks/lib/libdw_base.so.5.20
    [22-08-2024 13:01:40] Context::findResourcesPackageInPathWalk: Could not find ./resources/resources.pak in upto 7 parent directories from /usr/local/driveworks-5.20/targets/aarch64-Linux/lib/libdw_base.so.5.20
    [22-08-2024 13:01:40] SDK: No resources(.pak) mounted, some modules will not function properly
    [22-08-2024 13:01:40] [22-08-2024 13:01:40] Initialize DriveWorks SDK v5.20.24
    [22-08-2024 13:01:40] [22-08-2024 13:01:40] Release build with GNU 9.3.0 from buildbrain-branch-0-gb4c0b405b15 against Drive PDK v6.0.10.0
    Context of Driveworks SDK successfully initialized.
    Version: 5.20.24
    GPU devices detected: 1
    [22-08-2024 13:01:40] Platform: currently selected GPU device 0, Resource Data Dir: trt_08_06_12_04, Arch: ga10b
    [22-08-2024 13:01:40] Platform: currently selected GPU device integrated ID 0
    ----------------------------------------------
    CUDA Driver Version / Runtime Version : 12.1 / 11.4
    CUDA Capability Major/Minor version number: 8.7
    Memory Bus Width bits: 256
    L2 Cache Size: 4194304
    Maximum 1D Texture Dimension Size (x): 131072
    Maximum 2D Texture Dimension Size (x,y): 131072, 65536
    Maximum 3D Texture Dimension Size (x,y,z): 16384, 16384, 16384
    Total amount of constant memory bytes: 65536
    Total amount of shared memory per block bytes: 49152
    Total number of registers available per block: 65536
    Warp size: 32
    Maximum number of threads per multiprocessor: 1536
    Maximum number of threads per block: 1024
    Max dimension size of a thread block (x,y,z): 1024,1024,64
    Max dimension size of a grid size (x,y,z): 2147483647,65535,65535
    Maximum memory pitch bytes: 2147483647
    Texture alignment bytes: 512
    Concurrent copy and kernel execution: Yes, copy engines num: 16
    Run time limit on kernels: No
    Integrated GPU sharing Host Memory: Yes
    Support host page-locked memory mapping: Yes
    Device has ECC support: Disabled
    Device supports Unified Addressing (UVA): Yes
    Device PCI Domain ID: 0, Device PCI Bus ID: 0, Device PCI location ID: 0
    Compute Mode: Default (multiple host threads can use ::cudaSetDevice() with device simultaneously)
    Concurrent kernels: 1

    [22-08-2024 13:01:40] [22-08-2024 13:01:40] Releasing Driveworks SDK Context
    Happy autonomous driving!

I will try to integrate some sensors to see if it works. Thanks for your support.

1 Like
17

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.