Running precompiled Sample: "sample_camera_replay" results in Error: Error cudaErrorUnknown: unknown error

Please provide the following info (tick the boxes after creating this topic):
Software Version
DRIVE OS 6.0.4 SDK
DRIVE OS 6.0.4 (rev. 1)
other

Target Operating System
Linux
QNX
other

Hardware Platform
DRIVE AGX Orin Developer Kit (940-63710-0010-D00)
DRIVE AGX Orin Developer Kit (940-63710-0010-C00)
DRIVE AGX Orin Developer Kit (not sure its number)
other

SDK Manager Version
1.8.3.10426
other (1.8.4.10431)

Host Machine Version
native Ubuntu Linux 20.04 Host installed with SDK Manager
native Ubuntu Linux 20.04 Host installed with DRIVE OS Docker Containers
native Ubuntu Linux 18.04 Host installed with DRIVE OS Docker Containers
other

Hello Community,
I am trying to run the “sample_camera_replay” Sample on my Linux host machine, but I get the following output:

$ /usr/local/driveworks/bin/sample_camera_replay 
ProgramArguments: Missing argument 'dwTracePath' requested
[14-10-2022 12:56:13] Platform: Detected Generic x86 Platform
[14-10-2022 12:56:13] TimeSource: monotonic epoch time offset is 1665744621036525
[14-10-2022 12:56:13] Adding variable DW_Base:DW_Version
[14-10-2022 12:56:13] Addded variable DW_Base:DW_Version
[14-10-2022 12:56:13] Platform: number of GPU devices detected 1
[14-10-2022 12:56:13] Platform: currently selected GPU device 0, Resource Data Dir: trt_08_04_11_06, Arch: tu1xx-discrete
[14-10-2022 12:56:13] Platform: currently selected GPU device discrete ID 0
[14-10-2022 12:56:13] Context::mountResourceCandidateDataPath resource FAILED to mount from './resources': VirtualFileSystem: Failed to mount './resources/resources.pak'
[14-10-2022 12:56:13] Context::mountResourceCandidateDataPath resource FAILED to mount from '/usr/local/driveworks-5.6/targets/aarch64-Linux/include/dwshared/dwfoundation/dw/cuda/misc/data': VirtualFileSystem: Failed to mount '/usr/local/driveworks-5.6/targets/aarch64-Linux/include/dwshared/dwfoundation/dw/cuda/misc/data/resources.pak'
[14-10-2022 12:56:13] Context::findDataRootInPathWalk data/DATA_ROOT found at: /usr/local/driveworks-5.6/bin/../data
[14-10-2022 12:56:13] Context::mountResourceCandidateDataPath resource FAILED to mount from '/usr/local/driveworks-5.6/bin/../data': VirtualFileSystem: Failed to mount '/usr/local/driveworks-5.6/bin/../data/resources.pak'
[14-10-2022 12:56:13] Context::findDataRootInPathWalk data/DATA_ROOT found at: /usr/local/driveworks-5.6/data
[14-10-2022 12:56:13] Context::mountResourceCandidateDataPath resource FAILED to mount from '/usr/local/driveworks-5.6/data': VirtualFileSystem: Failed to mount '/usr/local/driveworks-5.6/data/resources.pak'
[14-10-2022 12:56:13] Context::findResourcesPackageInPathWalk: Could not find ./resources/resources.pak in upto 7 parent directories from /usr/local/driveworks-5.6/bin/../lib/libdw_base.so.5.6
[14-10-2022 12:56:13] Context::findResourcesPackageInPathWalk: Could not find ./resources/resources.pak in upto 7 parent directories from /usr/local/driveworks-5.6/targets/x86_64-Linux/lib/libdw_base.so.5.6
[14-10-2022 12:56:13] SDK: No resources(.pak) mounted, some modules will not function properly
[14-10-2022 12:56:13] TimeSource: monotonic epoch time offset is 1665744621036525
[14-10-2022 12:56:13] Initialize DriveWorks SDK v5.6.184
[14-10-2022 12:56:13] Release build with GNU 9.3.0 from buildbrain-branch-0-g17b6da04cd1
[14-10-2022 12:56:13] Initialize DriveWorks VisualizationSDK v5.6.184
[14-10-2022 12:56:13] Initialize DriveWorksGL SDK v5.6.184
[14-10-2022 12:56:13] GL-SDK: initialize OpenGL
[14-10-2022 12:56:13] SensorFactory::createSensor() -> camera.virtual, video=/usr/local/driveworks-5.6/data/samples/sfm/triangulation/video_0.h264
[14-10-2022 12:56:13] CameraVirtual: defaulting to non SIPL
[14-10-2022 12:56:13] CameraBase: pool size set to 8
[14-10-2022 12:56:13] CameraNVCUVID: no seek table found at /usr/local/driveworks-5.6/data/samples/sfm/triangulation/video_0.h264.seek, seeking is not available.
Camera image with 1280x800 at 30.000000 FPS
[14-10-2022 12:56:13] Context: create, error returned by cudaGLGetDevices. GL interop will be disabled.
[14-10-2022 12:56:13] Driveworks exception thrown: . Line: src/dw/cuda/CudaEventQueue.cpp:63. Error cudaErrorUnknown: unknown error

terminate called after throwing an instance of 'std::runtime_error'
  what():  [2022-10-14 12:56:13] DW Error DW_CUDA_ERROR executing DW function:
 dwImageStreamerGL_initialize(&m_image2GL, &from, DW_IMAGE_GL, m_context)
 at samples/sensors/camera/camera_replay/main.cpp:144
Aborted (core dumped)

NVIDIA-SMI:

:~$ nvidia-smi 
Fri Oct 14 13:03:12 2022       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.141.03   Driver Version: 470.141.03   CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Quadro T1000        Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   46C    P0     9W /  N/A |      9MiB /  3914MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1927      G   /usr/lib/xorg/Xorg                  4MiB |
|    0   N/A  N/A      3408      G   /usr/lib/xorg/Xorg                  4MiB |
+-----------------------------------------------------------------------------+

When I open the source Code in VSC, I found out, that ist has trouble finding the cuda_runtime_api.h
I did set the following in .bashrc after trying some fix ideas from the internet:

export CUDA_HOME="/usr/local/cuda" # change to your path
export CUDA_TOOLKIT_ROOT_DIR=$CUDA_HOME
export LD_LIBRARY_PATH="$CUDA_HOME/extras/CUPTI/lib64:$LD_LIBRARY_PATH"
export LIBRARY_PATH=$CUDA_HOME/lib64:$LIBRARY_PATH
export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH
export CFLAGS="-I$CUDA_HOME/include $CFLAGS"

export CUDA_INSTALL_DIR=$CUDA_HOME
# export CUDNN_INSTALL_DIR="/usr/include/cudnn.h"

Now I don’t know how to continue, or where to look for. Any idea is helpful, thanks for looking into.

To update anyone looking into this, I tried updating my driver to 515. But the error is unchanged.

Dear @christopher.tuch,
Is this still an issue?

Hi @SivaRamaKrishnaNV,

yes it is still an issue.

Dear @christopher.tuch,
Do you have any host with Ampere GPU?

Dear @SivaRamaKrishnaNV,
I think i got a Turing GPU since I have a Quadro T1000, please correct me if I am wrong with this.

Dear @christopher.tuch,
Ampere GPU is recommended for DRIVE OS 6.0.4.

I found a solution here CUDA 11.6 OpenGL interoperability broken?
I just set the env variable before running the sample and the it works. I think its fine setting it permanent

__NV_PRIME_RENDER_OFFLOAD=1 __GLX_VENDOR_LIBRARY_NAME=nvidia ./camera_replay

OR (set permanent)

echo "export __NV_PRIME_RENDER_OFFLOAD=1 __GLX_VENDOR_LIBRARY_NAME=nvidia" >> ~/.bashrc

then
/.camera_replay works directly

sounds like it wrongly used iGPU to run the application, right? Thanks for sharing back.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.