Cannot run TensorRT samples on Thor DRIVE AGX

Autonomous Vehicles DRIVE AGX Thor DRIVE AGX Thor General
1

Description

Using ‘samples’ supplied tensorRT_mode.bin fails with error DW_DNN_INVALID_MODEL

Environment

I am not certain how to determine the following environment parameters; I am using a stock Thor Drive AGX 7.0.3.0-0010.

TensorRT Version:
GPU Type:
Nvidia Driver Version:
CUDA Version:
CUDNN Version:
Operating System + Version:
Python Version (if applicable):
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

Relevant Files

Steps To Reproduce

I have followed the instructions, and used the following to install on my Thor DRIVE AGX:
export IMG=nvcr.io/drive/driveos-sdk/drive-agx-linux-nsr-aarch64-sdk-build-x86
export VER=7.0.3.0-0010
export IMGVER=${IMG}:${VER}
export WORKSPACE=/home/${USER}/nvid_wksp_00
sudo docker run -it --privileged --net=host -v /drive_flashing:/drive_flashing
-v /dev/:/dev/ -v ${WORKSPACE}:/home/nvidia/ ${IMGVER}
cd $NV_WORKSPACE/drive-foundation
export BDVAR=p3960-10-sw03
./make/bind_partitions -b $BDVAR drive_av.linux -p dev_nsr
cd $NV_WORKSPACE/drive-foundation/tools/flashtools/bootburn/
First time:
./bootburn.py -b $BDVAR -x /dev/ttyACM2 --init_persistent_partitions
Subsequent times:
./bootburn.py -b $BDVAR -x /dev/ttyACM2
From there, I have downloaded the following from NVIDIA NGC

  • driveworks-data-all-7.0.3.0-406622460.tar.gz
  • driveworks-samples-linux-nsr-aarch64-7.0.3.0-406622460.tar.gz
  • driveworks-stm-samples-linux-nsr-aarch64-7.0.3.0-406622460.tar.gz
    I transfer (scp) these files to the Thor, and extract in place:
    sudo tar --keep-directory-symlink --no-overwrite-dir -xzf
    driveworks-linux-nsr-aarch64/driveworks-data-all-.tar.gz -C /
    sudo tar --keep-directory-symlink --no-overwrite-dir -xzf
    driveworks-linux-nsr-aarch64/driveworks-samples-linux-nsr-aarch64-    .tar.gz -C /
    sudo tar --keep-directory-symlink --no-overwrite-dir -xzf
    driveworks-linux-nsr-aarch64/driveworks-stm-samples-linux-nsr-aarch64-*.tar.gz -C /
    However, when I run the example sample_object_detector_tracker:
    export BASE=/usr/local/driveworks/
    export APP=${BASE}/bin/sample_object_detector_tracker
    export VID=${BASE}/data/samples/sfm/triangulation/video_0.h264
    export MDL=${BASE}/data/samples/detector/ampere-integrated/tensorRT_model.bin
    export MDL=${BASE}/data/samples/detector/ampere-discrete/tensorRT_model.bin
    export MDL=${BASE}/data/samples/detector/blackwell-integrated/tensorRT_model.bin
    ${APP} --video=${VID} --tensorRT_model=${MDL}
    – using any of the three models present – I get the same error:
    DW][ERROR][getBasePathFor] Warning: SamplesDataPath::getBasePathFor(): ‘DATA_ROOT’ not found in search paths [/usr/local/driveworks/data | /usr/local/driveworks/bin.runfiles/av/data | /usr/local/driveworks/bin/sample_object_detector_tracker.runfiles/av/data | /usr/local/driveworks/bin/data | /usr/local/driveworks/data | /usr/local/data | /usr/data]
    [DW][ERROR][getBasePathFor] Warning: SamplesDataPath::getBasePathFor(): ‘DATA_ROOT’ not found in search paths [/usr/local/driveworks/data | /usr/local/driveworks/bin.runfiles/av/data | /usr/local/driveworks/bin/sample_object_detector_tracker.runfiles/av/data | /usr/local/driveworks/bin/data | /usr/local/driveworks/data | /usr/local/data | /usr/data]
    [DW][ERROR][getBasePathFor] Warning: SamplesDataPath::getBasePathFor(): ‘DATA_ROOT’ not found in search paths [/usr/local/driveworks/data | /usr/local/driveworks/bin.runfiles/av/data | /usr/local/driveworks/bin/sample_object_detector_tracker.runfiles/av/data | /usr/local/driveworks/bin/data | /usr/local/driveworks/data | /usr/local/data | /usr/data]
    [DW][ERROR][createSensor] No output-format specified. Specify processing pipelines to use in rig-params
    [21-05-2026 03:49:35] IRuntime::deserializeCudaEngine: Error Code 1: Internal Error (Failed due to an old deserialization call on a newer plan file. This might happen when the plan file was built from an older TensorRT version. You can use trtexec --getPlanVersionOnly to check the version of TensorRT that was used to create the plan file.)
    [21-05-2026 03:49:35] Driveworks exception thrown: DW_DNN_INVALID_MODEL: DNN: Unable to load model.
    I get the same error if I instead: sudo -E ${APP} --video=${VID} --tensorRT_model=${MDL}
    Is this example supposed to work?
    Are these not the correct pre-built samples to use?
    Are the supplied TensorRT models not usable?
    Am I missing a step?

stderr:

[DW][ERROR][getBasePathFor] Warning: SamplesDataPath::getBasePathFor(): ‘DATA_ROOT’ not found in search paths [/usr/local/driveworks/data | /usr/local/driveworks/bin.runfiles/av/data | /usr/local/driveworks/bin/sample_object_detector_tracker.runfiles/av/data | /usr/local/driveworks/bin/data | /usr/local/driveworks/data | /usr/local/data | /usr/data]
[DW][ERROR][getBasePathFor] Warning: SamplesDataPath::getBasePathFor(): ‘DATA_ROOT’ not found in search paths [/usr/local/driveworks/data | /usr/local/driveworks/bin.runfiles/av/data | /usr/local/driveworks/bin/sample_object_detector_tracker.runfiles/av/data | /usr/local/driveworks/bin/data | /usr/local/driveworks/data | /usr/local/data | /usr/data]
[DW][ERROR][getBasePathFor] Warning: SamplesDataPath::getBasePathFor(): ‘DATA_ROOT’ not found in search paths [/usr/local/driveworks/data | /usr/local/driveworks/bin.runfiles/av/data | /usr/local/driveworks/bin/sample_object_detector_tracker.runfiles/av/data | /usr/local/driveworks/bin/data | /usr/local/driveworks/data | /usr/local/data | /usr/data]
[DW][ERROR][createSensor] No output-format specified. Specify processing pipelines to use in rig-params
[21-05-2026 03:49:35] IRuntime::deserializeCudaEngine: Error Code 1: Internal Error (Failed due to an old deserialization call on a newer plan file. This might happen when the plan file was built from an older TensorRT version. You can use trtexec --getPlanVersionOnly to check the version of TensorRT that was used to create the plan file.)
[21-05-2026 03:49:35] Driveworks exception thrown: DW_DNN_INVALID_MODEL: DNN: Unable to load model.

**** Stack trace ****
0xffff28c89c0c : dw::core::ExceptionWithStackTrace::traceStack()+188 at /usr/local/driveworks/bin/../lib/libdwshared.so.7.0
0xffffa6de4388 : ???()+0 at /usr/local/driveworks/bin/../lib/libdw_dnn_base.so.7.0
0xffffa6e10994 : dw::dnn::StdTRTNetworkWrapper::deserializeTensorRTEngineFromMemory(dw::core::span<char const, 1ul> const&)+484 at /usr/local/driveworks/bin/../lib/libdw_dnn_base.so.7.0
0xffffa6e125c0 : dw::dnn::StdTRTNetworkWrapper::loadNetworkFromFile(char const*)+96 at /usr/local/driveworks/bin/../lib/libdw_dnn_base.so.7.0
0xffffa6e15764 : dw::dnn::StdTRTNetworkWrapper::StdTRTNetworkWrapper(dw::core::Context*, char const*, dw::dnn::EDNNPrecision, dwDNNPluginConfiguration const*, dwProcessorType, bool)+116 at /usr/local/driveworks/bin/../lib/libdw_dnn_base.so.7.0
0xffffa6e0f5b4 : dw::dnn::INetwork::createEngineFromFile(dw::core::Context*, char const*, dwDNNPluginConfiguration const*, dwProcessorType, unsigned int, unsigned int)+180 at /usr/local/driveworks/bin/../lib/libdw_dnn_base.so.7.0
0xffffa6e09844 : dwDNN_initializeTensorRTFromFileWithEngineId()+148 at /usr/local/driveworks/bin/../lib/libdw_dnn_base.so.7.0
0xaaaad2dc2bdc : ???()+0 at /usr/local/driveworks//bin/sample_object_detector_tracker
0xaaaad2db0574 : ???()+0 at /usr/local/driveworks//bin/sample_object_detector_tracker
0xaaaad2dd2424 : ???()+0 at /usr/local/driveworks//bin/sample_object_detector_tracker
0xaaaad2db0088 : ???()+0 at /usr/local/driveworks//bin/sample_object_detector_tracker
0xffff233184c4 : ???()+0 at /lib/aarch64-linux-gnu/libc.so.6
0xffff23318598 : __libc_start_main()+152 at /lib/aarch64-linux-gnu/libc.so.6
0xaaaad2db02f0 : ???()+0 at /usr/local/driveworks//bin/sample_object_detector_tracker

terminate called after throwing an instance of ‘std::runtime_error’
what(): [2026-05-21 03:49:35] [DW][ERROR][dwDNN_initializeTensorRTFromFile] DW Error DW_DNN_INVALID_MODEL executing DW function:
dwDNN_initializeTensorRTFromFile(&m_dnn, tensorRTModel.c_str(), nullptr, DW_PROCESSOR_TYPE_GPU, m_sdk)
at dw/samples/dnn/sample_object_detector_tracker/ObjectTrackerApp.cpp:495

stdout:

[DW][INFO] ProgramArguments: Missing argument ‘dwTracePath’ requested
[21-05-2026 03:49:35] Adding variable DW_Base:DW_Version
[21-05-2026 03:49:35] Added variable DW_Base:DW_Version
[21-05-2026 03:49:35] Platform: number of GPU devices detected 1
[21-05-2026 03:49:35] Platform: currently selected GPU device 0, Resource Data Dir: trt_10_08_10_05, Arch: ga10b
[21-05-2026 03:49:35] Platform: currently selected GPU device integrated ID 0
[21-05-2026 03:49:35] Context::mountResourceCandidateDataPath resource FAILED to mount from ‘./resources’: VirtualFileSystem: Failed to mount ‘./resources/resources.pak’
[21-05-2026 03:49:35] Context::mountResourceCandidateDataPath resource FAILED to mount from ‘/usr/local/driveworks/data’: VirtualFileSystem: Failed to mount ‘/usr/local/driveworks/data/resources.pak’
[21-05-2026 03:49:35] Context::findDataRootInPathWalk: Could not find ./data/DATA_ROOT in upto 7 parent directories from /usr/local/driveworks/bin/../lib/libdw_base.so.7.0
[21-05-2026 03:49:35] Context::findDataRootInPathWalk: Could not find ./data/DATA_ROOT in upto 7 parent directories from /usr/local/driveworks/targets/aarch64-Linux/lib/libdw_base.so.7.0
[21-05-2026 03:49:35] Context::findResourcesPackageInPathWalk: Could not find ./resources/resources.pak in upto 7 parent directories from /usr/local/driveworks/bin/../lib/libdw_base.so.7.0
[21-05-2026 03:49:35] Context::findResourcesPackageInPathWalk: Could not find ./resources/resources.pak in upto 7 parent directories from /usr/local/driveworks/targets/aarch64-Linux/lib/libdw_base.so.7.0
[21-05-2026 03:49:35] SDK: No resources(.pak) mounted, some modules will not function properly
[21-05-2026 03:49:35] [21-05-2026 03:49:35] Initialize DriveWorks SDK v7.0.3
[21-05-2026 03:49:35] [21-05-2026 03:49:35] Release build with GNU 13.2.0 from against Drive PDK v7.0.3.0
[21-05-2026 03:49:35] SensorFactory::createSensor() → time.nvpps, nvpps-device=/dev/nvpps0
[21-05-2026 03:49:35] Found driver: time.nvpps
[21-05-2026 03:49:35] TimeSensorNVPPS: initialized with no UTC time reference.
[21-05-2026 03:49:35] output-timestamp parameter is deprecated and will be removed in next release[21-05-2026 03:49:35] SensorFactory: Override default timeSensor with the user passed one
[21-05-2026 03:49:35] EndpointNVPPS: started on /dev/nvpps0
[21-05-2026 03:49:35] SensorFactory::loadPluginLibraryHelper:Succeed to load dynamic lib - /usr/local/driveworks/bin/../lib/libsensor_plugin_public_radar.so
[21-05-2026 03:49:35] Initialize DriveWorks VisualizationSDK v7.0.3
[21-05-2026 03:49:35] Initialize DriveWorksGL SDK v7.0.3
[21-05-2026 03:49:35] egl::Display: found 1 EGL devices
[21-05-2026 03:49:35] egl::Display: use drm device: /dev/dri/card1
[21-05-2026 03:49:35] SensorFactory::createSensor() → camera.virtual, video=/usr/local/driveworks//data/samples/sfm/triangulation/video_0.h264
[21-05-2026 03:49:35] Found driver: camera.virtual
[21-05-2026 03:49:35] DecoderVideo::setGlobalMaxReferenceFrames:: Setting Deocder Max reference to: 4
[21-05-2026 03:49:35] output-timestamp parameter is deprecated and will be removed in next release[21-05-2026 03:49:35] CameraBase: pool size set to 16
[21-05-2026 03:49:35] GenericImage: generic image cuda
[21-05-2026 03:49:35] SyncDriver: GPU device available, enabling CUDA sync
[21-05-2026 03:49:35] SyncDriver: enableCUDAPresync, object successfully created
[21-05-2026 03:49:35] SyncDriverInitializer::updateHasTaskStatus: cannot get num task slots from NvSciSyncAttrList 256
[21-05-2026 03:49:35] GenericImage: generic image nvm
[21-05-2026 03:49:35] GenericImage: generic image nvm
[21-05-2026 03:49:35] GenericImage: generic image nvm
[21-05-2026 03:49:35] GenericImage: generic image nvm
[21-05-2026 03:49:35] GenericImage: generic image nvm
[21-05-2026 03:49:35] GenericImage: generic image nvm
[21-05-2026 03:49:35] GenericImage: generic image nvm
[21-05-2026 03:49:35] Try allocate texture cache! filterMode: 1 readMode: 1 accessMode: 3 fromImage: 1
[21-05-2026 03:49:35] Cache count is: 1
[21-05-2026 03:49:35] Try allocate texture cache! filterMode: 1 readMode: 1 accessMode: 3 fromImage: 1
[21-05-2026 03:49:35] Cache count is: 2
[21-05-2026 03:49:35] SyncDriver: GPU device available, enabling CUDA sync
[21-05-2026 03:49:35] SyncDriver: enableCUDAPresync, object successfully created
[21-05-2026 03:49:35] SyncDriverInitializer::updateHasTaskStatus: cannot get num task slots from NvSciSyncAttrList 256
[21-05-2026 03:49:35] SyncDriverInitializer::updateHasTaskStatus: cannot get num task slots from NvSciSyncAttrList 256
[21-05-2026 03:49:35] DecoderNVMedia: reset
[21-05-2026 03:49:35] GenericImage: generic image cuda
[21-05-2026 03:49:35] GenericImage: generic image nvm
[21-05-2026 03:49:35] output-timestamp parameter is deprecated and will be removed in next release[21-05-2026 03:49:35] GenericImage: generic image cuda
[21-05-2026 03:49:35] ImageStreamer(NvMedia → CUDA)
[DW][INFO] Camera image with 1280x800 at 30 FPS
[21-05-2026 03:49:35] GenericImage: generic image cuda
[21-05-2026 03:49:35] Try allocate texture cache! filterMode: 1 readMode: 1 accessMode: 3 fromImage: 1
[21-05-2026 03:49:35] Cache count is: 3
[21-05-2026 03:49:35] GenericImage: generic image cpu
[21-05-2026 03:49:35] Loaded engine size: 29 MiB

3

Hi Jim,

Moving this discussion over to the Drive AGX section of the forum for better visibility.

Thanks,

Aharpster

5

Dear @jim.fron ,
DNN support in DriveWorks is deprecated in DOS 7.0.3 release. I expect the sample not work.
Do you see issue with other driveworks samples listed at Samples — NVIDIA DriveOS 7.0.3 Linux SDK Developer Guide

6

Well, that is disappointing, but at least it is an answer. I will give up on the DNN samples. The other samples – at least the ones I have tried so far – do seem to work.

Thank you for responding.