NVMAP_IOC_QUERY_HEAP_PARAMS failed [Inappropriate ioctl for device] - Cuda Initialization Error

Description

Cuda crash dump on trtexec after upgrade to LT4.4 from LT4.3.

NVMAP_IOC_QUERY_HEAP_PARAMS failed [Inappropriate ioctl for device]
Cuda failure: initialization error
Aborted (core dumped)

Followed these steps:
https://docs.nvidia.com/jetson/jetpack/install-jetpack/index.html#upgrade-jetpack

Environment

TensorRT Version: 7.1.0.16
GPU Type: Nvidia Xavier
Nvidia Driver Version:
CUDA Version: 10.2.89
CUDNN Version: 8.0.0.145
Operating System + Version: Ubuntu 18.04.4 LTS
Python Version (if applicable): 3.6
TensorFlow Version (if applicable):(tf)1.15.2+nv20.4, (tf-gpu) 1.15.0+nv20.1.tf1,
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag): Baremetal

Relevant Files

Steps To Reproduce

nvidia@nvidia-desktop:~/butterfly$ /usr/src/tensorrt/bin/trtexec --explicitBatch --onnx=frozen_graph2.onnx
&&&& RUNNING TensorRT.trtexec # /usr/src/tensorrt/bin/trtexec --explicitBatch --onnx=frozen_graph2.onnx
[06/17/2020-21:20:08] [I] === Model Options ===
[06/17/2020-21:20:08] [I] Format: ONNX
[06/17/2020-21:20:08] [I] Model: frozen_graph2.onnx
[06/17/2020-21:20:08] [I] Output:
[06/17/2020-21:20:08] [I] === Build Options ===
[06/17/2020-21:20:08] [I] Max batch: explicit
[06/17/2020-21:20:08] [I] Workspace: 16 MB
[06/17/2020-21:20:08] [I] minTiming: 1
[06/17/2020-21:20:08] [I] avgTiming: 8
[06/17/2020-21:20:08] [I] Precision: FP32
[06/17/2020-21:20:08] [I] Calibration:
[06/17/2020-21:20:08] [I] Safe mode: Disabled
[06/17/2020-21:20:08] [I] Save engine:
[06/17/2020-21:20:08] [I] Load engine:
[06/17/2020-21:20:08] [I] Builder Cache: Enabled
[06/17/2020-21:20:08] [I] NVTX verbosity: 0
[06/17/2020-21:20:08] [I] Inputs format: fp32:CHW
[06/17/2020-21:20:08] [I] Outputs format: fp32:CHW
[06/17/2020-21:20:08] [I] Input build shapes: model
[06/17/2020-21:20:08] [I] Input calibration shapes: model
[06/17/2020-21:20:08] [I] === System Options ===
[06/17/2020-21:20:08] [I] Device: 0
[06/17/2020-21:20:08] [I] DLACore:
[06/17/2020-21:20:08] [I] Plugins:
[06/17/2020-21:20:08] [I] === Inference Options ===
[06/17/2020-21:20:08] [I] Batch: Explicit
[06/17/2020-21:20:08] [I] Input inference shapes: model
[06/17/2020-21:20:08] [I] Iterations: 10
[06/17/2020-21:20:08] [I] Duration: 3s (+ 200ms warm up)
[06/17/2020-21:20:08] [I] Sleep time: 0ms
[06/17/2020-21:20:08] [I] Streams: 1
[06/17/2020-21:20:08] [I] ExposeDMA: Disabled
[06/17/2020-21:20:08] [I] Spin-wait: Disabled
[06/17/2020-21:20:08] [I] Multithreading: Disabled
[06/17/2020-21:20:08] [I] CUDA Graph: Disabled
[06/17/2020-21:20:08] [I] Skip inference: Disabled
[06/17/2020-21:20:08] [I] Inputs:
[06/17/2020-21:20:08] [I] === Reporting Options ===
[06/17/2020-21:20:08] [I] Verbose: Disabled
[06/17/2020-21:20:08] [I] Averages: 10 inferences
[06/17/2020-21:20:08] [I] Percentile: 99
[06/17/2020-21:20:08] [I] Dump output: Disabled
[06/17/2020-21:20:08] [I] Profile: Disabled
[06/17/2020-21:20:08] [I] Export timing to JSON file:
[06/17/2020-21:20:08] [I] Export output to JSON file:
[06/17/2020-21:20:08] [I] Export profile to JSON file:
[06/17/2020-21:20:08] [I]
NVMAP_IOC_QUERY_HEAP_PARAMS failed [Inappropriate ioctl for device]
Cuda failure: initialization error
Aborted (core dumped)

Please include:

  • Exact steps/commands to build your repro
  • Exact steps/commands to run your repro
  • Full traceback of errors encountered

Hi,
Moving this to Jetson Xavier Forum.
Thanks!

My Board is Nvidia Xavier AGX (16GB RAM). Everywhere xavier stated as t194 but my board shows t186.

nvidia@nvidia-desktop:~$ cat /etc/nv_tegra_release
R32 (release), REVISION: 4.2, GCID: 20074772, BOARD: t186ref, EABI: aarch64, DATE: Thu Apr 9 01:26:40 UTC 2020

Even nvidia-l4t-apt-source.list had t194.
nvidia@nvidia-desktop:~$ sudo vi /etc/apt/sources.list.d/nvidia-l4t-apt-source.list
deb https://repo.download.nvidia.com/jetson/common r32.4 main
deb https://repo.download.nvidia.com/jetson/t194 r32.4 main

nvidia@nvidia-desktop:~/butterfly$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.4 LTS
Release: 18.04
Codename: bionic

@AakankshaS Hi, is there any update for TRT7.0 trtexec crash on DP LT4.4?

Hi,

Sorry for keeping you waiting.

May I know how do you upgrade your device into JetPack4.4?
Would you mind to reflash the Xavier and install all the package from the sdkmanager again?

CUDA initialization failure usually occurs from the incompatible software.
To install all the requirement from the same JetPack can guarantee the compatibility.

By the way, please also make sure your TensorFlow package is built with JetPack4.4.
Thanks.