System Reboot when running TensorRT on 536.40 driver

AI & Data Science Deep Learning (Training & Inference) TensorRT
, , , , ,
1

Description

I am running object detection on my GPU inside a container. When the object detection runs, my system will hard reboot, no bluescreen, and no warnings in any system logs.

I rolled back to driver version 528.49 and the issue goes away and object detection runs without issue. I’m not yet sure where between 528 and 536 this starts happening.

Environment

TensorRT Version: 8.4.1.5 - 8.6.1
GPU Type: GTX 1080ti
Nvidia Driver Version: 536.40
CUDA Version: 11.7 - 12.1
CUDNN Version: 8.7 - 8.9
Operating System + Version: Windows 11 + WSL2 (Ubuntu 20.04)
Python Version (if applicable): 3.9
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag): Package frigate · GitHub

Relevant Files

Issue observed using various yolo models generated from GitHub - yeahme49/tensorrt_demos: TensorRT MODNet, YOLOv4, YOLOv3, SSD, MTCNN, and GoogLeNet.
This model is automatically downloaded and converted when the above container runs.

docker-compose.yml (640 Bytes)
config.yml (816 Bytes)

Steps To Reproduce

  1. Create test folder and copy docker-compose.yml inside
  2. Create folder named “config” in test folder
  3. Copy config.yml into config folder
  4. Edit config.yml with path to rstp stream (or follow Frigate documentation to pass in video file) and edit detect height and width to match video frame size.
  5. From test folder, run “docker compose up”
  6. System reboots, no traceback or errors found
4

Hi,

We recommend you please install the TensorRT from https://developer.nvidia.com/nvidia-tensorrt-8x-download or TensorRT | NVIDIA NGC and try again.

Thank you.