Xid 31 error when two CudaGraph captured ExecutionContexts are executed concurrently

suchir · August 8, 2024, 12:07am

Xid 31 error in TensorRT 8.6.1.6 when running two CudaGraph captured ExecutionContexts concurrently on RTX 4070 or RTX A4500

opened 05:27PM - 07 Aug 24 UTC

## Description I have two TensorRT plans compiled from ONNX using the standar…d TensorRT builder and ONNX parser. I can successfully capture the `ExecutionContext`s derived from these plans to `CudaGraph`s and launch these on `Stream`s (with outputs as expected). However, when launching these operations repeatedly in a loop, and if certain conditions are met, we will eventually encounter a Xid 31 error after an arbitrary, large number of loop iterations. This error manifests itself in the program as a cuda error 700 (illegal memory access) when synchronizing the first stream. The following conditions must all be true to trigger the error: * The `ExectionContext`s must be captured to graphs. * The two `ExectionContext`s must be executing in parallel (on two `Stream`s). * There must be other compute processes on the same GPU. compute-sanitizer (all tools) and cuda-memcheck (all tools) report no problems. The issue doesn't seem to pop up when running with cuda-gdb. when CUDA_LAUNCH_BLOCKING=1 is used, the error is still received when synchronizing. ## Environment **TensorRT Version**: 8.6.1.6 **GPU Type**: tested with RTX 4070 and RTX A4500 **Nvidia Driver Version**: 550.78 (RTX 4070) or 525.60.13 (RTX A4500) **CUDA Version**: tested with 11.8 and 12.3.2 **CUDNN Version**: 8.9.7 **Operating System + Version**: tested with linux 6.6 and linux 6.1 **Python Version (if applicable)**: N/A **TensorFlow Version (if applicable)**: N/A **PyTorch Version (if applicable)**: N/A **Baremetal or Container (if container which image + tag)**: tested on nvidia/cuda:11.8.0-cudnn8-devel-ubuntu20.04 and nvcr.io/nvidia/tensorrt:24.01-py3 ## Relevant Files https://github.com/soooch/weird-trt-thing ## Steps To Reproduce ``` git clone git@github.com:soooch/weird-trt-thing.git cd weird-trt-thing docker run --gpus all -it --rm -v .:/workspace nvcr.io/nvidia/tensorrt:24.01-py3 ``` once inside container: ``` apt update apt-get install -y parallel make # need at least 2, but will fail faster if more (hence 16) parallel -j0 --delay 0.3 ./fuzzer ::: {1..16} # wait up to ~ 10 minutes. usually much faster ```

It appears this is a known, tracked CUDA bug.

Topic		Replies	Views
Xid 31 error when running two CudaGraph captured ExecutionContexts concurrently Jetson Orin Nano tensorrt , cuda	6	87	September 25, 2024
cudaErrorIllegalAddress (700) from cudaStreamSynchronize only if multiple processes use same GPU CUDA Programming and Performance tensorrt , cuda	5	151	August 7, 2024
error when run two different context parallel in TensorRT7 TensorRT	1	840	February 11, 2020
Trtexec with CUDAGraph happen rarely Cuda failure: an illegal memory access was encountered.? TensorRT cuda	3	536	January 29, 2024
error when run two different context parallel in TensorRT7 TensorRT	1	1062	February 19, 2020
Error run 2 context parallel in TensorRT7 TensorRT	13	2633	July 5, 2021
Internal error: CASK: all shaders must have unique names TensorRT	26	3324	November 4, 2019
Running 2 models on the same GPU with TensorRT TensorRT	7	1265	January 15, 2021
[executionContext.cpp::executeInternal::652] Error Code 1: Cuda Runtime (an illegal memory access was encountered) \| Cuda failure: 700 TensorRT tensorrt	5	3055	April 11, 2022
Random Xid 62 error on ML workloads - Titan RTX Linux	0	721	July 8, 2020

Xid 31 error when two CudaGraph captured ExecutionContexts are executed concurrently

Related topics