Xid 31 error when running two CudaGraph captured ExecutionContexts concurrently

suchir · August 8, 2024, 12:00am

This issue is not strictly related to Orin Nano, but impacts our ability to test systems which we deploy to Orin Nano. We were asked to create a post on the Jetson forum after contacting Nvidia.

For details see the following links:

github.com/NVIDIA/TensorRT

Xid 31 error in TensorRT 8.6.1.6 when running two CudaGraph captured ExecutionContexts concurrently on RTX 4070 or RTX A4500

opened 05:27PM - 07 Aug 24 UTC

soooch

## Description I have two TensorRT plans compiled from ONNX using the standar…d TensorRT builder and ONNX parser. I can successfully capture the `ExecutionContext`s derived from these plans to `CudaGraph`s and launch these on `Stream`s (with outputs as expected). However, when launching these operations repeatedly in a loop, and if certain conditions are met, we will eventually encounter a Xid 31 error after an arbitrary, large number of loop iterations. This error manifests itself in the program as a cuda error 700 (illegal memory access) when synchronizing the first stream. The following conditions must all be true to trigger the error: * The `ExectionContext`s must be captured to graphs. * The two `ExectionContext`s must be executing in parallel (on two `Stream`s). * There must be other compute processes on the same GPU. compute-sanitizer (all tools) and cuda-memcheck (all tools) report no problems. The issue doesn't seem to pop up when running with cuda-gdb. when CUDA_LAUNCH_BLOCKING=1 is used, the error is still received when synchronizing. ## Environment **TensorRT Version**: 8.6.1.6 **GPU Type**: tested with RTX 4070 and RTX A4500 **Nvidia Driver Version**: 550.78 (RTX 4070) or 525.60.13 (RTX A4500) **CUDA Version**: tested with 11.8 and 12.3.2 **CUDNN Version**: 8.9.7 **Operating System + Version**: tested with linux 6.6 and linux 6.1 **Python Version (if applicable)**: N/A **TensorFlow Version (if applicable)**: N/A **PyTorch Version (if applicable)**: N/A **Baremetal or Container (if container which image + tag)**: tested on nvidia/cuda:11.8.0-cudnn8-devel-ubuntu20.04 and nvcr.io/nvidia/tensorrt:24.01-py3 ## Relevant Files https://github.com/soooch/weird-trt-thing ## Steps To Reproduce ``` git clone git@github.com:soooch/weird-trt-thing.git cd weird-trt-thing docker run --gpus all -it --rm -v .:/workspace nvcr.io/nvidia/tensorrt:24.01-py3 ``` once inside container: ``` apt update apt-get install -y parallel make # need at least 2, but will fail faster if more (hence 16) parallel -j0 --delay 0.3 ./fuzzer ::: {1..16} # wait up to ~ 10 minutes. usually much faster ```

AastaLLL · August 8, 2024, 4:57am

Hi,

Jetson doesn’t support concurrent memory access.
If the error is triggered when concurrent access, it is expected.

You can find below for more Jetson-specific usage:

Thanks.

suchirkavi · August 8, 2024, 9:49am

Are you referring to MPS? If so, that’s unrelated to the issue. Maybe you’re referring to concurrent access to unified memory? Also not relevant.

Please note this issue does not impact us on our Jetson based platform (because we currently only have a single process using GPU compute). But we have CI servers which we use to test code which runs on our Jetson platform. And we would like to be able to run many GPU compute processes on these servers.

Yes, I recognize this is a somewhat tenuous connection to Jetson, but our contact at Nvidia asked us to post the issue to this forum in order for it to be tracked internally.

AastaLLL · August 12, 2024, 6:04am

Hi,

To check it further, could you share a reproducible source with us?
The link you shared has a precompiled TensorRT plan, is it compiled for Orin Nano?

Thanks.

suchir · August 19, 2024, 10:35pm

Hi AastaLLL, the steps for reproduction are available here. The steps are for a desktop system. We do have these plans compiled for Orin Nano, but the plan we’ve provided is built for desktop (for CI/CD regression testing purposes).

We have confirmation that the issue is with CUDA. What we’re looking for is a workaround.

AastaLLL · August 26, 2024, 5:15am

There is no update from you for a period, assuming this is not an issue anymore.
Hence, we are closing this topic. If need further support, please open a new one.
Thanks

Hi,

Sorry for the late update.
We will try to reproduce this issue in our environment.

Do you mean the model0.plan and model1.plan are compiled on Orin Nano with JetPack 6.0?
So we can use it without re-converting the engine?

Thanks.