cudaErrorIllegalAddress (700) from cudaStreamSynchronize only if multiple processes use same GPU

[suchir]

Edit: TL;DR: CUDA bug. see details in below link

So I’ve reduced this issue to the point where nearly all the context provided above is unnecessary. I would like to close this topic, but it doesn’t appear I have permission. So I’m just going to open another one based on the minified repro.

Xid 31 error when two CudaGraph captured ExecutionContexts are executed concurrently

Description I have two TensorRT plans compiled from ONNX using the standard TensorRT builder and ONNX parser. I can successfully capture the ExecutionContexts derived from these plans to CudaGraphs and launch these on Streams (with outputs as expected). However, when launching these operations repeatedly in a loop, and if certain conditions are met, we will eventually encounter a Xid 31 error after an arbitrary, large number of loop iterations. This error manifests itself in the program as a c…