Implicit CUDA Graph Capturing with TensorRT on DRIVE Thor

Autonomous Vehicles DRIVE AGX Thor DRIVE AGX Thor General
1

DRIVE OS Version: 7.0.3

Issue Description:
Hello NVIDIA team,

I am testing a TensorRT application on NVIDIA DRIVE Thor and seeing behavior that I would like to confirm.

Environment:

  • Platform: NVIDIA DRIVE Thor Development Kit
  • DRIVE OS: 7.0.3
  • Guest OS: Ubuntu 24.04
  • CUDA Toolkit: 12.8
  • TensorRT: 10.10.10
  • CPU + GPU (Blackwell SoC with ARM64 CPU)

Observation:

  • In my application code, I do not call cudaStreamBeginCapture() or any other CUDA Graph capture APIs.
  • During TensorRT execution, a custom plugin enqueue path is called multiple times on the same stream.
  • First call: stream is not in capture mode, confirmed with cudaStreamIsCapturing().
  • Second call: stream reports capture mode is active according to cudaStreamIsCapturing().
  • At that point, a Thrust operation (thrust::sequence()) in plugin enqueue throws:
    • thrust::system_error
    • cudaErrorStreamCaptureUnsupported: operation not permitted when stream is capturing
      • thrust::sequence() calls cudaMalloc() internally, which should not be called in capturing mode.
  • Based on these observed facts, I’m suspecting that CUDA Graph Capturing is enabled implicitly.

Additional finding:

  • Replacing the Thrust operation with a CUB/kernel-based implementation avoids this specific capture-related failure.

Questions:

  1. Can TensorRT 10.10.10 on DRIVE Thor internally enable CUDA stream capture (during build or runtime) even when user code does not explicitly begin capturing?
  2. If yes, which TensorRT phases may do this implicitly (engine build, tactic profiling, enqueue/runtime optimization, etc.)?
  3. If yes, is this specific to DRIVE OS or DRIVE Thor? I don’t see any similar problem on other NVIDIA platforms.
  4. If yes, is there any API or setting to disable this behavior?
  5. What are the official requirements/restrictions for custom plugin enqueue implementations under this behavior?
  6. Is Thrust in plugin enqueue considered unsupported/risky in this context, and is NVIDIA’s recommendation to avoid Thrust there?

I tried to write a small reproduction code for this issue as my application is too large, but failed.
The code cannot reproduce the behavior.
There seems to be some conditions to trigger the “implicit capturing”.

In the process of searching for similar questions (there is none) before posting this message, I came across the following topic.
issues-running-tensorrt-inference-on-jetson-orin-cuda-stream-capture-errors/335388

this seems to be an issue with CUDA Graphs. Could you try setting the enable_cuda_graphs parameter of the inference operator to false?
The inference works for the first iteration because CUDA Graph usage is only enabled for the second and later iterations.

Though the situation is quite different from mine, the answer says “CUDA Graph usage is only enabled for the second and later iterations”.
I tried to find the “enable_cuda_graphs” parameter, but failed.
Edit: enable_cuda_graph is a parameter for Holoscan.

Any guidance on expected behavior and best practices for plugin implementation on DRIVE Thor would be greatly appreciated.

5

Dear @masaki.nishikawa2 ,

  1. Can TensorRT 10.10.10 on DRIVE Thor internally enable CUDA stream capture (during build or runtime) even when user code does not explicitly begin capturing?

Yes

  1. If yes, which TensorRT phases may do this implicitly (engine build, tactic profiling, enqueue/runtime optimization, etc.)?

This behavior is most commonly triggered during engine build, tactic profiling, and runtime optimization

  1. If yes, is this specific to DRIVE OS or DRIVE Thor? I don’t see any similar problem on other NVIDIA platforms.

This is not specific to Thor as such. we have seen it in Orin as well.

4 If yes, is there any API or setting to disable this behavior?

No.

  1. What are the official requirements/restrictions for custom plugin enqueue implementations under this behavior?
  2. Is Thrust in plugin enqueue considered unsupported/risky in this context, and is NVIDIA’s recommendation to avoid Thrust there?

The recommended workaround is to check the stream’s capture status inside your plugin enqueue() method and avoid incompatible CUDA API calls (such as cudaMalloc, cudaStreamSynchronize, or Thrust operations that allocate memory) when capture is active and use CUB/kernel-based implementation as you did already.

6

Dear @SivaRamaKrishnaNV ,

Thank you very much for your swift answer. It helped me a lot.
I need one more clarification.

I interpret your statement as “this behavior is common for DRIVE Thor and DRIVE Orin”.

How about Jetson AGX Thor and Jetson AGX Orin?
How about other discreet GPUs?
I’ve never observed similar behavior on these platforms.

In anyway, I think our team needs to modify the application so that it works on DRIVE Thor.
It is nice to identify the risk beforehand, rather than finding it just before the deadline.