How to use input_consumed in execute_async

AI & Data Science Deep Learning (Training & Inference) TensorRT
,
1 
execute_async(self: tensorrt.tensorrt.IExecutionContext, batch_size: int = 1, 
    bindings: List[int], stream_handle: int, input_consumed: capsule = None) → bool

I noticed in the TensorRT 7 Python API, the execute_async function has an optional argument input_consumed. The doc doesn’t clarify how to use this argument. I want to utilize it so that I can safely copy new input to the page locked host buffer for my next inference as soon as possible. What is this capsule type exactly?

2

Hi @alexis.yang
We do not have a sample around this.
input_consumed consumes a cudaEvent and the Event is used to sync the input.
Please check the below link for reference.
https://docs.nvidia.com/deeplearning/tensorrt/api/python_api/infer/Core/ExecutionContext.html#tensorrt.IExecutionContext.execute_async
Thanks!

3

@AakankshaS
Thanks for your response. Are you referring to the Cuda Event here :
https://documen.tician.de/pycuda/driver.html?highlight=pagelocked_empty#pycuda.driver.Event
Can I pass in a pycuda.driver.Event as the input_consumed argument? Can I assume calling Event.synchronize() afterwards will block until the input buffer has been consumed?

4

Hi @alexis.yang,

Could you please let us know if you are still facing this issue?

Thanks

5

Sorry, I ended up not using that parameter. But I think the link I posted should point people to the right direction if they face similar issue.

6

I tried to run a source code including

buffer_refill_event = pycuda.driver.Event()
context.execute_async_v2(bindings=[int(d_input), int(d_output)], stream_handle=stream.handle, input_consumed=buffer_refill_event)

As a result, I get the error message

TypeError: execute_async_v2(): incompatible function arguments. The following argument types are supported:
    1. (self: tensorrt.tensorrt.IExecutionContext, bindings: List[int], stream_handle: int, input_consumed: capsule = None) -> bool

Invoked with: <tensorrt.tensorrt.IExecutionContext object at 0x7fd847a7cc70>; kwargs: bindings=[140561916887040, 140561917489152], stream_handle=94093906828240, input_consumed=<pycuda._driver.Event object at 0x7fd847a88180>

If I remove the argument input_consumed=buffer_refill_event, there is no error. I do not understand what the capsule type exactly is or which type of argument the fuction expects respectively.