Does Python backend in Triton Server for Jetson supports GPU?

[m.elkateb]

Hi ! I have a question about Triton Inference Server on Jetson :
Does Triton Inference Server python backend support GPU or using the output of another model that is running on GPU in an ensembling scheme ?

[AastaLLL]

Hi,

If you use some underlying frameworks that support GPU, ex. PyTorch or TensorFlow.
Then the inference can run on GPU.

Thanks.

[m.elkateb]

Thanks you for your answer. The idea is to use model ensemble logic to append post processing to an onnx model with python backend. Unfortunately when the onnx model is run on GPU, python backend errors out with “try to use CUDA copy while GPU is not supported”. I suspect the python backend can’t process because the memory output is on GPU. Is there any workaround or api available to just copy that memory to CPU and do the processing?

[AastaLLL]

There is no update from you for a period, assuming this is not an issue any more.
Hence we are closing this topic. If need further support, please open a new one.
Thanks

Hi,

Could you please share the steps and source code to reproduce this error?
We want to check this issue further.

Thanks.