Multi model inference - Swap GPU memory

[spolisetty]

Hi @hanoch.kremer,

Looks like this query is already posted. We request you to respond in the below thread.

Multi model inference - Swap GPU memory

HI, I’ve converted few CNN models via TF-TRT (TRT=7.2.2.3; TF=2.4.1; CUDA=11.1, python API) but im limited by GPU (2080i) memory rather than throughput. Is there a way to swap/copy a model memory from the GPU out to a fast DDR(slow latency copy) or CPU(high latency copy-less preferred) hence release GPU memory for other models? Of course copy back the model when needed, sort of a multi-tasking way of doing? Thanks, Hanoch

Thank you.