Hello,
I am trying to specify GPU for inference with TensorRT like that:
with tf.device(“/device:GPU:1”):
# Create session and do inference
But it seems that framework uses GPU:0 (GPU with the lowest ID) anyway.
When I inference not-quantized model with tensorflow, with tf.device(“/device:GPU:1”) works perfectly and model is allocated correctly on GPU:1 but when I use quantized TensorRT model with the same code tf.device(“/device:GPU:1”) seems to have no effect and model is allocates on GPU:0 and utilizes GPU:0.
My environment:
tensorflow-gpu==1.13.1
CUDA==10.0
CUDNN==7.5.0
TensorRT==5.1.0-GA