How to use TF + TRT together? Running TRT after opening TF's session Makes an FATAL error!

AI & Data Science Deep Learning (Training & Inference) TensorRT
1 
from moviepy.editor import VideoFileClip
clip = VideoFileClip('/home/me/Downloads/tensorrtyolov3/test.ts')

with tf.Session(config=tf.ConfigProto(gpu_options=tf.GPUOptions(per_process_gpu_memory_fraction=0.3))) as sess:
   for im in clip.iter_frames():  
       outputs = tensorRT.run() # run() is binded to my TensorRT C++ codes by using python boost C++ interface

above code makes an fatar error

ERROR: cuda/reformat.cu (511) - Cuda Error in NCHWToNCQHW4: 33
ERROR: cuda/reformat.cu (511) - Cuda Error in NCHWToNCQHW4: 33

but if I change the order of calling the TF’s session and TRT’s doInference() like below,

from moviepy.editor import VideoFileClip
clip = VideoFileClip('/home/me/Downloads/tensorrtyolov3/test.ts')

for im in clip.iter_frames():  
    outputs = tensorRT.run() # run() is binded to my TensorRT C++ codes by using python boost C++ interface
    with tf.Session(config=tf.ConfigProto(gpu_options=tf.GPUOptions(per_process_gpu_memory_fraction=0.3))) as sess:
         ... 
         ...

no more error occurs. But I must run TF’s session out of the for loop of course!! Otherwise,
it will open/close a TF session on every video frame read which makes it run super slow !!

Please let me know any guide to solve this strange issue!

Thank you!

2

It’s unclear what TensorRT.run() does.

  • Does your TensorRT module also use TF?
  • Is TensorRT.run() asynchronous?
  • How much memory does it need?
  • If TensorRT module doesn't use TF, then why do you run it inside a TF session?
  • If TensorRT module does use TF, then does it create a TF session under the hood?