Hi,
I’ve been following your example to speed up inference with TensorRT, but when I try conversion using INT8 precision mode my program is killed during .build() call.
Output of sudo dmesg -T is:
[Thu Sep 14 04:40:02 2023] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-54.scope,task=python,pid=218008,uid=1000
[Thu Sep 14 04:40:02 2023] Out of memory: Killed process 218008 (python) total-vm:87797408kB, anon-rss:0kB, file-rss:1177636kB, shmem-rss:0kB, UID:1000 pgtables:38348kB oom_score_adj:0
[Thu Sep 14 04:40:07 2023] oom_reaper: reaped process 218008 (python), now anon-rss:0kB, file-rss:1179088kB, shmem-rss:0kB
Moreover, whenever I use Tensorflow, my memory gets used up almost completely, is this normal? Here’s the output of jtop:
My code:
import sys
import os
import cv2
import numpy as np
import tensorflow as tf
from tensorflow.python.compiler.tensorrt import trt_convert as trt
from pose_estimation.pose_net import PoseNet
POSE_SAVED_MODEL_DIR="./weights/pose_native_saved_model"
def calibration_input():
for i in range(100):
batched_input = np.random.random((1, 224, 224, 3)).astype(np.float32)
batched_input = tf.constant(batched_input)
yield (batched_input,)
# Instantiate the TF-TRT converter
converter = trt.TrtGraphConverterV2(
input_saved_model_dir=POSE_SAVED_MODEL_DIR,
use_dynamic_shape=True,
dynamic_shape_profile_strategy='Optimal',
max_workspace_size_bytes=4000000000,
precision_mode=trt.TrtPrecisionMode.INT8
)
# Convert the model into TRT compatible segments
trt_func = converter.convert(calibration_input_fn=calibration_input)
converter.summary()
POSE_TRT_MODEL_DIR="./weights/pose_trt_09.14_01_int8"
def input_fn():
batched_input = np.random.random((1, 224, 224, 3)).astype(np.float32)
batched_input = tf.constant(batched_input)
yield batched_input
converter.build(input_fn=input_fn)
converter.save(output_saved_model_dir=POSE_TRT_MODEL_DIR)