How to convert a trained model to TensorRT for inference?

Hi All:

I used David Sandberg’s model to make inferences at Xavier.
model name : 20180402-114759

I want to convert the model to TensorRT. Is there any example or file reference?

Thanks a lot for help
Nick

Hi,

There are two possible workflows to convert a TensorFlow into TensorRT:

Pure TensorRT: Please check this tutorial for information
https://github.com/NVIDIA-AI-IOT/tf_to_trt_image_classification

TF-TRT: Please check this tutorial for information
https://github.com/NVIDIA-AI-IOT/tf_trt_models

Thanks.

Hi AastaLLL,

Thank you for your answer.

I have found similar resources on github.

But it takes a long time to rebuild the model, and the efficiency has not increased.

Thanks a lot for help
Nick

Hi,

May I know the exact performance result of your model?
Ex. the fps on desktop GPU vs. Xavier?

Thanks.

Hi AastaLLL,

I only use Xavier.

The following is the test time result.
original network: 0.062 sec
tensorrt network FP16(frozen meta-graph and checkpoint): 0.063 sec
tensorrt network FP16(SavedModel[pb]): 0.073 sec

Tensorrt doesn’t accelerate 30-40% efficiency.

Thanks a lot for help
Nick

Hi,

1. Have you maximized the CPU/GPU performance?

sudo ./jetson_clocks.sh

2. Please remember that TF-TRT will still use TensorFlow implementation if a layer doesn’t support by TensorRT.
It’s recommended to check if all the layers are listed as supported in our document first:
https://docs.nvidia.com/deeplearning/sdk/tensorrt-support-matrix/index.html#layers-matrix

Thanks.

Hi AastaLLL,

I have used " sudo ./jetson_clocks.sh " and " nvpmodel -m 0 "

nvpmodel -q --verbo
NVPM VERB: Config file: /etc/nvpmodel.conf
NVPM VERB: parsing done for /etc/nvpmodel.conf
NVPM VERB: Current mode: NV Power Mode: MAXN
0
NVPM VERB: PARAM CPU_ONLINE: ARG CORE_0: PATH /sys/devices/system/cpu/cpu0/online: REAL_VAL: 1 CONF_VAL: 1
NVPM VERB: PARAM CPU_ONLINE: ARG CORE_1: PATH /sys/devices/system/cpu/cpu1/online: REAL_VAL: 1 CONF_VAL: 1
NVPM VERB: PARAM CPU_ONLINE: ARG CORE_2: PATH /sys/devices/system/cpu/cpu2/online: REAL_VAL: 1 CONF_VAL: 1
NVPM VERB: PARAM CPU_ONLINE: ARG CORE_3: PATH /sys/devices/system/cpu/cpu3/online: REAL_VAL: 1 CONF_VAL: 1
NVPM VERB: PARAM CPU_ONLINE: ARG CORE_4: PATH /sys/devices/system/cpu/cpu4/online: REAL_VAL: 1 CONF_VAL: 1
NVPM VERB: PARAM CPU_ONLINE: ARG CORE_5: PATH /sys/devices/system/cpu/cpu5/online: REAL_VAL: 1 CONF_VAL: 1
NVPM VERB: PARAM CPU_ONLINE: ARG CORE_6: PATH /sys/devices/system/cpu/cpu6/online: REAL_VAL: 1 CONF_VAL: 1
NVPM VERB: PARAM CPU_ONLINE: ARG CORE_7: PATH /sys/devices/system/cpu/cpu7/online: REAL_VAL: 1 CONF_VAL: 1
NVPM VERB: PARAM TPC_POWER_GATING: ARG TPC_PG_MASK: PATH /sys/devices/gpu.0/tpc_pg_mask: REAL_VAL: 0 CONF_VAL: 0
NVPM VERB: PARAM GPU_POWER_CONTROL_ENABLE: ARG GPU_PWR_CNTL_EN: PATH /sys/devices/gpu.0/power/control: REAL_VAL: auto CONF_VAL: on
NVPM VERB: PARAM CPU_DENVER_0: ARG MIN_FREQ: PATH /sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq: REAL_VAL: 1190400 CONF_VAL: 1200000
NVPM VERB: PARAM CPU_DENVER_0: ARG MAX_FREQ: PATH /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq: REAL_VAL: 2265600 CONF_VAL: 2147483647
NVPM VERB: PARAM CPU_DENVER_1: ARG MIN_FREQ: PATH /sys/devices/system/cpu/cpu2/cpufreq/scaling_min_freq: REAL_VAL: 1190400 CONF_VAL: 1200000
NVPM VERB: PARAM CPU_DENVER_1: ARG MAX_FREQ: PATH /sys/devices/system/cpu/cpu2/cpufreq/scaling_max_freq: REAL_VAL: 2265600 CONF_VAL: 2147483647
NVPM VERB: PARAM CPU_DENVER_2: ARG MIN_FREQ: PATH /sys/devices/system/cpu/cpu4/cpufreq/scaling_min_freq: REAL_VAL: 1190400 CONF_VAL: 1200000
NVPM VERB: PARAM CPU_DENVER_2: ARG MAX_FREQ: PATH /sys/devices/system/cpu/cpu4/cpufreq/scaling_max_freq: REAL_VAL: 2265600 CONF_VAL: 2147483647
NVPM VERB: PARAM CPU_DENVER_3: ARG MIN_FREQ: PATH /sys/devices/system/cpu/cpu6/cpufreq/scaling_min_freq: REAL_VAL: 1190400 CONF_VAL: 1200000
NVPM VERB: PARAM CPU_DENVER_3: ARG MAX_FREQ: PATH /sys/devices/system/cpu/cpu6/cpufreq/scaling_max_freq: REAL_VAL: 2265600 CONF_VAL: 2147483647
NVPM VERB: PARAM GPU: ARG MIN_FREQ: PATH /sys/devices/17000000.gv11b/devfreq/17000000.gv11b/min_freq: REAL_VAL: 318750000 CONF_VAL: 318750000
NVPM VERB: PARAM GPU: ARG MAX_FREQ: PATH /sys/devices/17000000.gv11b/devfreq/17000000.gv11b/max_freq: REAL_VAL: 1377000000 CONF_VAL: 2147483647
NVPM VERB: PARAM GPU_POWER_CONTROL_DISABLE: ARG GPU_PWR_CNTL_DIS: PATH /sys/devices/gpu.0/power/control: REAL_VAL: auto CONF_VAL: auto
NVPM ERROR: Error opening /sys/kernel/nvpmodel_emc_cap/emc_iso_cap: 13
NVPM ERROR: failed to read PARAM EMC: ARG MAX_FREQ: PATH /sys/kernel/nvpmodel_emc_cap/emc_iso_cap
NVPM ERROR: Error opening /sys/kernel/nvpmodel_emc_cap/nafll_dla: 13
NVPM ERROR: failed to read PARAM DLA_CORE: ARG MAX_FREQ: PATH /sys/kernel/nvpmodel_emc_cap/nafll_dla
NVPM ERROR: Error opening /sys/kernel/nvpmodel_emc_cap/nafll_dla_falcon: 13
NVPM ERROR: failed to read PARAM DLA_FALCON: ARG MAX_FREQ: PATH /sys/kernel/nvpmodel_emc_cap/nafll_dla_falcon
NVPM ERROR: Error opening /sys/kernel/nvpmodel_emc_cap/nafll_pva_vps: 13
NVPM ERROR: failed to read PARAM PVA_VPS: ARG MAX_FREQ: PATH /sys/kernel/nvpmodel_emc_cap/nafll_pva_vps
NVPM ERROR: Error opening /sys/kernel/nvpmodel_emc_cap/nafll_pva_core: 13
NVPM ERROR: failed to read PARAM PVA_CORE: ARG MAX_FREQ: PATH /sys/kernel/nvpmodel_emc_cap/nafll_pva_core
NVPM ERROR: Error opening /sys/kernel/nvpmodel_emc_cap/nafll_cvnas: 13
NVPM ERROR: failed to read PARAM CVNAS: ARG MAX_FREQ: PATH /sys/kernel/nvpmodel_emc_cap/nafll_cvnas

What is wrong with the content(ERROR)?

I will confirm that all layers are supported.

Thanks a lot for the help
Nick

Hi,

You need root authority to run the nvpmodel exec.

sudo nvpmodel -m 0

And please noticed that the execution order is matter.
Please enable the performance mode first and then lock the CPU/GPU clock to the maximal.

Thanks.