Long load time for tf-trt graph with tensorflow c api

[celia.martin1]

Hi,

I am trying to use tensorflow C++ API to run tf-trt graphs, and my problem is that it takes around 5 minutes to load the graph (in function TF_LoadSessionFromSavedModel, it stucks at this step : “Adding visible gpu devices: 0”)

I first had the problem on python API and C++ API. I found a solution for the python api (I followed this steps : JetPack-4.5), but couldn’t find one for C++ API.

I have those versions installed :

• Jetpack 4.5
• Tensorflow 2.4.0 - nv21.5
• tensorrt 7.1.3.0
• Numpy 1.19.4
• Python 3.6.9
• Protobuf 3.19.4
• Protoc 3.8.0

Does someone have a solution ?

Thanks !

[AastaLLL]

Hi,

Could you enable the logger to get more information about the issue?

tf.get_logger  |  TensorFlow v2.4.0

Thanks.

[celia.martin1]

The default value of the logger is to WARNING. I don’t see any difference in the log messages when I change its value (with tf.get_logger().setLevel()). However I was able to change the log level using “export TF_CPP_MIN_LOG_LEVEL=0”, but only with the python api. It does not change the log level when I use the c++ api.

[AastaLLL]

Hi,

Since JetPack 4.5 is released for a while, could you try our latest JetPack 4.6.2 or JetPack 5 instead?
You can find the corresponding TF-TRT package below:

https://developer.download.nvidia.com/compute/redist/jp/

Thanks.

[celia.martin1]

Hi,

I installed the jetpack 5.0.2, with tensorflow 2.9.1. It works well on python, so I am now searching to use it in c++. I wasn’t able to find the prebuilt libraries. Is it available somewhere ?
If not I will try to build it from source

Thanks

[celia.martin1]

Hi,

I can’t build tensorflow c_api on my jetson nano with jetpack 5.0.2 because the disk space is too small.

So I tried do it with the jetpack 4.6.2. The problem is that I can’t find a version of tensorflow built with tensorrt.

Until today I was installing tensorflow using this command :
python3 -m pip install --pre --extra-index-url https://developer.download.nvidia.com/compute/redist/jp/v$JP_VERSION tensorflow==$TF_VERSION+nv$NV_VERSION

But I can’t find any version available for the jetpack 4.6.2
ERROR: Could not find a version that satisfies the requirement tensorflow==2.7.0+nv22.01 (from versions: 2.10.0rc0, 2.10.0rc1, 2.10.0rc2, 2.10.0rc3)

So I installed it with this command :
python3 -m pip install --pre --extra-index-url https://developer.download.nvidia.com/compute/redist/jp/v462 tensorflow
Which installed the version 2.10.0rc3

Then when I try to use it with tensorrt I have this error :
RuntimeError: Tensorflow has not been built with TensorRT support.

Do you have a solution ?

[AastaLLL]

Hi,

Please noted that our package all contains the nv[year].[month] flag.
So 2.10.0rc3 might come from the third-party

For JetPack 4.6.2, since it is a minor change from 4.6.1, please use 461 for $JP_VERSION instead.
For example:

sudo pip3 install --pre --extra-index-url https://developer.download.nvidia.com/compute/redist/jp/v461 tensorflow==2.7.0+nv22.1

Thanks.