Memory Issues and Conversion issues with TF-TRT on Nano

TP2049 · December 6, 2019, 11:12am

Heya :)

I have got a script working that takes about a minute and half to load the model but once it does the inference is fast. However it is giving me a memory failure that I thought I could resolve by converting the keras h5 file to trt.

Note: The keras h5 file I am using was a model I trained on teachable machine with google and exported from them.

The inference script without conversion:

import tensorflow as tf
import numpy as np
from keras_preprocessing import image
import pathlib as plib

keras_File = plib.Path('/home/tom/Python_Progs/keras_model.h5')        #Jetson

model = tf.keras.models.load_model(keras_File, compile = False)
model.summary()

chosen_img = plib.Path('/home/tom/Python_Progs/Images/cartest01.jpg')

img = image.load_img(chosen_img, target_size = (224,224))
x = image.img_to_array(img)
x = np.expand_dims(x, axis = 0)

images = np.vstack([x])

classes = model.predict(images, batch_size = 10)
print(chosen_img)
print(classes)

Below is the output from this script:

2019-12-05 11:48:28.323261: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 178 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
2019-12-05 11:49:08.690666: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2019-12-05 11:49:10.673712: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2019-12-05 11:49:45.063301: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 331.79MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
[[0.9999987  0.00000129]]

If I run it on PC CPU version its perfect however on Jetson Nano GPU version it produces above memory warning.

So I have been trying to convert as per the script below using a TF saved model exported from teachable machine with google:

from tensorflow.python.compiler.tensorrt import trt_convert as trt

import pathlib as plib

input_saved_model_dir = plib.Path('/home/tom/Python_Progs/Tensor/savedmodel/')
output_saved_model_dir = plib.Path('/home/tom/Python_Progs/Tensor/outmodel/')
input_saved_model_dir = str(input_saved_model_dir)
output_saved_model_dir = str(output_saved_model_dir)

converter = trt.TrtGraphConverterV2(input_saved_model_dir=input_saved_model_dir)
converter.convert()
converter.save(output_saved_model_dir)

And this is the output from the conversion script:

/usr/bin/python3 /home/tom/Python_Progs/Tensor/convSavedModel.py
2019-12-05 15:44:35.792007: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-12-05 15:44:42.490310: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer.so.5
WARNING:tensorflow:TensorRT mismatch. Compiled against version 5.0.6, but loaded 5.1.6. Things may not work
2019-12-05 15:45:02.995995: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2019-12-05 15:45:03.039523: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:03.039697: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
pciBusID: 0000:00:00.0
2019-12-05 15:45:03.039774: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-12-05 15:45:03.039883: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2019-12-05 15:45:03.055304: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2019-12-05 15:45:03.093331: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2019-12-05 15:45:03.109549: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2019-12-05 15:45:03.124828: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2019-12-05 15:45:03.125019: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2019-12-05 15:45:03.125255: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:03.125521: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:03.125648: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2019-12-05 15:45:03.128144: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:03.128302: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
pciBusID: 0000:00:00.0
2019-12-05 15:45:03.128386: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-12-05 15:45:03.128449: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2019-12-05 15:45:03.128506: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2019-12-05 15:45:03.128563: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2019-12-05 15:45:03.128616: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2019-12-05 15:45:03.128664: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2019-12-05 15:45:03.128711: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2019-12-05 15:45:03.128891: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:03.129111: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:03.129198: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2019-12-05 15:45:03.129300: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-12-05 15:45:15.797571: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-12-05 15:45:15.802419: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0 
2019-12-05 15:45:15.802447: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N 
2019-12-05 15:45:15.808713: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:15.808989: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:15.809209: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:15.817460: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 143 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py:1781: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
WARNING:tensorflow:Issue encountered when serializing variables.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
to_proto not supported in EAGER mode.
WARNING:tensorflow:Issue encountered when serializing trainable_variables.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
to_proto not supported in EAGER mode.
2019-12-05 15:45:34.273897: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:34.274067: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0
2019-12-05 15:45:34.274297: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2019-12-05 15:45:34.275038: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:34.275173: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
pciBusID: 0000:00:00.0
2019-12-05 15:45:34.275361: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-12-05 15:45:34.275456: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2019-12-05 15:45:34.275523: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2019-12-05 15:45:34.275572: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2019-12-05 15:45:34.275620: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2019-12-05 15:45:34.275666: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2019-12-05 15:45:34.275709: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2019-12-05 15:45:34.275861: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:34.276050: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:34.276126: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2019-12-05 15:45:34.288217: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-12-05 15:45:34.288293: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0 
2019-12-05 15:45:34.288319: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N 
2019-12-05 15:45:34.288569: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:34.288822: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:34.288945: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 143 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
2019-12-05 15:45:34.872098: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:716] Optimization results for grappler item: graph_to_optimize
2019-12-05 15:45:34.872190: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718]   function_optimizer: function_optimizer did nothing. time = 0.005ms.
2019-12-05 15:45:34.872223: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718]   function_optimizer: function_optimizer did nothing. time = 0.001ms.
2019-12-05 15:45:42.493717: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:42.493950: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0
2019-12-05 15:45:42.494158: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2019-12-05 15:45:42.494981: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:42.495138: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
pciBusID: 0000:00:00.0
2019-12-05 15:45:42.495362: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-12-05 15:45:42.495472: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2019-12-05 15:45:42.495569: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2019-12-05 15:45:42.495653: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2019-12-05 15:45:42.495733: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2019-12-05 15:45:42.495810: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2019-12-05 15:45:42.495889: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2019-12-05 15:45:42.496183: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:42.496449: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:42.496529: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2019-12-05 15:45:42.496606: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-12-05 15:45:42.496638: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0 
2019-12-05 15:45:42.496668: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N 
2019-12-05 15:45:42.496860: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:42.497257: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:42.497424: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 143 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
2019-12-05 15:45:43.844906: I tensorflow/compiler/tf2tensorrt/segment/segment.cc:460] There are 4 ops of 3 different types in the graph that are not converted to TensorRT: Softmax, NoOp, Placeholder, (For more information see https://docs.nvidia.com/deeplearning/frameworks/tf-trt-user-guide/index.html#supported-ops).
2019-12-05 15:45:43.974022: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:633] Number of TensorRT candidate segments: 1
2019-12-05 15:45:44.037919: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:734] TensorRT node TRTEngineOp_0 added for segment 0 consisting of 430 nodes succeeded.
2019-12-05 15:45:44.468514: W tensorflow/compiler/tf2tensorrt/convert/trt_optimization_pass.cc:183] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-12-05 15:45:44.589282: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:716] Optimization results for grappler item: tf_graph
2019-12-05 15:45:44.589380: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718]   constant folding: Graph size after: 430 nodes (-263), 439 edges (-263), time = 170.223ms.
2019-12-05 15:45:44.589409: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718]   layout: Graph size after: 437 nodes (7), 446 edges (7), time = 206.055ms.
2019-12-05 15:45:44.589440: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718]   constant folding: Graph size after: 432 nodes (-5), 441 edges (-5), time = 136.568ms.
2019-12-05 15:45:44.589469: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718]   TensorRTOptimizer: Graph size after: 3 nodes (-429), 2 edges (-439), time = 296.231ms.
2019-12-05 15:45:44.589493: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718]   constant folding: Graph size after: 3 nodes (0), 2 edges (0), time = 53.271ms.
2019-12-05 15:45:44.589520: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:716] Optimization results for grappler item: TRTEngineOp_0_native_segment
2019-12-05 15:45:44.589549: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718]   constant folding: Graph size after: 432 nodes (0), 441 edges (0), time = 48.034ms.
2019-12-05 15:45:44.589578: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718]   layout: Graph size after: 432 nodes (0), 441 edges (0), time = 60.248ms.
2019-12-05 15:45:44.589605: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718]   constant folding: Graph size after: 432 nodes (0), 441 edges (0), time = 53.446ms.
2019-12-05 15:45:44.589626: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718]   TensorRTOptimizer: Graph size after: 432 nodes (0), 441 edges (0), time = 6.602ms.
2019-12-05 15:45:44.589647: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718]   constant folding: Graph size after: 432 nodes (0), 441 edges (0), time = 57.387ms.
2019-12-05 15:46:17.035132: W tensorflow/core/framework/op_kernel.cc:1622] OP_REQUIRES failed at trt_engine_resource_ops.cc:183 : Not found: Container TF-TRT does not exist. (Could not find resource: TF-TRT/TRTEngineOp_0)
Traceback (most recent call last):
  File "/home/tom/Python_Progs/Tensor/convSavedModel.py", line 12, in <module>
    converter.save(output_saved_model_dir)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/compiler/tensorrt/trt_convert.py", line 1069, in save
    save.save(self._saved_model, output_saved_model_dir, signatures)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/saved_model/save.py", line 872, in save
    signatures = signature_serialization.canonicalize_signatures(signatures)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/saved_model/signature_serialization.py", line 119, in canonicalize_signatures
    **tensor_spec_signature)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/def_function.py", line 776, in get_concrete_function
    self._initialize(args, kwargs, add_initializers_to=initializer_map)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/def_function.py", line 408, in _initialize
    *args, **kwds))
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 1848, in _get_concrete_function_internal_garbage_collected
    graph_function, _, _ = self._maybe_define_function(args, kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 2150, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 2041, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/func_graph.py", line 915, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/def_function.py", line 358, in wrapped_fn
    return weak_wrapped_fn().__wrapped__(*args, **kwds)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/func_graph.py", line 905, in wrapper
    raise e.ag_error_metadata.to_exception(e)
ValueError: in converted code:
    relative to /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/saved_model:

    signature_serialization.py:107 signature_wrapper  *
        return _normalize_outputs(
    signature_serialization.py:156 _normalize_outputs
        .format(value, key, compat.as_str_any(function_name)))

    ValueError: Got a dictionary containing non-Tensor value None for key __saved_model_init_op in the output of the function __inference_pruned_9207 used to generate a SavedModel signature. Dictionaries outputs for functions used as signatures should have one Tensor output per string key.

Please advise!! I’m close but no biscuit yet… Thanks :)

DaneLLL · December 9, 2019, 5:48am

Hi,
There is a sampleof TF-TRT conversion.

/usr/src/tensorrt/samples/sampleUffSSD/

Please check if it helps your case.

TP2049 · February 17, 2020, 9:20am

How is this an answer to my question?? Doesn’t help in the slightest. For a start the example is in c++ not python. And it doesn’t address the issues I was having. I swear this is the worse documented product I have ever come across. Every admin has a different answer for same problems. The official nvidia tensor setup guide is dated and doesn’t work either. One admin telling me I have to use Pure TensorRT not TF-TRT another admin says I should use TF-TRT on the nano not tesnorRT. It’s a nightmare.

How come the company that manufacters this doesn’t have an answer as to how to setup it’s own hardware?!???

AastaLLL · February 18, 2020, 6:19am

Hi,

Pure TensorRT will give you a better performance than TF-TRT, especially on memory.
It is known that TensorFlow tends to occupy all the GPU memory resource, which is not good for a shared memory system like Jetson.

For your TF-TRT issue:

Not found: Container TF-TRT does not exist. (Could not find resource: TF-TRT/TRTEngineOp_0

Based on the log, the error occurs in the package.
How do you install your TensorFlow package?

It is recommended to use JetPack4.3 and TensorFlow here:
https://docs.nvidia.com/deeplearning/frameworks/install-tf-jetson-platform/index.html

Thanks.

TP2049 · March 4, 2020, 7:14am

Ok I have since written my own trainer from my own data set. Before I was using saved model from googles teachable machine.

I still have same issue so must be the installation of TF or TRT.

Could you please help me figure out which version is incorrect.

I see this message at the top of my output:

WARNING:tensorflow:TensorRT mismatch. Compiled against version 5.0.6, but loaded 5.1.6. Things may not work

But I am not sure what it means as to which version of TF or TRT needs changing :/

Thank You!

So close… :D

TP2049 · March 5, 2020, 2:03pm

I got this back when checking TensorRT and TF versions:

dpkg -l | grep TensorRT
ii  graphsurgeon-tf                               5.1.6-1+cuda10.0                                 arm64        GraphSurgeon for TensorRT package
ii  libnvinfer-dev                                5.1.6-1+cuda10.0                                 arm64        TensorRT development libraries and headers
ii  libnvinfer-samples                            5.1.6-1+cuda10.0                                 all          TensorRT samples and documentation
ii  libnvinfer5                                   5.1.6-1+cuda10.0                                 arm64        TensorRT runtime libraries
ii  python-libnvinfer                             5.1.6-1+cuda10.0                                 arm64        Python bindings for TensorRT
ii  python-libnvinfer-dev                         5.1.6-1+cuda10.0                                 arm64        Python development package for TensorRT
ii  python3-libnvinfer                            5.1.6-1+cuda10.0                                 arm64        Python 3 bindings for TensorRT
ii  python3-libnvinfer-dev                        5.1.6-1+cuda10.0                                 arm64        Python 3 development package for TensorRT
ii  tensorrt                                      5.1.6.1-1+cuda10.0                               arm64        Meta package of TensorRT
ii  uff-converter-tf                              5.1.6-1+cuda10.0                                 arm64        UFF converter for TensorRT package

sudo pip list | grep tensorflow
tensorflow                    2.0.0              
tensorflow-estimator          2.0.1

I also have CUDA 10 and cudnn 7 installed. Using Ubuntu 18.04

So I am guessing the aforementioned warning means my tensorflow was compiled against tensorRT 5.0.6? How do I either update the TF2 installation or regress the TensorRT to 5.0.6? Thank Ye

Just for reference this is what my conversion code currently looks like:

from tensorflow.python.compiler.tensorrt import trt_convert as trt
import pathlib as plib

input_saved_model_dir = plib.Path('3conv-64nodes-2dense-CNN-001.model/')
output_saved_model_dir = plib.Path('outconv/')

input_saved_model_dir = str(input_saved_model_dir)
output_saved_model_dir = str(output_saved_model_dir)

params = trt.DEFAULT_TRT_CONVERSION_PARAMS._replace(
    precision_mode='FP16')

converter = trt.TrtGraphConverterV2(input_saved_model_dir=input_saved_model_dir, conversion_params=params)
converter.convert()
converter.save(output_saved_model_dir)

AastaLLL · March 16, 2020, 9:03am

Hi,

TensorRT 5.0.6 is from JetPack4.2.0.
You can reflash your TX2 with JetPack4.2.0 to get the TensorRT v5.0.6 package.

Thanks.

piotreko96 · April 27, 2020, 9:26am

Hey @TP2049,

when it comes to that warning:

OP_REQUIRES failed at trt_engine_resource_ops.cc:183 : Not found: Container TF-TRT does not exist. (Could not find resource: TF-TRT/TRTEngineOp_...

it looks like it may be ignored. Please check:

It states in Known Issues section.

Regards!