Editing detectnet.py from hello ai world to run a custom model

Autonomous Machines Jetson & Embedded Systems Jetson Nano
1

Hi,

I am currently trying to edit the detectnet.py code given in the tutorial:

My aim is to have it run my own custom trained model instead of ssd-mobilenet

I have made the changes i think neccesary but get the following error:

$ /usr/bin/python3 /home/james/findingfauna_pythonfiles/inference/detectinf_file_file_experimental.py
jetson.inference – detectNet loading build-in network ‘/home/james/jetson-inference/python/training/detection/ssd/models/whale_openimages/1/ssd-mobilenet.onnx’
jetson.inference – detectNet invalid built-in network was requested (‘/home/james/jetson-inference/python/training/detection/ssd/models/whale_openimages/1/ssd-mobilenet.onnx’)
Traceback (most recent call last):
File “/home/james/findingfauna_pythonfiles/inference/detectinf_file_file_experimental.py”, line 36, in
net = jetson.inference.detectNet(opt.model, opt.labels, opt.threshold)
Exception: jetson.inference – detectNet invalid built-in network was requested

I dont understand this error as i am not trying to request a built in network im specifying to utilise my own custom model.

My full code is below, the changes made are on lines 20, 21 and 36.

Any advice would be appreciated.

Thanks

import jetson.inference
import jetson.utils
import numpy as np
import argparse
import sys
import os, os.path

image_in = “/home/james/test_input/”
image_out = “/home/james/test_output/anomaly_%i*.jpg”
model_path = “ssd-mobilenet-v2”
custom_model_path = “/home/james/jetson-inference/python/training/detection/ssd/models/whale_openimages/1/ssd-mobilenet.onnx”
custom_labels_path = “/home/james/jetson-inference/python/training/detection/ssd/models/whale_openimages/1/labels.txt”

parse the command line

parser = argparse.ArgumentParser(description=“Locate objects in a live camera stream using an object detection DNN.”)

parser.add_argument(“input_URI”, type=str, default=image_in, nargs=‘?’, help=“URI of the input stream”)
parser.add_argument(“output_URI”, type=str, default=image_out, nargs=‘?’, help=“URI of the output stream”)
parser.add_argument(“–model”, type=str, default=custom_model_path, help=“path to your model” )
parser.add_argument(“–labels”, type=str, default=custom_labels_path, help=“path to your labels” )
parser.add_argument(“–network”, type=str, default=model_path, help=“pre-trained model to load (see below for options)”)
parser.add_argument(“–overlay”, type=str, default=“box,labels,conf”, help=“detection overlay flags (e.g. --overlay=box,labels,conf)\nvalid combinations are: ‘box’, ‘labels’, ‘conf’, ‘none’”)
parser.add_argument(“–threshold”, type=float, default=0.5, help=“minimum detection threshold to use”)

try:
opt = parser.parse_known_args()[0]
except:
print(“”)
parser.print_help()
sys.exit(0)

load the object detection network

#net = jetson.inference.detectNet(opt.network, sys.argv, opt.threshold)
#net = jetson.inference.detectNet(custom_model_path, custom_labels_path, threshold=0.5)
net = jetson.inference.detectNet(opt.model, opt.labels, opt.threshold)

create video sources & outputs

input = jetson.utils.videoSource(opt.input_URI, argv=sys.argv)
output = jetson.utils.videoOutput(opt.output_URI, argv=sys.argv)

process frames until the user exits

while True:
# capture the next image
img = input.Capture()
print(“this is the image”, img)
# detect objects in the image (with overlay)
detections = net.Detect(img, overlay=opt.overlay)

# print the detections
print("detected {:d} objects in image".format(len(detections)))

for detection in detections:

	print(detection)

# render the image
output.Render(img)

# update the title bar
output.SetStatus("{:s} | Network {:.0f} FPS".format(opt.network, net.GetNetworkFPS()))

# print out performance info
net.PrintProfilerTimes()

# exit on input/output EOS
if not input.IsStreaming() or not output.IsStreaming():
	break
2

Hi @jeheesom, try changing your code to be like this instead:

net = jetson.inference.detectNet(argv=['--model=/home/james/jetson-inference/python/training/detection/ssd/models/whale_openimages/1/ssd-mobilenet.onnx', '--labels=/home/james/jetson-inference/python/training/detection/ssd/models/whale_openimages/1/labels.txt', '--input-blob=input_0', '--output-cvg=scores', '--output-bbox=boxes'])
3

worked perfectly thank you

4

HI NVIDIA Developers,

I want to do exactly the same thing i.e. to be able to use my own model but in my case I would like to edit the detectNet-camera.py file since I want to do real time processing. I noticed that the python codes detectNet.py and detectNet-camera.py were identical.
Can the same changes be applied?

Basically, I had a saved_model.pb that I optimized via TF-TRT in dynamic mode (creation of engines during inference). The problem is that when I want to do the onnx conversion, it looks for the TRT Engines…

2021-04-07 09:39:16.584977: I tensorflow/stream_executor/platform/default/dso_lo                                                                                             ader.cc:49] Successfully opened dynamic library libcudart.so.10.2
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPREC                                                                                             ATION_WARNINGS=1 to re-enable them.
/usr/lib/python3.6/runpy.py:125: RuntimeWarning: 'tf2onnx.convert' found in sys.                                                                                             modules after import of package 'tf2onnx', but prior to execution of 'tf2onnx.co                                                                                             nvert'; this may result in unpredictable behaviour
  warn(RuntimeWarning(msg))
WARNING:tensorflow:From /home/pi21/.local/lib/python3.6/site-packages/tf2onnx/ve                                                                                             rbose_logging.py:76: The name tf.logging.set_verbosity is deprecated. Please use                                                                                              tf.compat.v1.logging.set_verbosity instead.

2021-04-07 09:39:21,257 - WARNING - From /home/pi21/.local/lib/python3.6/site-pa                                                                                             ckages/tf2onnx/verbose_logging.py:76: The name tf.logging.set_verbosity is depre                                                                                             cated. Please use tf.compat.v1.logging.set_verbosity instead.

2021-04-07 09:39:21.303585: I tensorflow/stream_executor/platform/default/dso_lo                                                                                             ader.cc:49] Successfully opened dynamic library libcuda.so.1
2021-04-07 09:39:21.317950: I tensorflow/stream_executor/cuda/cuda_gpu_executor.                                                                                             cc:1049] ARM64 does not support NUMA - returning NUMA node zero
2021-04-07 09:39:21.318090: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1                                                                                             666] Found device 0 with properties:
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
pciBusID: 0000:00:00.0
2021-04-07 09:39:21.318161: I tensorflow/stream_executor/platform/default/dso_lo                                                                                             ader.cc:49] Successfully opened dynamic library libcudart.so.10.2
2021-04-07 09:39:21.321952: I tensorflow/stream_executor/platform/default/dso_lo                                                                                             ader.cc:49] Successfully opened dynamic library libcublas.so.10
2021-04-07 09:39:21.325099: I tensorflow/stream_executor/platform/default/dso_lo                                                                                             ader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-04-07 09:39:21.326041: I tensorflow/stream_executor/platform/default/dso_lo                                                                                             ader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-04-07 09:39:21.330785: I tensorflow/stream_executor/platform/default/dso_lo                                                                                             ader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-04-07 09:39:21.334901: I tensorflow/stream_executor/platform/default/dso_lo                                                                                             ader.cc:49] Successfully opened dynamic library libcusparse.so.10
2021-04-07 09:39:21.335478: I tensorflow/stream_executor/platform/default/dso_lo                                                                                             ader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-04-07 09:39:21.335764: I tensorflow/stream_executor/cuda/cuda_gpu_executor.                                                                                             cc:1049] ARM64 does not support NUMA - returning NUMA node zero
2021-04-07 09:39:21.336019: I tensorflow/stream_executor/cuda/cuda_gpu_executor.                                                                                             cc:1049] ARM64 does not support NUMA - returning NUMA node zero
2021-04-07 09:39:21.336143: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1                                                                                             794] Adding visible gpu devices: 0
2021-04-07 09:39:21.358093: W tensorflow/core/platform/profile_utils/cpu_utils.c                                                                                             c:98] Failed to find bogomips in /proc/cpuinfo; cannot determine CPU frequency
2021-04-07 09:39:21.358646: I tensorflow/compiler/xla/service/service.cc:168] XL                                                                                             A service 0x3c5fead0 initialized for platform Host (this does not guarantee that                                                                                              XLA will be used). Devices:
2021-04-07 09:39:21.358717: I tensorflow/compiler/xla/service/service.cc:176]                                                                                                StreamExecutor device (0): Host, Default Version
2021-04-07 09:39:21.433903: I tensorflow/stream_executor/cuda/cuda_gpu_executor.                                                                                             cc:1049] ARM64 does not support NUMA - returning NUMA node zero
2021-04-07 09:39:21.434178: I tensorflow/compiler/xla/service/service.cc:168] XL                                                                                             A service 0x3baf20a0 initialized for platform CUDA (this does not guarantee that                                                                                              XLA will be used). Devices:
2021-04-07 09:39:21.434229: I tensorflow/compiler/xla/service/service.cc:176]                                                                                                StreamExecutor device (0): NVIDIA Tegra X1, Compute Capability 5.3
2021-04-07 09:39:21.434609: I tensorflow/stream_executor/cuda/cuda_gpu_executor.                                                                                             cc:1049] ARM64 does not support NUMA - returning NUMA node zero
2021-04-07 09:39:21.434713: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1                                                                                             666] Found device 0 with properties:
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
pciBusID: 0000:00:00.0
2021-04-07 09:39:21.434788: I tensorflow/stream_executor/platform/default/dso_lo                                                                                             ader.cc:49] Successfully opened dynamic library libcudart.so.10.2
2021-04-07 09:39:21.434869: I tensorflow/stream_executor/platform/default/dso_lo                                                                                             ader.cc:49] Successfully opened dynamic library libcublas.so.10
2021-04-07 09:39:21.434926: I tensorflow/stream_executor/platform/default/dso_lo                                                                                             ader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-04-07 09:39:21.434977: I tensorflow/stream_executor/platform/default/dso_lo                                                                                             ader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-04-07 09:39:21.435027: I tensorflow/stream_executor/platform/default/dso_lo                                                                                             ader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-04-07 09:39:21.435078: I tensorflow/stream_executor/platform/default/dso_lo                                                                                             ader.cc:49] Successfully opened dynamic library libcusparse.so.10
2021-04-07 09:39:21.435130: I tensorflow/stream_executor/platform/default/dso_lo                                                                                             ader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-04-07 09:39:21.435288: I tensorflow/stream_executor/cuda/cuda_gpu_executor.                                                                                             cc:1049] ARM64 does not support NUMA - returning NUMA node zero
2021-04-07 09:39:21.435473: I tensorflow/stream_executor/cuda/cuda_gpu_executor.                                                                                             cc:1049] ARM64 does not support NUMA - returning NUMA node zero
2021-04-07 09:39:21.435535: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1                                                                                             794] Adding visible gpu devices: 0
2021-04-07 09:39:21.435619: I tensorflow/stream_executor/platform/default/dso_lo                                                                                             ader.cc:49] Successfully opened dynamic library libcudart.so.10.2
2021-04-07 09:39:23.952042: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1206] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-04-07 09:39:23.952126: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1212]      0
2021-04-07 09:39:23.952188: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1225] 0:   N
2021-04-07 09:39:23.952567: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1049] ARM64 does not support NUMA - returning NUMA node zero
2021-04-07 09:39:23.952837: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1049] ARM64 does not support NUMA - returning NUMA node zero
2021-04-07 09:39:23.952985: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1351] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 468 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
2021-04-07 09:39:23,956 - WARNING - '--tag' not specified for saved_model. Using --tag serve
2021-04-07 09:39:23,956 - WARNING - '--signature_def' not provided. Using all signatures.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1348, in _run_fn
    self._extend_graph()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1388, in _extend_graph
    tf_session.ExtendSession(self._session)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation Preprocessor/TRTEngineOp_4: Could not satisfy explicit device specification '/device:CPU:0' because no supported kernel for CPU devices is available.
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:CPU:0' assigned_device_name_='' resource_device_name_='' supported_device_types_=[GPU] possible_devices_=[]
TRTEngineOp: GPU

Colocation members, user-requested devices, and framework assigned devices, if any:
  Preprocessor/TRTEngineOp_4 (TRTEngineOp) /device:CPU:0

Op: TRTEngineOp
Node attrs: output_shapes=[[?,?,?,3]], workspace_size_bytes=2995095, max_cached_engines_count=1, segment_func=Preprocessor/TRTEngineOp_4_native_segment[], segment_funcdef_name="", use_calibration=false, fixed_input_size=true, input_shapes=[[?,?,?,3]], OutT=[DT_FLOAT], precision_mode="FP32", static_engine=false, serialized_segment="", cached_engine_batches=[], InT=[DT_FLOAT], calibration_data=""
Registered kernels:
  device='GPU'

         [[{{node Preprocessor/TRTEngineOp_4}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/pi21/.local/lib/python3.6/site-packages/tf2onnx/convert.py", line 488, in <module>
    main()
  File "/home/pi21/.local/lib/python3.6/site-packages/tf2onnx/convert.py", line 213, in main
    args.large_model, return_initialized_tables=True, return_tensors_to_rename=True)
  File "/home/pi21/.local/lib/python3.6/site-packages/tf2onnx/tf_loader.py", line 523, in from_saved_model
    _from_saved_model_v1(sess, model_path, input_names, output_names, tag, signatures)
  File "/home/pi21/.local/lib/python3.6/site-packages/tf2onnx/tf_loader.py", line 320, in _from_saved_model_v1
    tf.tables_initializer().run()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 2439, in run
    _run_using_default_session(self, feed_dict, self.graph, session)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 5442, in _run_using_default_session
    session.run(operation, feed_dict)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 956, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1180, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
    run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation Preprocessor/TRTEngineOp_4: Could not satisfy explicit device specification '/device:CPU:0' because no supported kernel for CPU devices is available.
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:CPU:0' assigned_device_name_='' resource_device_name_='' supported_device_types_=[GPU] possible_devices_=[]
TRTEngineOp: GPU

Colocation members, user-requested devices, and framework assigned devices, if any:
  Preprocessor/TRTEngineOp_4 (TRTEngineOp) /device:CPU:0

Op: TRTEngineOp
Node attrs: output_shapes=[[?,?,?,3]], workspace_size_bytes=2995095, max_cached_engines_count=1, segment_func=Preprocessor/TRTEngineOp_4_native_segment[], segment_funcdef_name="", use_calibration=false, fixed_input_size=true, input_shapes=[[?,?,?,3]], OutT=[DT_FLOAT], precision_mode="FP32", static_engine=false, serialized_segment="", cached_engine_batches=[], InT=[DT_FLOAT], calibration_data=""
Registered kernels:
  device='GPU'

         [[node Preprocessor/TRTEngineOp_4 (defined at usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]]

I saw that in the Hello Ai World code, there were 3 files:

  • the labels file
  • the model.uff file
  • the .engine file

Would the conversion of my model.pb to .uff work? Is it easy to create the .engine file ?

Thanks !

Paul Griffoul

5

You will want to edit detectnet.py - during cmake step, detectnet-camera.py is copied from detectnet.py for legacy reasons.

This was the tool used to convert the TensorFlow model: GitHub - AastaNV/TRT_object_detection: Python sample for referencing object detection model with TensorRT

6

Hi @dusty_nv,

Thanks for your answer.

As a reminder, I trained my own neural network to do detection and I optimized it with Tensorflow-TensorRT to be able to use it on the Jetson Nano and get better performances.

I thought I would try to convert my optimized saved_model.pb to .uff and .engine and add it to the files in the Hello AI World project and then call my model in the detectNet code.
If I understood correctly, I can’t do this conversion since in the following link there are only 3 models supported and I can’t use my own model: GitHub - AastaNV/TRT_object_detection: Python sample for referencing object detection model with TensorRT

Is this correct?
I also read that you need to have a frozen model to be able to do the conversion to uff.

The second possibility was to convert it to .onnx but I have the error message in the previous comment preventing me from doing this conversion. Is it necessary to switch to static mode when optimizing Tensorflow-TensorRT for it to work properly?

Would you have a solution for me ?
Thanks

Paul Griffoul

7

I have only done the conversion on SSD-Mobilenet that is from that repo, so unfortunately I can’t speak to the TensorFlow conversion of other models (although I believe the process to be similar). You can find the docs on TensorFlow->TensorRT conversion here:

https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#working_tf

It seems you encountered an error during the ONNX export, unfortunately I’m not familiar with exporting ONNX from TensorFlow. However there is a third option you can try, which is TF-TRT interoperability. It will run TensorRT for the layers that it can and fallback to TensorFlow for the others. You can find docs on TF-TRT API through the link above.

If you have further questions or issues on the TensorFlow conversion, please create a new topic as this area is not my specialty. Thanks and good luck!