Hello everyone.
I’m here because I’ve been unable to finish the execution of the faster-rcnn tlt example.
I would appreciate any help with this, and I’ll deliver any info that you need.
=========== SOME INFO ===========
I got my key and I replaced my environmental variables correctly, the folder map is also validated.
The only other changes made are the batch sizes since video card memory is only 2 GB.
Everything else is exactly as shown in the example, I’m even using the exact same dataset provided.
I got the docker from here: Transfer Learning Toolkit for Video Streaming Analytics | NVIDIA NGC
using the provided command: docker pull nvcr.io/nvidia/tlt-streamanalytics:v2.0_dp_py2
I have a Nvidia Jetson Nano, which is incompatible with tlt, so I’m running tlt on my computer.
I’ve deactivated the gui on the Jetson Nano, I’m operating only on console mode and by ssh and shared folders with other computers on the network, in order to save some resources.
=========== Errors ===========
I can go through the tutorial smoothly until section eight. I can tarin the model, prune it, retrain the pruned model, and evaluate its metrics.
Problems start with this cell
# Running inference for detection on n images
# Please go to $USER_EXPERIMENT_DIR/data/faster_rcnn/inference_results_imgs_retrain to see the visualizatons.
!tlt-infer faster_rcnn -e $SPECS_DIR/default_spec_resnet18_retrain_spec.txt
Error log
Using TensorFlow backend.
2020-06-08 18:11:24,442 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/spec_loader/spec_loader.pyc: Loading experiment spec at /workspace/examples/faster_rcnn/specs/default_spec_resnet18_retrain_spec.txt.
2020-06-08 18:11:24,471 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/inference.pyc: Running inference with TensorRT as backend.
2020-06-08 18:11:24,480 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/tensorrt_inference/tensorrt_model.pyc: Loading TensorRT engine file: /workspace/tlt-experiments/data/faster_rcnn/trt.fp16.engine
for inference.
2020-06-08 18:11:26,201 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/inference.pyc: 000000.png
[TensorRT] WARNING: Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles
#assertion/trt_oss_src/TensorRT/plugin/common/kernels/proposalKernel.cu,709
Aborted (core dumped)
so I can’t see resulting images when the net works.
I have troubles on section nine too. When I try to export the net I got these messages.
Exporting to FP32.
# Export in FP32 mode. \
!tlt-export faster_rcnn -m $USER_EXPERIMENT_DIR/data/faster_rcnn/frcnn_kitti_resnet18_retrain.epoch12.tlt \
-o $USER_EXPERIMENT_DIR/data/faster_rcnn/frcnn_kitti_resnet18_retrain.etlt \
-e $SPECS_DIR/default_spec_resnet18_retrain_spec.txt \
-k $KEY
Error log
Using TensorFlow backend.
2020-06-08 20:24:11,397 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/spec_loader/spec_loader.pyc: Loading experiment spec at /workspace/examples/faster_rcnn/specs/default_spec_resnet18_retrain_spec.txt.
2020-06-08 20:25:02,715 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/spec_loader/spec_loader.pyc: Loading experiment spec at /workspace/examples/faster_rcnn/specs/default_spec_resnet18_retrain_spec.txt.
NOTE: UFF has been tested with TensorFlow 1.14.0.
WARNING: The version of TensorFlow installed on this system is not guaranteed to work with UFF.
DEBUG: convert reshape to flatten node
Warning: No conversion function registered for layer: CropAndResize yet.
Converting roi_pooling_conv_1/CropAndResize_new as custom op: CropAndResize
Warning: No conversion function registered for layer: Proposal yet.
Converting proposal as custom op: Proposal
DEBUG [/usr/lib/python2.7/dist-packages/uff/converters/tensorflow/converter.py:96] Marking ['proposal', 'dense_class_td/Softmax', 'dense_regress_td/BiasAdd'] as outputs
[TensorRT] INFO: Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output.
[TensorRT] INFO: Detected 1 inputs and 3 output network tensors.
Exporting to FP16.
# Export in FP16 mode. \
# Note that the .etlt model in FP16 mode is \
# the same as in FP32 mode. \
!rm $USER_EXPERIMENT_DIR/data/faster_rcnn/frcnn_kitti_resnet18_retrain_fp16.etlt
!tlt-export faster_rcnn -m $USER_EXPERIMENT_DIR/data/faster_rcnn/frcnn_kitti_resnet18_retrain.epoch12.tlt \
-o $USER_EXPERIMENT_DIR/data/faster_rcnn/frcnn_kitti_resnet18_retrain_fp16.etlt \
-e $SPECS_DIR/default_spec_resnet18_retrain_spec.txt \
-k $KEY \
--data_type fp16
Error log
Using TensorFlow backend.
2020-06-08 20:37:51,065 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/spec_loader/spec_loader.pyc: Loading experiment spec at /workspace/examples/faster_rcnn/specs/default_spec_resnet18_retrain_spec.txt.
2020-06-08 20:38:41,524 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/spec_loader/spec_loader.pyc: Loading experiment spec at /workspace/examples/faster_rcnn/specs/default_spec_resnet18_retrain_spec.txt.
NOTE: UFF has been tested with TensorFlow 1.14.0.
WARNING: The version of TensorFlow installed on this system is not guaranteed to work with UFF.
DEBUG: convert reshape to flatten node
Warning: No conversion function registered for layer: CropAndResize yet.
Converting roi_pooling_conv_1/CropAndResize_new as custom op: CropAndResize
Warning: No conversion function registered for layer: Proposal yet.
Converting proposal as custom op: Proposal
DEBUG [/usr/lib/python2.7/dist-packages/uff/converters/tensorflow/converter.py:96] Marking ['proposal', 'dense_class_td/Softmax', 'dense_regress_td/BiasAdd'] as outputs
2020-06-08 20:39:41,057 [ERROR] modulus.export._tensorrt: Specified FP16 but not supported on platform.
Traceback (most recent call last):
File "/usr/local/bin/tlt-export", line 8, in <module>
sys.exit(main())
File "./common/export/app.py", line 234, in main
File "./common/export/base_exporter.py", line 411, in export
File "./modulus/export/_tensorrt.py", line 515, in __init__
File "./modulus/export/_tensorrt.py", line 380, in __init__
AttributeError: Specified FP16 but not supported on platform.
=========== Discussion ===========
It’s strange to find the warning about the version of TensorFlow installed not being guaranteed to work with UFF, since it’s the images installed version.
I also tryed to parse the FP32 model and the last retrain epoch to the deepstream-app on my jetson nano, the idea was letting it to create the engine file as it did every time I parsed another example, but I got a core dumped.
How could I get the image model to continue with the developing process?
=========== HARDWARE DETAILS ===========
Machine: Device: laptop System: Dell product: Inspiron 7559 v: 1.2.9 serial: N/A
UEFI: Dell v: 1.2.9 date: 09/03/2018
CPU: Quad core Intel Core i7-6700HQ (-MT-MCP-) cache: 6144 KB
clock speeds: max: 3500 MHz 1: 1513 MHz 2: 2220 MHz 3: 1842 MHz
4: 1430 MHz 5: 1860 MHz 6: 1892 MHz 7: 1971 MHz 8: 1995 MHz
Graphics: Card-1: Intel HD Graphics 530
Card-2: NVIDIA GM107M [GeForce GTX 960M]
Display Server: x11 (X.Org 1.20.5 )
drivers: modesetting,nvidia (unloaded: fbdev,vesa,nouveau)
Resolution: 1920x1080@60.02hz
OpenGL: renderer: GeForce GTX 960M/PCIe/SSE2
version: 4.6.0 NVIDIA 440.82
Drives: HDD Total Size: 750.2GB (14.9% used)
ID-1: /dev/sda model: Samsung_SSD_860 size: 250.1GB
ID-2: /dev/sdb model: CT500MX500SSD1 size: 500.1GB
Info: Processes: 365 Uptime: 3:00 Memory: 3920.5/7827.7MB
Client: Shell (bash) inxi: 2.3.56
=========== SOFTWARE DETAILS ===========
System: Host: jpablo-Inspiron-7559 Kernel: 5.3.0-53-generic x86_64
bits: 64
Desktop: Gnome 3.28.4 Distro: Ubuntu 18.04.4 LTS
Docker: Server Version: 19.03.8
Image repository: nvcr.io/nvidia/tlt-streamanalytics
Image tag: v2.0_dp_py2
Nvidia: Driver Version: 440.82
CUDA Version 10.2.89
Machine: Device: laptop System: Dell product: Inspiron 7559 v: 1.2.9 serial: N/A
UEFI: Dell v: 1.2.9 date: 09/03/2018
============== ON JETSON TEST ============
I tryed to make the deepstream-app to compile the net directly on the Jetson Nano.
Here is an extract of the config file that’s suppoused to use the net
[primary-gie]
enable=1
gpu-id=0
model-engine-file=/opt/nvidia/deepstream/deepstream/controlflow/models/Controlflow_tlt/frcnn_kitti_resnet18_retrain.epoch12.tlt.engine
batch-size=8
#Required by the app for OSD, not a plugin property
## 0=FP32, 1=INT8, 2=FP16 mode
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;0;1;1
bbox-border-color3=0;1;0;1
interval=10
gie-unique-id=1
nvbuf-memory-type=2
config-file=config_infer_controlflow.txt
And here is the config_infer_controlflow.txt
# Copyright (c) 2020 NVIDIA Corporation. All rights reserved.
#
# NVIDIA Corporation and its licensors retain all intellectual property
# and proprietary rights in and to this software, related documentation
# and any modifications thereto. Any use, reproduction, disclosure or
# distribution of this software and related documentation without an express
# license agreement from NVIDIA Corporation is strictly prohibited.
[property]
gpu-id=0
net-scale-factor=0.0039215697906911373
tlt-model-key=<I'm not posting my key on the internet>
#tlt-encoded-model=../models/Controlflow_tlt/frcnn_kitti_resnet18_retrain_pf16.etlt
#tlt-encoded-model=../models/Controlflow_tlt/frcnn_kitti_resnet18_retrain.epoch12.tlt
tlt-encoded-model=../models/Controlflow_tlt/frcnn_kitti_resnet18_retrain.etlt
labelfile-path=../models/Controlflow_tlt/labels.txt
#int8-calib-file=../models/Controlflow_tlt/dashcamnet_int8.txt
#model-engine-file=../models/Controlflow_tlt/frcnn_kitti_resnet18_retrain_fp16.etlt.engine
#model-engine-file=../models/Controlflow_tlt/frcnn_kitti_resnet18_retrain.epoch12.tlt.engine
model-engine-file=../models/Controlflow_tlt/frcnn_kitti_resnet18_retrain.etlt.engine
#input-dims=3;384;1248;0
input-dims=3;544;960;0
uff-input-blob-name=input_1
batch-size= 1 #8 #3
process-mode=1
model-color-format=0
## 0=FP32, 1=INT8, 2=FP16 mode
network-mode=0
num-detected-classes=7
interval=2
gie-unique-id=1
output-blob-names=output_bbox/BiasAdd;output_cov/Sigmoid
[class-attrs-all]
pre-cluster-threshold=0.2
group-threshold=1
## Set eps=0.7 and minBoxes for cluster-mode=1(DBSCAN)
eps=0.2
#minBoxes=3
roi-top-offset=0
roi-bottom-offset=0
detected-min-w=0
detected-min-h=0
detected-max-w=0
detected-max-h=0
================= Clarification ===============
I’m running the tlt image on my personal computer, described above, not on the jetson nano.