Please provide the following information when requesting support.
• Hardware is T4
• Network Type is Detectnet_v2, specifically trafficcamnet
Configuration of the TAO Toolkit Instance
dockers:
nvidia/tao/tao-toolkit:
4.0.0-tf2.9.1:
docker_registry: nvcr.io
tasks:
1. classification_tf2
2. efficientdet_tf2
4.0.0-tf1.15.5:
docker_registry: nvcr.io
tasks:
1. augment
2. bpnet
3. classification_tf1
4. detectnet_v2
5. dssd
6. emotionnet
7. efficientdet_tf1
8. faster_rcnn
9. fpenet
10. gazenet
11. gesturenet
12. heartratenet
13. lprnet
14. mask_rcnn
15. multitask_classification
16. retinanet
17. ssd
18. unet
19. yolo_v3
20. yolo_v4
21. yolo_v4_tiny
22. converter
4.0.1-tf1.15.5:
docker_registry: nvcr.io
tasks:
1. mask_rcnn
2. unet
4.0.0-pyt:
docker_registry: nvcr.io
tasks:
1. action_recognition
2. deformable_detr
3. segformer
4. re_identification
5. pointpillars
6. pose_classification
7. n_gram
8. speech_to_text
9. speech_to_text_citrinet
10. speech_to_text_conformer
11. spectro_gen
12. vocoder
13. text_classification
14. question_answering
15. token_classification
16. intent_slot_classification
17. punctuation_and_capitalization
format_version: 2.0
toolkit_version: 4.0.1
published_date: 03/06/2023
Inference Spec (just the inferencer_config part)
inferencer_config{
# defining target class names for the experiment.
# Note: This must be mentioned in order of the networks classes.
target_classes: "car"
target_classes: "bicycle"
target_classes: "person"
target_classes: "road_sign"
# Inference dimensions.
image_width: 960
image_height: 544
# Must match what the model was trained for.
image_channels: 3
batch_size: 1
gpu_index: 0
# model handler config
tensorrt_config{
etlt_model: "/workspace/tao-experiments/trafficcamnet/pretrained_pruned_v1.0.3/trafficcamnet_vpruned_v1.0.3/resnet18_trafficcamnet_pruned.etlt"
save_engine: true
trt_engine: "/workspace/tao-experiments/trafficcamnet/pretrained_pruned_v1.0.3/trafficcamnet_vpruned_v1.0.3/resnet18_trafficcamnet_pruned.etlt.trt"
backend_data_type: FP32
}
}
when I run tao detectnet_v2 inference calling this spec file on a small set of images, I get the following error
2023-04-12 00:16:16,882 [INFO] root: Registry: ['nvcr.io']
2023-04-12 00:16:17,140 [INFO] tlt.components.instance_handler.local_instance: Running command in container: nvcr.io/nvidia/tao/tao-toolkit:4.0.0-tf1.15.5
Using TensorFlow backend.
2023-04-12 00:16:24.756665: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
/usr/local/lib/python3.6/dist-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (3.0.4) doesn't match a supported version!
RequestsDependencyWarning)
Matplotlib created a temporary config/cache directory at /tmp/matplotlib-t28tkc0v because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
/usr/local/lib/python3.6/dist-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (3.0.4) doesn't match a supported version!
RequestsDependencyWarning)
INFO: Merging specification from /workspace/tao-experiments/trafficcamnet/specs/trafficcamnet_pruned_inference_kitti_etlt.txt
INFO: Overlain images will be saved in the output path.
INFO: Constructing inferencer
INFO: Engine file not found at /workspace/tao-experiments/trafficcamnet/pretrained_pruned_v1.0.3/trafficcamnet_vpruned_v1.0.3/resnet18_trafficcamnet_pruned.etlt.trt
INFO: Using TensorRT to optimize model and generate an engine.
[04/12/2023-00:16:39] [TRT] [W] CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage. See `CUDA_MODULE_LOADING` in https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars
INFO: Initializing int8 calibration table.
INFO: A valid tensorfile doesn't exist at None. The calibrator will attempt to read from a cache file if provided.
[04/12/2023-00:16:39] [TRT] [W] The implicit batch dimension mode has been deprecated. Please create the network with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag whenever possible.
Traceback (most recent call last):
File "</usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/scripts/inference.py>", line 3, in <module>
File "<frozen iva.detectnet_v2.scripts.inference>", line 210, in <module>
File "<frozen iva.detectnet_v2.scripts.inference>", line 206, in main
File "<frozen iva.detectnet_v2.scripts.inference>", line 117, in inference_wrapper_batch
File "<frozen iva.detectnet_v2.inferencer.trt_inferencer>", line 345, in network_init
File "<frozen iva.detectnet_v2.inferencer.trt_inferencer>", line 213, in _parse_caffe_model
File "/usr/lib/python3.6/genericpath.py", line 30, in isfile
st = os.stat(path)
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType
The error does not seem very informative, but when I use a “tlt_config” in the spec file, e.g. with the unpruned resnet18_trafficcamnet.tlt model, the inference works. Any suggestions?