Tao export

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

tlt.components.instance_handler.local_instance: Running command in container: nvcr.io/nvidia/tao/tao-toolkit-tf:v3.22.05-tf1.15.5-py3
Using TensorFlow backend.
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
/usr/local/lib/python3.6/dist-packages/requests/init.py:91: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (3.0.4) doesn’t match a supported version!
RequestsDependencyWarning)
/usr/local/lib/python3.6/dist-packages/numba/cuda/envvars.py:17: NumbaWarning:
Environment variables with the ‘NUMBAPRO’ prefix are deprecated and consequently ignored, found use of NUMBAPRO_NVVM=/usr/local/cuda/nvvm/lib64/libnvvm.so.

For more information about alternatives visit: (‘Overview — Numba 0.50.1 documentation’, ‘#cudatoolkit-lookup’)
warnings.warn(errors.NumbaWarning(msg))
/usr/local/lib/python3.6/dist-packages/numba/cuda/envvars.py:17: NumbaWarning:
Environment variables with the ‘NUMBAPRO’ prefix are deprecated and consequently ignored, found use of NUMBAPRO_LIBDEVICE=/usr/local/cuda/nvvm/libdevice/.

For more information about alternatives visit: (‘Overview — Numba 0.50.1 documentation’, ‘#cudatoolkit-lookup’)
warnings.warn(errors.NumbaWarning(msg))
2022-06-23 09:23:16,497 [INFO] iva.common.export.keras_exporter: Using input nodes: [‘input_1’]
2022-06-23 09:23:16,498 [INFO] iva.common.export.keras_exporter: Using output nodes: [‘predictions/Softmax’]
NOTE: UFF has been tested with TensorFlow 1.14.0.
WARNING: The version of TensorFlow installed on this system is not guaranteed to work with UFF.
DEBUG: convert reshape to flatten node
DEBUG [/usr/local/lib/python3.6/dist-packages/uff/converters/tensorflow/converter.py:96] Marking [‘predictions/Softmax’] as outputs
2022-06-23 09:23:20,405 [INFO] iva.common.export.keras_exporter: Calibration takes time especially if number of batches is large.
terminate called after throwing an instance of ‘pybind11::error_already_set’
what(): ValueError: Batch size yielded from data source 8 < requested batch size from calibrator 16

At:
/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/export/tensorfile_calibrator.py(79): get_data_from_source
/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/export/tensorfile_calibrator.py(95): get_batch
/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/core/build_wheel.runfiles/ai_infra/moduluspy/modulus/export/_tensorrt.py(537): init
/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/core/build_wheel.runfiles/ai_infra/moduluspy/modulus/export/_tensorrt.py(696): init
/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/export/keras_exporter.py(445): export
/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/export/app.py(250): run_export
/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/makenet/scripts/export.py(42): main
/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/makenet/scripts/export.py(46):

2022-06-23 14:53:24,294 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.

3

Please share full command. Thanks.
Also please share the spec file as well.

4

!tao classification export
-m $USER_EXPERIMENT_DIR/output/weights/resnet_450.tlt
-o $USER_EXPERIMENT_DIR/output/weights/final_model.etlt
-k $KEY
–cal_data_file $USER_EXPERIMENT_DIR/output/weights/calibration.tensor
–data_type int8
–batches 16
–cal_cache_file $USER_EXPERIMENT_DIR/outpu/weights/final_model_int8_cache.bin
-v

5

model_config {
arch: “resnet”,
n_layers: 10

Setting these parameters to true to match the template downloaded from NGC.

use_batch_norm: true
all_projections: true
freeze_blocks: 0
freeze_blocks: 1
input_image_size: “3,224,224”
}
train_config {
train_dataset_path: “/workspace/tao-experiments/data/split/train”
val_dataset_path: “/workspace/tao-experiments/data/split/val”
pretrained_model_path: “/workspace/tao-experiments/classification/pretrained_resnet10/pretrained_classification_vresnet10/resnet_10.hdf5”
optimizer {
sgd {
lr: 0.01
decay: 0.0
momentum: 0.9
nesterov: False
}
}
batch_size_per_gpu: 8
n_epochs: 450
n_workers: 4
preprocess_mode: “caffe”
enable_random_crop: True
enable_center_crop: True
label_smoothing: 0.0
mixup_alpha: 0.1

regularizer

reg_config {
type: “L2”
scope: “Conv2D,Dense”
weight_decay: 0.00005
}

learning_rate

lr_config {
step {
learning_rate: 0.001
step_size: 10
gamma: 0.1
}
}
}
eval_config {
eval_dataset_path: “/workspace/tao-experiments/data/split/test”
model_path: “/workspace/tao-experiments/classification/output/weights/resnet_450.tlt”
top_k: 3
batch_size: 1
n_workers: 4
enable_center_crop: True
}

6

Anything yet?

7

How many images in your training dataset and val dataset?

8

TRAIN IMAGE COUNT

DATA1 11297
DATA2 3009
DATA3 4656
DATA4 615
DATA5 98
… 839
… 856
… 7332
… 210
… 634
… 271
… 656
… 672
… 541
… 101
…1456
… 3185
… 145
… 1088
… 1546
… 87

VAL IMAGE COUNT

DATA1 1614
DATA2 430
DATA3 665
DATA4 88
DATA5 14
… 120
… 123
… 1048
… 30
… 90
… 39
… 94
… 96
… 78
… 15
… 208
… 455
… 21
… 156
… 221
… 13

This is count per class of my training and validation datasset.

9

Please run below and share the result.
! tao classification run ls /workspace/tao-experiments/data/split/train
! tao classification run ls /workspace/tao-experiments/data/split/val
! tao classification run ls /workspace/tao-experiments/data/split/test

10

DATA1 DATA2 DATA3 DATA4 DATA5 DATA6 DATA7
DATA8 DATA9 DATA10 DATA11 DATA12 DATA13 DATA14
DATA15 DATA16 DATA17 DATA18 DATA19 DATA20 DATA21

These are the same results for train / val / test

11

Make sure the tensor file is available.
! tao classification run ls $USER_EXPERIMENT_DIR/output/weights/calibration.tensor

12

I have generated the file

13

Can you set "-m 8 " when you run
! tao classification calibration_tensorfile

14

Did that before posting the error.
Tried with 8, 16, 4 values

15

2022-06-23 16:19:58,454 [INFO] root: Registry: [‘nvcr.io’]
2022-06-23 16:19:58,533 [INFO] tlt.components.instance_handler.local_instance: Running command in container: nvcr.io/nvidia/tao/tao-toolkit-tf:v3.22.05-tf1.15.5-py3
Using TensorFlow backend.
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
/usr/local/lib/python3.6/dist-packages/requests/init.py:91: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (3.0.4) doesn’t match a supported version!
RequestsDependencyWarning)
2022-06-23 10:50:04,522 [INFO] main: Loading experiment spec at /workspace/tao-experiments/classification/specs/classification_spec.cfg
2022-06-23 10:50:04,523 [INFO] main: Setting up input generator.
Found 39294 images belonging to 21 classes.
Writing calibration tensorfile: 100%|█████████████| 8/8 [00:00<00:00, 9.46it/s]
2022-06-23 10:50:06,137 [INFO] main: Calibration tensorfile written.
2022-06-23 16:20:07,129 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.

16

Please set batch_size_per_gpu: 16

17

Are you telling me to re run the training?

18

No, just modify in the spec file and export again.

19

Oh! that fixed the issue.
Thank you!

20

Another solution:
When you run "classification export ", please add parameter “--batch_size” accordingly.
--batch_size your_batch_size_setting_in_spec

21

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.