Data_type problem in generating engine file for DINO model

Following the notebook “tao-getting-started_v5.3.0/notebooks/tao_launcher_starter_kit/dino/dino.ipynb”, I want to generate the engine file in different data_type, and I found following problems:

  1. use “fp16” instead of “FP16” (https://docs.nvidia.com/tao/tao-toolkit/text/tao_deploy/dino.html), otherwise the generated engine is actually still same size as the default fp32 version.
  2. cannot generate int8 version engine, and report following error (same for using gen_trt_engine.tensorrt.data_type=int8 or change the data_type in the spec yaml file):
    2024-07-11 14:50:57,394 [TAO Toolkit] [INFO] root 160: Registry: [‘nvcr.io’]
    2024-07-11 14:50:57,477 [TAO Toolkit] [INFO] nvidia_tao_cli.components.instance_handler.local_instance 360: Running command in container: nvcr.io/nvidia/tao/tao-toolkit:5.3.0-deploy
    2024-07-11 14:50:57,524 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 301: Printing tty value True
    sys:1: UserWarning:
    ‘gen_trt_engine_int8.yaml’ is validated against ConfigStore schema with the same name.
    This behavior is deprecated in Hydra 1.1 and will be removed in Hydra 1.2.
    See https://hydra.cc/docs/next/upgrades/1.0_to_1.1/automatic_schema_matching for migration instructions.
    :-1: UserWarning:
    ‘gen_trt_engine_int8.yaml’ is validated against ConfigStore schema with the same name.
    This behavior is deprecated in Hydra 1.1 and will be removed in Hydra 1.2.
    See https://hydra.cc/docs/next/upgrades/1.0_to_1.1/automatic_schema_matching for migration instructions.
    /usr/local/lib/python3.10/dist-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
    See https://hydra.cc/docs/next/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information.
    ret = run_job(
    Log file already exists at /results/status.json
    Starting dino gen_trt_engine.
    [07/11/2024-06:50:59] [TRT] [I] [MemUsageChange] Init CUDA: CPU +2, GPU +0, now: CPU 32, GPU 11679 (MiB)
    [07/11/2024-06:51:04] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU +1444, GPU +260, now: CPU 1553, GPU 11930 (MiB)
    Parsing ONNX model
    [07/11/2024-06:51:04] [TRT] [W] The NetworkDefinitionCreationFlag::kEXPLICIT_PRECISION flag has been deprecated and has no effect. Please do not use this flag when creating the network.
    [07/11/2024-06:51:04] [TRT] [W] parsers/onnx/onnx2trt_utils.cpp:372: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
    [07/11/2024-06:51:04] [TRT] [W] parsers/onnx/onnx2trt_utils.cpp:400: One or more weights outside the range of INT32 was clamped
    [07/11/2024-06:51:04] [TRT] [W] parsers/onnx/onnx2trt_utils.cpp:511: Your ONNX model has been generated with double-typed weights, while TensorRT does not natively support double. Attempting to cast down to float.
    [07/11/2024-06:51:05] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as plugin.
    [07/11/2024-06:51:05] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespace:
    [07/11/2024-06:51:05] [TRT] [I] Successfully created plugin: MultiscaleDeformableAttnPlugin_TRT
    [07/11/2024-06:51:05] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as plugin.
    [07/11/2024-06:51:05] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespace:
    [07/11/2024-06:51:05] [TRT] [I] Successfully created plugin: MultiscaleDeformableAttnPlugin_TRT
    [07/11/2024-06:51:05] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as plugin.
    [07/11/2024-06:51:05] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespace:
    [07/11/2024-06:51:05] [TRT] [I] Successfully created plugin: MultiscaleDeformableAttnPlugin_TRT
    [07/11/2024-06:51:05] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as plugin.
    [07/11/2024-06:51:05] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespace:
    [07/11/2024-06:51:05] [TRT] [I] Successfully created plugin: MultiscaleDeformableAttnPlugin_TRT
    [07/11/2024-06:51:05] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as plugin.
    [07/11/2024-06:51:05] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespace:
    [07/11/2024-06:51:05] [TRT] [I] Successfully created plugin: MultiscaleDeformableAttnPlugin_TRT
    [07/11/2024-06:51:05] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as plugin.
    [07/11/2024-06:51:05] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespace:
    [07/11/2024-06:51:05] [TRT] [I] Successfully created plugin: MultiscaleDeformableAttnPlugin_TRT
    [07/11/2024-06:51:05] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as plugin.
    [07/11/2024-06:51:05] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespace:
    [07/11/2024-06:51:05] [TRT] [I] Successfully created plugin: MultiscaleDeformableAttnPlugin_TRT
    [07/11/2024-06:51:05] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as plugin.
    [07/11/2024-06:51:05] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespace:
    [07/11/2024-06:51:05] [TRT] [I] Successfully created plugin: MultiscaleDeformableAttnPlugin_TRT
    [07/11/2024-06:51:05] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as plugin.
    [07/11/2024-06:51:05] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespace:
    [07/11/2024-06:51:05] [TRT] [I] Successfully created plugin: MultiscaleDeformableAttnPlugin_TRT
    [07/11/2024-06:51:05] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as plugin.
    [07/11/2024-06:51:05] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespace:
    [07/11/2024-06:51:05] [TRT] [I] Successfully created plugin: MultiscaleDeformableAttnPlugin_TRT
    [07/11/2024-06:51:05] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as plugin.
    [07/11/2024-06:51:05] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespace:
    [07/11/2024-06:51:05] [TRT] [I] Successfully created plugin: MultiscaleDeformableAttnPlugin_TRT
    [07/11/2024-06:51:05] [TRT] [I] No importer registered for op: MultiscaleDeformableAttnPlugin_TRT. Attempting to import as plugin.
    [07/11/2024-06:51:05] [TRT] [I] Searching for plugin: MultiscaleDeformableAttnPlugin_TRT, plugin_version: 1, plugin_namespace:
    [07/11/2024-06:51:05] [TRT] [I] Successfully created plugin: MultiscaleDeformableAttnPlugin_TRT
    Network Description
    Input ‘inputs’ with shape (-1, 3, 544, 960) and dtype DataType.FLOAT
    Output ‘pred_logits’ with shape (-1, 300, 91) and dtype DataType.FLOAT
    Output ‘pred_boxes’ with shape (-1, 300, 4) and dtype DataType.FLOAT
    dynamic batch size handling
    Enabling INT8 builder
    Calibrating using ImageBatcher
    stat: path should be string, bytes, os.PathLike or integer, not NoneType
    Error executing job with overrides: [‘gen_trt_engine.onnx_file=/results/export/dino_model.onnx’, ‘gen_trt_engine.trt_engine=/results/gen_trt_engine/dino_model_int8.engine’, ‘results_dir=/results’]
    Traceback (most recent call last):
    File “/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py”, line 213, in run_and_report
    return func()
    File “/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py”, line 453, in
    lambda: hydra.run(
    File “/usr/local/lib/python3.10/dist-packages/hydra/_internal/hydra.py”, line 132, in run
    _ = ret.return_value
    File “/usr/local/lib/python3.10/dist-packages/hydra/core/utils.py”, line 260, in return_value
    raise self._return_value
    File “/usr/local/lib/python3.10/dist-packages/hydra/core/utils.py”, line 186, in run_job
    ret.return_value = task_function(task_cfg)
    File “”, line 63, in _func
    File “”, line 47, in _func
    File “”, line 104, in main
    File “”, line 165, in create_engine
    File “”, line 203, in set_calibrator
    File “/usr/lib/python3.10/genericpath.py”, line 19, in exists
    os.stat(path)
    TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “</usr/local/lib/python3.10/dist-packages/nvidia_tao_deploy/cv/dino/scripts/gen_trt_engine.py>”, line 3, in
File “”, line 117, in
File “”, line -1, in wrapper
File “/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py”, line 389, in _run_hydra
_run_app(
File “/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py”, line 452, in _run_app
run_and_report(
File “/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py”, line 215, in run_and_report
if _is_env_set(“HYDRA_FULL_ERROR”) or is_under_debugger():
File “/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py”, line 165, in is_under_debugger
frames = inspect.stack()
File “/usr/lib/python3.10/inspect.py”, line 1673, in stack
return getouterframes(sys._getframe(1), context)
File “/usr/lib/python3.10/inspect.py”, line 1650, in getouterframes
frameinfo = (frame,) + getframeinfo(frame, context)
File “/usr/lib/python3.10/inspect.py”, line 1622, in getframeinfo
start = lineno - 1 - context//2
TypeError: unsupported operand type(s) for -: ‘NoneType’ and ‘int’
2024-07-11 06:51:05,530 [TAO Toolkit] [INFO] nvidia_tao_deploy.cv.common.entrypoint.entrypoint_hydra: Sending telemetry data.
2024-07-11 06:51:09,484 [TAO Toolkit] [INFO] nvidia_tao_deploy.cv.common.entrypoint.entrypoint_hydra: Execution status: FAIL
2024-07-11 14:51:09,622 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 363: Stopping container.

Yes, you can set to fp32/fp16. Refer to DINO with TAO Deploy - NVIDIA Docs. We will fix it as well in the example.

It is needed to set cal_image_dir. Refer to DINO with TAO Deploy - NVIDIA Docs.

I generate a gen_trt_engine_int8.yaml for int8 calibration, but still failed, what’s the prblem? Thank you very much for the support!

Spec file:
gen_trt_engine:
gpu_id: 0
input_width: 960
input_height: 544
tensorrt:
data_type: int8
workspace_size: 4096
min_batch_size: 1
opt_batch_size: 8
max_batch_size: 8
calibration:
cal_image_dir: ‘/data/raw-data/test2017’
cal_cache_file: ‘/results/export/cal.bin’
cal_batch_size: 2
cal_batches: 10
dataset:
num_classes: 91
batch_size: -1
model:
backbone: fan_small

Error Report:
2024-07-11 17:20:26,798 [TAO Toolkit] [INFO] root 160: Registry: [‘nvcr.io’]
2024-07-11 17:20:26,891 [TAO Toolkit] [INFO] nvidia_tao_cli.components.instance_handler.local_instance 360: Running command in container: nvcr.io/nvidia/tao/tao-toolkit:5.3.0-deploy
2024-07-11 17:20:26,940 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 301: Printing tty value True
sys:1: UserWarning:
‘gen_trt_engine_int8.yaml’ is validated against ConfigStore schema with the same name.
This behavior is deprecated in Hydra 1.1 and will be removed in Hydra 1.2.
See https://hydra.cc/docs/next/upgrades/1.0_to_1.1/automatic_schema_matching for migration instructions.
:-1: UserWarning:
‘gen_trt_engine_int8.yaml’ is validated against ConfigStore schema with the same name.
This behavior is deprecated in Hydra 1.1 and will be removed in Hydra 1.2.
See https://hydra.cc/docs/next/upgrades/1.0_to_1.1/automatic_schema_matching for migration instructions.
Traceback (most recent call last):
File “/usr/local/lib/python3.10/dist-packages/hydra/_internal/config_loader_impl.py”, line 457, in _load_single_config
merged = OmegaConf.merge(schema.config, config)
File “/usr/local/lib/python3.10/dist-packages/omegaconf/omegaconf.py”, line 268, in merge
target.merge_with(*configs[1:])
File “/usr/local/lib/python3.10/dist-packages/omegaconf/basecontainer.py”, line 490, in merge_with
self._format_and_raise(key=None, value=None, cause=e)
File “/usr/local/lib/python3.10/dist-packages/omegaconf/base.py”, line 231, in _format_and_raise
format_and_raise(
File “/usr/local/lib/python3.10/dist-packages/omegaconf/_utils.py”, line 793, in format_and_raise
_raise(ex, cause)
File “/usr/local/lib/python3.10/dist-packages/omegaconf/_utils.py”, line 771, in _raise
raise ex.with_traceback(sys.exc_info()[2]) # set env var OC_CAUSE=1 for full trace
File “/usr/local/lib/python3.10/dist-packages/omegaconf/basecontainer.py”, line 488, in merge_with
self._merge_with(*others)
File “/usr/local/lib/python3.10/dist-packages/omegaconf/basecontainer.py”, line 512, in _merge_with
BaseContainer._map_merge(self, other)
File “/usr/local/lib/python3.10/dist-packages/omegaconf/basecontainer.py”, line 397, in _map_merge
dest_node._merge_with(src_node)
File “/usr/local/lib/python3.10/dist-packages/omegaconf/basecontainer.py”, line 512, in _merge_with
BaseContainer._map_merge(self, other)
File “/usr/local/lib/python3.10/dist-packages/omegaconf/basecontainer.py”, line 397, in _map_merge
dest_node._merge_with(src_node)
File “/usr/local/lib/python3.10/dist-packages/omegaconf/basecontainer.py”, line 512, in _merge_with
BaseContainer._map_merge(self, other)
File “/usr/local/lib/python3.10/dist-packages/omegaconf/basecontainer.py”, line 397, in _map_merge
dest_node._merge_with(src_node)
File “/usr/local/lib/python3.10/dist-packages/omegaconf/basecontainer.py”, line 512, in _merge_with
BaseContainer._map_merge(self, other)
File “/usr/local/lib/python3.10/dist-packages/omegaconf/basecontainer.py”, line 399, in _map_merge
dest.setitem(key, src_node)
File “/usr/local/lib/python3.10/dist-packages/omegaconf/dictconfig.py”, line 316, in setitem
self._format_and_raise(key=key, value=value, cause=e)
File “/usr/local/lib/python3.10/dist-packages/omegaconf/base.py”, line 231, in _format_and_raise
format_and_raise(
File “/usr/local/lib/python3.10/dist-packages/omegaconf/_utils.py”, line 873, in format_and_raise
_raise(ex, cause)
File “/usr/local/lib/python3.10/dist-packages/omegaconf/_utils.py”, line 771, in _raise
raise ex.with_traceback(sys.exc_info()[2]) # set env var OC_CAUSE=1 for full trace
File “/usr/local/lib/python3.10/dist-packages/omegaconf/dictconfig.py”, line 310, in setitem
self.__set_impl(key=key, value=value)
File “/usr/local/lib/python3.10/dist-packages/omegaconf/dictconfig.py”, line 320, in __set_impl
self._set_item_impl(key, value)
File “/usr/local/lib/python3.10/dist-packages/omegaconf/basecontainer.py”, line 602, in _set_item_impl
self.dict[“_content”][key]._set_value(value)
File “/usr/local/lib/python3.10/dist-packages/omegaconf/listconfig.py”, line 612, in _set_value
raise e
File “/usr/local/lib/python3.10/dist-packages/omegaconf/listconfig.py”, line 608, in _set_value
self._set_value_impl(value, flags)
File “/usr/local/lib/python3.10/dist-packages/omegaconf/listconfig.py”, line 640, in _set_value_impl
raise ValidationError(msg)
omegaconf.errors.ValidationError: Invalid value assigned: AnyNode is not a ListConfig, list or tuple.
full_key: gen_trt_engine.tensorrt.calibration.cal_image_dir
reference_type=CalibrationConfig
object_type=CalibrationConfig

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py”, line 213, in run_and_report
return func()
File “/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py”, line 453, in
lambda: hydra.run(
File “/usr/local/lib/python3.10/dist-packages/hydra/_internal/hydra.py”, line 105, in run
cfg = self.compose_config(
File “/usr/local/lib/python3.10/dist-packages/hydra/_internal/hydra.py”, line 594, in compose_config
cfg = self.config_loader.load_configuration(
File “/usr/local/lib/python3.10/dist-packages/hydra/_internal/config_loader_impl.py”, line 141, in load_configuration
return self._load_configuration_impl(
File “/usr/local/lib/python3.10/dist-packages/hydra/_internal/config_loader_impl.py”, line 252, in _load_configuration_impl
cfg = self._compose_config_from_defaults_list(
File “/usr/local/lib/python3.10/dist-packages/hydra/_internal/config_loader_impl.py”, line 518, in _compose_config_from_defaults_list
loaded = self._load_single_config(default=default, repo=repo)
File “/usr/local/lib/python3.10/dist-packages/hydra/_internal/config_loader_impl.py”, line 465, in _load_single_config
raise ConfigCompositionException(
hydra.errors.ConfigCompositionException: Error merging ‘gen_trt_engine_int8.yaml’ with schema

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “</usr/local/lib/python3.10/dist-packages/nvidia_tao_deploy/cv/dino/scripts/gen_trt_engine.py>”, line 3, in
File “”, line 117, in
File “”, line -1, in wrapper
File “/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py”, line 389, in _run_hydra
_run_app(
File “/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py”, line 452, in _run_app
run_and_report(
File “/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py”, line 215, in run_and_report
if _is_env_set(“HYDRA_FULL_ERROR”) or is_under_debugger():
File “/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py”, line 165, in is_under_debugger
frames = inspect.stack()
File “/usr/lib/python3.10/inspect.py”, line 1673, in stack
return getouterframes(sys._getframe(1), context)
File “/usr/lib/python3.10/inspect.py”, line 1650, in getouterframes
frameinfo = (frame,) + getframeinfo(frame, context)
File “/usr/lib/python3.10/inspect.py”, line 1622, in getframeinfo
start = lineno - 1 - context//2
TypeError: unsupported operand type(s) for -: ‘NoneType’ and ‘int’
2024-07-11 09:20:28,825 [TAO Toolkit] [INFO] nvidia_tao_deploy.cv.common.entrypoint.entrypoint_hydra: Sending telemetry data.
2024-07-11 09:20:48,339 [TAO Toolkit] [INFO] nvidia_tao_deploy.cv.common.entrypoint.entrypoint_hydra: Execution status: FAIL
2024-07-11 17:20:48,481 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 363: Stopping container.

Please refer to the format in DINO with TAO Deploy - NVIDIA Docs.

    calibration:
      cal_image_dir:
        - /path/to/cal/images

Thank you! I can generate the int8 engine now. But the generated engine file size makes me concern. The file size for int8 seems too big compared to fp16 and fp32.

File size:
fp32: 240M
fp16: 128M
int8: 120M

Actually int8 is not supported for DINO yet. Although the engine is generated, it is not int8 mode.

Thank you very much