Description
Hi, I’m using the MMDetection framework from OpenMMLab. https://github.com/open-mmlab/mmdeploy
I used it with a set of models and without any conversion issue. However I’m having conversion issues with one template in particular (Mask2Former).
Following the MMDeploy model conversion tutorial: https://mmdeploy.readthedocs.io/en/latest/02-how-to-run/convert_model.html, when I use /configs/mmdet/panoptic-seg/panoptic-seg_maskformer_tensorrt_dynamic-320x512-1344x1344.py as model cfg, the process hang up with some errors described in this issue: https://github.com/open-mmlab/mmdeploy/issues/2632
However I found a “workaround” to succesfully convert my model from pth to trt.
Changing the max_workspace_size inside model_cfg python (panoptic-seg_maskformer_tensorrt_dynamic-320x512-1344x1344.py) from 1 << 30 to ( N-Gb * ( 2 ** 30 ) ) I’m able to convert my model without any issue. For my GPU I used 24 * ( 2 ** 30).
This works for FP32:
common_config=dict(fp16_mode=False, max_workspace_size= ( 4 * (2 ** 30 )) ),
and for INT8:
common_config=dict(fp16_mode=False, max_workspace_size= ( 4 * (2 ** 30 )) , int8_mode=True),
Instead when I try to use FP16
common_config=dict(fp16_mode=True, max_workspace_size= ( 4 * (2 ** 30 ))),
I got this (Also tryied to play with onnx OPSet version from 13 to 17) :
nvrtc_compile.cpp:940: CHECK(false) failed. NVRTC Compilation failure
[03/01/2024-10:51:53] [TRT] [E] 10: Could not find any implementation for node {ForeignNode[(Unnamed Layer* 14288) [Constant] + (Unnamed Layer* 14289) [Shuffle].../panoptic_head/Einsum_1]}.
[03/01/2024-10:51:54] [TRT] [E] 10: [optimizer.cpp::computeCosts::3869] Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[(Unnamed Layer* 14288) [Constant] + (Unnamed Layer* 14289) [Shuffle].../panoptic_head/Einsum_1]}.)
Process Process-3:
Full log provided into “Steps To Reproduce” section.
Environment
TensorRT Version: 8.6.1.6
GPU Type: NVidia RTX 3090 24GB
Nvidia Driver Version: 525.60.11
CUDA Version: 11.7
CUDNN Version: 8.6.0
Operating System + Version: Kubuntu 18.04
Python Version (if applicable): 3.8
TensorFlow Version (if applicable):
PyTorch Version (if applicable): 2.0.1
Baremetal or Container (if container which image + tag):
Miniconda: 3
MMDeploy: 1.3.1
ONNX: 1.15
ONNXRuntime-GPU: 1.12.0
Relevant Files
Please attach or include links to any models, data, files, or scripts necessary to reproduce your issue. (Github repo, Google Drive, Dropbox, etc.)
Steps To Reproduce
When I try to use FP16
common_config=dict(fp16_mode=True, max_workspace_size= ( 4 * (2 ** 30 ))),
I got this (Also tryied to play with onnx OPSet version from 13 to 17) :
03/01 10:44:41 - mmengine - WARNING - Failed to search registry with scope "mmdet" in the "Codebases" registry tree. As a workaround, the current "Codebases" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmdet" is a correct scope, or whether the registry is initialized.
03/01 10:44:41 - mmengine - WARNING - Failed to search registry with scope "mmdet" in the "mmdet_tasks" registry tree. As a workaround, the current "mmdet_tasks" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmdet" is a correct scope, or whether the registry is initialized.
03/01 10:44:42 - mmengine - INFO - Start pipeline mmdeploy.apis.pytorch2onnx.torch2onnx in subprocess
03/01 10:44:43 - mmengine - WARNING - Failed to search registry with scope "mmdet" in the "Codebases" registry tree. As a workaround, the current "Codebases" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmdet" is a correct scope, or whether the registry is initialized.
03/01 10:44:43 - mmengine - WARNING - Failed to search registry with scope "mmdet" in the "mmdet_tasks" registry tree. As a workaround, the current "mmdet_tasks" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmdet" is a correct scope, or whether the registry is initialized.
Loads checkpoint by local backend from path: /home/gallegretti/OpenMMLab/mask2former/iter_10000.pth
03/01 10:44:45 - mmengine - WARNING - DeprecationWarning: get_onnx_config will be deprecated in the future.
03/01 10:44:45 - mmengine - INFO - Export PyTorch model to ONNX: /home/gallegretti/OpenMMLab/msk2formerTRT/end2end.onnx.
03/01 10:44:46 - mmengine - WARNING - Can not find torch.nn.functional._scaled_dot_product_attention, function rewrite will not be applied
03/01 10:44:46 - mmengine - WARNING - Can not find mmdet.models.utils.transformer.PatchMerging.forward, function rewrite will not be applied
/home/gallegretti/miniconda3/envs/openmmlab/lib/python3.8/site-packages/mmdet/models/layers/transformer/utils.py:167: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
output_h = math.ceil(input_h / stride_h)
/home/gallegretti/miniconda3/envs/openmmlab/lib/python3.8/site-packages/mmdet/models/layers/transformer/utils.py:168: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
output_w = math.ceil(input_w / stride_w)
/home/gallegretti/miniconda3/envs/openmmlab/lib/python3.8/site-packages/mmdet/models/layers/transformer/utils.py:169: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
pad_h = max((output_h - 1) * stride_h +
/home/gallegretti/miniconda3/envs/openmmlab/lib/python3.8/site-packages/mmdet/models/layers/transformer/utils.py:171: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
pad_w = max((output_w - 1) * stride_w +
/home/gallegretti/miniconda3/envs/openmmlab/lib/python3.8/site-packages/mmdet/models/layers/transformer/utils.py:177: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if pad_h > 0 or pad_w > 0:
/home/gallegretti/OpenMMLab/mmdeploy/mmdeploy/codebase/mmdet/models/backbones.py:189: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
assert L == H * W, 'input feature has wrong size'
/home/gallegretti/OpenMMLab/mmdeploy/mmdeploy/codebase/mmdet/models/backbones.py:147: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
B = int(windows.shape[0] / (H * W / window_size / window_size))
/home/gallegretti/miniconda3/envs/openmmlab/lib/python3.8/site-packages/mmcv/cnn/bricks/wrappers.py:167: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if x.numel() == 0 and obsolete_torch_version(TORCH_VERSION, (1, 5)):
/home/gallegretti/miniconda3/envs/openmmlab/lib/python3.8/site-packages/mmdet/models/layers/transformer/utils.py:414: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
assert L == H * W, 'input feature has wrong size'
/home/gallegretti/miniconda3/envs/openmmlab/lib/python3.8/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1682343962757/work/aten/src/ATen/native/TensorShape.cpp:3483.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/home/gallegretti/miniconda3/envs/openmmlab/lib/python3.8/site-packages/mmcv/ops/multi_scale_deform_attn.py:335: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
assert (spatial_shapes[:, 0] * spatial_shapes[:, 1]).sum() == num_value
/home/gallegretti/miniconda3/envs/openmmlab/lib/python3.8/site-packages/mmcv/ops/multi_scale_deform_attn.py:351: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if reference_points.shape[-1] == 2:
/home/gallegretti/miniconda3/envs/openmmlab/lib/python3.8/site-packages/mmcv/ops/multi_scale_deform_attn.py:136: TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
value_list = value.split([H_ * W_ for H_, W_ in value_spatial_shapes],
/home/gallegretti/miniconda3/envs/openmmlab/lib/python3.8/site-packages/mmcv/ops/multi_scale_deform_attn.py:140: TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
for level, (H_, W_) in enumerate(value_spatial_shapes):
/home/gallegretti/miniconda3/envs/openmmlab/lib/python3.8/site-packages/mmcv/cnn/bricks/wrappers.py:44: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if x.numel() == 0 and obsolete_torch_version(TORCH_VERSION, (1, 4)):
[W shape_type_inference.cpp:1920] Warning: The shape inference of mmdeploy::grid_sampler type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
[W shape_type_inference.cpp:1920] Warning: The shape inference of mmdeploy::grid_sampler type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
[W shape_type_inference.cpp:1920] Warning: The shape inference of mmdeploy::grid_sampler type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
[W shape_type_inference.cpp:1920] Warning: The shape inference of mmdeploy::grid_sampler type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
[W shape_type_inference.cpp:1920] Warning: The shape inference of mmdeploy::grid_sampler type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
[W shape_type_inference.cpp:1920] Warning: The shape inference of mmdeploy::grid_sampler type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
[W shape_type_inference.cpp:1920] Warning: The shape inference of mmdeploy::grid_sampler type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
[W shape_type_inference.cpp:1920] Warning: The shape inference of mmdeploy::grid_sampler type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
[W shape_type_inference.cpp:1920] Warning: The shape inference of mmdeploy::grid_sampler type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
[W shape_type_inference.cpp:1920] Warning: The shape inference of mmdeploy::grid_sampler type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
[W shape_type_inference.cpp:1920] Warning: The shape inference of mmdeploy::grid_sampler type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
[W shape_type_inference.cpp:1920] Warning: The shape inference of mmdeploy::grid_sampler type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
[W shape_type_inference.cpp:1920] Warning: The shape inference of mmdeploy::grid_sampler type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
[W shape_type_inference.cpp:1920] Warning: The shape inference of mmdeploy::grid_sampler type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
[W shape_type_inference.cpp:1920] Warning: The shape inference of mmdeploy::grid_sampler type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
[W shape_type_inference.cpp:1920] Warning: The shape inference of mmdeploy::grid_sampler type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
[W shape_type_inference.cpp:1920] Warning: The shape inference of mmdeploy::grid_sampler type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
[W shape_type_inference.cpp:1920] Warning: The shape inference of mmdeploy::grid_sampler type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function. (function UpdateReliable)
03/01 10:45:34 - mmengine - INFO - Execute onnx optimize passes.
================ Diagnostic Run torch.onnx.export version 2.0.1 ================
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 18 WARNING 0 ERROR =======================
18 WARNING were not printed due to the log level.
03/01 10:45:37 - mmengine - INFO - Finish pipeline mmdeploy.apis.pytorch2onnx.torch2onnx
03/01 10:45:39 - mmengine - INFO - Start pipeline mmdeploy.apis.utils.utils.to_backend in subprocess
03/01 10:45:39 - mmengine - INFO - Successfully loaded tensorrt plugins from /home/xxxxxxxx/OpenMMLab/mmdeploy/mmdeploy/lib/libmmdeploy_tensorrt_ops.so
[03/01/2024-10:45:39] [TRT] [I] [MemUsageChange] Init CUDA: CPU +13, GPU +0, now: CPU 110, GPU 91 (MiB)
[03/01/2024-10:45:43] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU +1450, GPU +266, now: CPU 1636, GPU 357 (MiB)
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:604] Reading dangerously large protocol message. If the message turns out to be larger than 2147483647 bytes, parsing will be halted for security reasons. To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:81] The total number of bytes read was 916374538
[03/01/2024-10:45:43] [TRT] [I] ----------------------------------------------------------------
[03/01/2024-10:45:43] [TRT] [I] Input filename: /home/xxxxxx/OpenMMLab/msk2formerTRT/end2end.onnx
[03/01/2024-10:45:43] [TRT] [I] ONNX IR version: 0.0.8
[03/01/2024-10:45:43] [TRT] [I] Opset version: 17
[03/01/2024-10:45:43] [TRT] [I] Producer name: pytorch
[03/01/2024-10:45:43] [TRT] [I] Producer version: 2.0.1
[03/01/2024-10:45:43] [TRT] [I] Domain:
[03/01/2024-10:45:43] [TRT] [I] Model version: 0
[03/01/2024-10:45:43] [TRT] [I] Doc string:
[03/01/2024-10:45:43] [TRT] [I] ----------------------------------------------------------------
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:604] Reading dangerously large protocol message. If the message turns out to be larger than 2147483647 bytes, parsing will be halted for security reasons. To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:81] The total number of bytes read was 916374538
[03/01/2024-10:45:44] [TRT] [W] onnx2trt_utils.cpp:374: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[03/01/2024-10:45:44] [TRT] [W] onnx2trt_utils.cpp:400: One or more weights outside the range of INT32 was clamped
[03/01/2024-10:45:44] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[03/01/2024-10:45:44] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[03/01/2024-10:45:44] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[03/01/2024-10:45:44] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[03/01/2024-10:45:44] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[03/01/2024-10:45:44] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[03/01/2024-10:45:44] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[03/01/2024-10:45:44] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[03/01/2024-10:45:44] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[03/01/2024-10:45:44] [TRT] [I] No importer registered for op: grid_sampler. Attempting to import as plugin.
[03/01/2024-10:45:44] [TRT] [I] Searching for plugin: grid_sampler, plugin_version: 1, plugin_namespace:
[03/01/2024-10:45:44] [TRT] [I] Successfully created plugin: grid_sampler
[03/01/2024-10:45:44] [TRT] [I] No importer registered for op: grid_sampler. Attempting to import as plugin.
[03/01/2024-10:45:44] [TRT] [I] Searching for plugin: grid_sampler, plugin_version: 1, plugin_namespace:
[03/01/2024-10:45:44] [TRT] [I] Successfully created plugin: grid_sampler
[03/01/2024-10:45:44] [TRT] [I] No importer registered for op: grid_sampler. Attempting to import as plugin.
[03/01/2024-10:45:44] [TRT] [I] Searching for plugin: grid_sampler, plugin_version: 1, plugin_namespace:
[03/01/2024-10:45:44] [TRT] [I] Successfully created plugin: grid_sampler
[03/01/2024-10:45:44] [TRT] [I] No importer registered for op: grid_sampler. Attempting to import as plugin.
[03/01/2024-10:45:44] [TRT] [I] Searching for plugin: grid_sampler, plugin_version: 1, plugin_namespace:
[03/01/2024-10:45:44] [TRT] [I] Successfully created plugin: grid_sampler
[03/01/2024-10:45:44] [TRT] [I] No importer registered for op: grid_sampler. Attempting to import as plugin.
[03/01/2024-10:45:44] [TRT] [I] Searching for plugin: grid_sampler, plugin_version: 1, plugin_namespace:
[03/01/2024-10:45:44] [TRT] [I] Successfully created plugin: grid_sampler
[03/01/2024-10:45:44] [TRT] [I] No importer registered for op: grid_sampler. Attempting to import as plugin.
[03/01/2024-10:45:44] [TRT] [I] Searching for plugin: grid_sampler, plugin_version: 1, plugin_namespace:
[03/01/2024-10:45:44] [TRT] [I] Successfully created plugin: grid_sampler
[03/01/2024-10:45:44] [TRT] [I] No importer registered for op: grid_sampler. Attempting to import as plugin.
[03/01/2024-10:45:44] [TRT] [I] Searching for plugin: grid_sampler, plugin_version: 1, plugin_namespace:
[03/01/2024-10:45:44] [TRT] [I] Successfully created plugin: grid_sampler
[03/01/2024-10:45:44] [TRT] [I] No importer registered for op: grid_sampler. Attempting to import as plugin.
[03/01/2024-10:45:44] [TRT] [I] Searching for plugin: grid_sampler, plugin_version: 1, plugin_namespace:
[03/01/2024-10:45:44] [TRT] [I] Successfully created plugin: grid_sampler
[03/01/2024-10:45:44] [TRT] [I] No importer registered for op: grid_sampler. Attempting to import as plugin.
[03/01/2024-10:45:44] [TRT] [I] Searching for plugin: grid_sampler, plugin_version: 1, plugin_namespace:
[03/01/2024-10:45:44] [TRT] [I] Successfully created plugin: grid_sampler
[03/01/2024-10:45:44] [TRT] [I] No importer registered for op: grid_sampler. Attempting to import as plugin.
[03/01/2024-10:45:44] [TRT] [I] Searching for plugin: grid_sampler, plugin_version: 1, plugin_namespace:
[03/01/2024-10:45:44] [TRT] [I] Successfully created plugin: grid_sampler
[03/01/2024-10:45:44] [TRT] [I] No importer registered for op: grid_sampler. Attempting to import as plugin.
[03/01/2024-10:45:44] [TRT] [I] Searching for plugin: grid_sampler, plugin_version: 1, plugin_namespace:
[03/01/2024-10:45:44] [TRT] [I] Successfully created plugin: grid_sampler
[03/01/2024-10:45:44] [TRT] [I] No importer registered for op: grid_sampler. Attempting to import as plugin.
[03/01/2024-10:45:44] [TRT] [I] Searching for plugin: grid_sampler, plugin_version: 1, plugin_namespace:
[03/01/2024-10:45:44] [TRT] [I] Successfully created plugin: grid_sampler
[03/01/2024-10:45:44] [TRT] [I] No importer registered for op: grid_sampler. Attempting to import as plugin.
[03/01/2024-10:45:44] [TRT] [I] Searching for plugin: grid_sampler, plugin_version: 1, plugin_namespace:
[03/01/2024-10:45:44] [TRT] [I] Successfully created plugin: grid_sampler
[03/01/2024-10:45:44] [TRT] [I] No importer registered for op: grid_sampler. Attempting to import as plugin.
[03/01/2024-10:45:44] [TRT] [I] Searching for plugin: grid_sampler, plugin_version: 1, plugin_namespace:
[03/01/2024-10:45:44] [TRT] [I] Successfully created plugin: grid_sampler
[03/01/2024-10:45:44] [TRT] [I] No importer registered for op: grid_sampler. Attempting to import as plugin.
[03/01/2024-10:45:44] [TRT] [I] Searching for plugin: grid_sampler, plugin_version: 1, plugin_namespace:
[03/01/2024-10:45:44] [TRT] [I] Successfully created plugin: grid_sampler
[03/01/2024-10:45:44] [TRT] [I] No importer registered for op: grid_sampler. Attempting to import as plugin.
[03/01/2024-10:45:44] [TRT] [I] Searching for plugin: grid_sampler, plugin_version: 1, plugin_namespace:
[03/01/2024-10:45:44] [TRT] [I] Successfully created plugin: grid_sampler
[03/01/2024-10:45:44] [TRT] [I] No importer registered for op: grid_sampler. Attempting to import as plugin.
[03/01/2024-10:45:44] [TRT] [I] Searching for plugin: grid_sampler, plugin_version: 1, plugin_namespace:
[03/01/2024-10:45:44] [TRT] [I] Successfully created plugin: grid_sampler
[03/01/2024-10:45:44] [TRT] [I] No importer registered for op: grid_sampler. Attempting to import as plugin.
[03/01/2024-10:45:44] [TRT] [I] Searching for plugin: grid_sampler, plugin_version: 1, plugin_namespace:
[03/01/2024-10:45:44] [TRT] [I] Successfully created plugin: grid_sampler
[03/01/2024-10:45:44] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[03/01/2024-10:45:44] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[03/01/2024-10:45:44] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[03/01/2024-10:45:44] [TRT] [I] No importer registered for op: TRTInstanceNormalization. Attempting to import as plugin.
[03/01/2024-10:45:44] [TRT] [I] Searching for plugin: TRTInstanceNormalization, plugin_version: 1, plugin_namespace:
[03/01/2024-10:45:44] [TRT] [I] Successfully created plugin: TRTInstanceNormalization
[03/01/2024-10:45:44] [TRT] [I] Graph optimization time: 0.286143 seconds.
[03/01/2024-10:45:45] [TRT] [I] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +1012, GPU +10, now: CPU 3590, GPU 367 (MiB)
[03/01/2024-10:45:45] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +216, GPU +10, now: CPU 3806, GPU 377 (MiB)
[03/01/2024-10:45:45] [TRT] [W] TensorRT was linked against cuDNN 8.9.0 but loaded cuDNN 8.5.0
[03/01/2024-10:45:45] [TRT] [I] Local timing cache in use. Profiling results in this builder pass will not be stored.
nvrtc_compile.cpp:940: CHECK(false) failed. NVRTC Compilation failure
[03/01/2024-10:51:53] [TRT] [E] 10: Could not find any implementation for node {ForeignNode[(Unnamed Layer* 14288) [Constant] + (Unnamed Layer* 14289) [Shuffle].../panoptic_head/Einsum_1]}.
[03/01/2024-10:51:54] [TRT] [E] 10: [optimizer.cpp::computeCosts::3869] Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[(Unnamed Layer* 14288) [Constant] + (Unnamed Layer* 14289) [Shuffle].../panoptic_head/Einsum_1]}.)
Process Process-3: