FP4 Quantization fails on Jetson Thor with TensorRT 11.0.0.114

Description

I’m unable to compile a ModelOpt NVFP4-quantized model with Torch-TensorRT on a Jetson Thor. The same model compiles successfully with FP8 (mtq.FP8_DEFAULT_CFG) and with an FP4 weight-only configuration. The failure only occurs when using mtq.NVFP4_DEFAULT_CFG, which enables dynamic activation quantization.

TensorRT fails during engine building with:

ERROR:torch_tensorrt [TensorRT Conversion Context]:Error Code: 9: Skipping tactic 0x0000000000000000 due to exception [autotuner.cpp:3845: get_best_tactics] Autotuner: no tactics to implement operation:

 2012: fc: [ELEMENTWISE]-[aten_ops.linear.default]-[net.0/linear_add_bias]_output_linear.default.1-(f32[128,4096][]so[], mem_prop=0, align=4) | [DYNAMIC_QUANTIZE]-[aten_ops.dynamic_block_quantize_op.default]-[net.0.input_quantizer/dynamic_block_quantize_op_dynamic_quantize]_output.1-(fp4_e2m1[128,512][]so[], mem_prop=0, align=1), __mye2060_dconst-{-4, -6, 2, 1, 4, 0.5, -0.5, 2, ...}(fp4_e2m1[512,4096][1,512]so[0,1], mem_prop=0, align=1)<entry>, __mye2008-2.99895e-08F:(f32[][]so[], mem_prop=0, align=4)<entry>, __mye1884[MATRIX_MULTIPLY]-[aten_ops.linear.default]-[net.0/linear_matrix_multiply]_beta-0F:(f32[][]so[], mem_prop=0, align=4)<entry>, __mye1933_dconst_concat-{0.0267956, -0.0218608, -0.0203065, -0.0339911, -0.028881, -0.0388802, -0.0164647, -0.0386342, ...}(f32[1,4096][4096,1]so[1,0], mem_prop=0, align=4)<entry>, [DYNAMIC_QUANTIZE]-[aten_ops.dynamic_blo

ERROR:torch_tensorrt [TensorRT Conversion Context]:IBuilder::buildEngineWithConfig: Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[net.4.weight_quantizer/dynamic_block_quantize_op_5_weights_fp4...[ELEMENTWISE]-[aten_ops.linear.default]-[net.4/linear_2_add_bias]]}. In computeCosts at /_src/optimizer/common/tactic/optimizer.cpp:4284)

Traceback (most recent call last):

  File "/workspace/test_fp4_not_working.py", line 59, in <module>

    trt_model = torch_tensorrt.compile(

                ^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.12/dist-packages/torch_tensorrt/_compile.py", line 329, in compile

    trt_graph_module = dynamo_compile(

                       ^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.12/dist-packages/torch_tensorrt/dynamo/_compiler.py", line 772, in compile

    trt_gm = compile_module(

             ^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.12/dist-packages/torch_tensorrt/dynamo/_compiler.py", line 1118, in compile_module

    trt_module = convert_module(

                 ^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.12/dist-packages/torch_tensorrt/dynamo/conversion/_conversion.py", line 348, in convert_module

    serialized_interpreter_result = interpret_module_to_result(

                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.12/dist-packages/torch_tensorrt/dynamo/conversion/_conversion.py", line 281, in interpret_module_to_result

    interpreter_result = interpreter.run()

                         ^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.12/dist-packages/torch_tensorrt/dynamo/conversion/_TRTInterpreter.py", line 653, in run

    assert cuda_engine

           ^^^^^^^^^^^

The error consistently references dynamic_block_quantize_op inserted by the input quantizer. Since FP4 weight-only compilation succeeds, this appears to be related to the dynamic activation quantization path rather than FP4 weight quantization in general.

Environment

TensorRT Version: 11.0.0.114
GPU Type: Jetson Thor Blackwell GPU
NVIDIA ModelOpt Version: 0.44.0
Nvidia Driver Version: 595.78
CUDA Version: 13.3
CUDNN Version: 9.23.0
Operating System + Version: Ubuntu 24.04.4
Python Version (if applicable): 3.12.3
PyTorch Version (if applicable): 2.13.0a0+8145d630e8.nv26.6.54250401
Baremetal or Container (if container which image + tag): nvcr.io/nvidia/pytorch:26.06-py3

Relevant Files

import torch
import torch.nn as nn
import torch_tensorrt

import modelopt.torch.quantization as mtq
from modelopt.torch.quantization.utils import export_torch_mode

device = "cuda"

class SimpleMLP(nn.Module):
    def __init__(self):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(512, 4096),
            nn.GELU(),
            nn.Linear(4096,4096),
            nn.GELU(),
            nn.Linear(4096,512) 
        )

    def forward(self, x):
        return self.net(x)

model = SimpleMLP().to(device).eval()

quant_cfg = mtq.NVFP4_DEFAULT_CFG

calib_data = [
    (torch.randn(128, 512, device=device),)
    for _ in range(10)
]

def forward_loop(model):
    with torch.no_grad():
        for (x,) in calib_data:
            model(x)

model = mtq.quantize(
    model,
    quant_cfg,
    forward_loop,
)

# Verify the quantized model runs
x = torch.randn(128, 512, device=device)
with torch.no_grad():
    y = model(x)

print(y.shape)

inputs = [
    torch_tensorrt.Input(
        shape=(128, 512),
        dtype=torch.float32,
    )
]

with export_torch_mode():
    trt_model = torch_tensorrt.compile(
        model,
        ir='dynamo',
        inputs=inputs,
        min_block_size=1
    )

print("Compiled successfully!")

Steps To Reproduce

In order to reproduce, run the docker image nvcr.io/nvidia/pytorch:26.06-py3. Then, you should run the above script.

docker run --runtime=nvidia -it --rm \
  --shm-size=16g \
  -v "$PWD":/workspace \
  -w /workspace \
  nvcr.io/nvidia/pytorch:26.06-py3

python test_fp4_not_working.py

Please include:

  • Exact steps/commands to build your repro
  • Exact steps/commands to run your repro
  • Full traceback of errors encountered

Hello @shook18b!

Based on the title and content of your topic, it looks like it may receive better visibility and feedback in a different category. We took the liberty of moving it for you.

If this was an incorrect assessment, please send me a direct message.

Disclaimer: this moderation suggestion and message were generated with AI assistance.

Hi,

Thanks for reporting this issue.
Have you tried the same quantization on another device, like a desktop GPU?
If yes, does it work as expected?

Thanks.

Your model was fp32. TensorRT 11’s NVFP4 GEMM kernels (on Thor SM 11.0) only ship fp16/bf16 input/output tactics, so the autotuner found “no tactics” for the fp32 linear fused with the FP4 dequantize and the build aborted.

The torch-tensorrt repo NVFP4 test (tests/py/dynamo/models/test_models_export.py:297) uses torch.float16 for the same reason.

Change: cast the model, calibration data, and torch_tensorrt.Input dtype to torch.float16 before quantizing/compiling. Then it compiled.

I installed natively but docker should work:

curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv torchtrt -p 3.12.3 --seed
uv pip install torch torchvision --index-url https://download.pytorch.org/whl/cu132
uv pip install --pre torch-tensorrt --index-url https://download.pytorch.org/whl/nightly/cu132

cat test.py

#!/usr/bin/env python3

import torch
import torch.nn as nn
import torch_tensorrt

import modelopt.torch.quantization as mtq
from modelopt.torch.quantization.utils import export_torch_mode

device = "cuda"

# TRT 11 NVFP4 GEMM kernels only exist for fp16/bf16 in/out — fp32 fails with
# "Autotuner: no tactics"
dtype = torch.float16


class SimpleMLP(nn.Module):
    def __init__(self):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(512, 4096),
            nn.GELU(),
            nn.Linear(4096, 4096),
            nn.GELU(),
            nn.Linear(4096, 512),
        )

    def forward(self, x):
        return self.net(x)


model = SimpleMLP().to(device=device, dtype=dtype).eval()

quant_cfg = mtq.NVFP4_DEFAULT_CFG

calib_data = [
    (torch.randn(128, 512, device=device, dtype=dtype),)
    for _ in range(10)
]


def forward_loop(model):
    with torch.no_grad():
        for (x,) in calib_data:
            model(x)


model = mtq.quantize(
    model,
    quant_cfg,
    forward_loop,
)

# Verify the quantized model runs
x = torch.randn(128, 512, device=device, dtype=dtype)

with torch.no_grad():
    y = model(x)

print(y.shape)

inputs = [
    torch_tensorrt.Input(
        shape=(128, 512),
        dtype=dtype,
    )
]

with export_torch_mode():
    trt_model = torch_tensorrt.compile(
        model,
        ir="dynamo",
        inputs=inputs,
        min_block_size=1,
    )