Description
I’m unable to compile a ModelOpt NVFP4-quantized model with Torch-TensorRT on a Jetson Thor. The same model compiles successfully with FP8 (mtq.FP8_DEFAULT_CFG) and with an FP4 weight-only configuration. The failure only occurs when using mtq.NVFP4_DEFAULT_CFG, which enables dynamic activation quantization.
TensorRT fails during engine building with:
ERROR:torch_tensorrt [TensorRT Conversion Context]:Error Code: 9: Skipping tactic 0x0000000000000000 due to exception [autotuner.cpp:3845: get_best_tactics] Autotuner: no tactics to implement operation:
2012: fc: [ELEMENTWISE]-[aten_ops.linear.default]-[net.0/linear_add_bias]_output_linear.default.1-(f32[128,4096][]so[], mem_prop=0, align=4) | [DYNAMIC_QUANTIZE]-[aten_ops.dynamic_block_quantize_op.default]-[net.0.input_quantizer/dynamic_block_quantize_op_dynamic_quantize]_output.1-(fp4_e2m1[128,512][]so[], mem_prop=0, align=1), __mye2060_dconst-{-4, -6, 2, 1, 4, 0.5, -0.5, 2, ...}(fp4_e2m1[512,4096][1,512]so[0,1], mem_prop=0, align=1)<entry>, __mye2008-2.99895e-08F:(f32[][]so[], mem_prop=0, align=4)<entry>, __mye1884[MATRIX_MULTIPLY]-[aten_ops.linear.default]-[net.0/linear_matrix_multiply]_beta-0F:(f32[][]so[], mem_prop=0, align=4)<entry>, __mye1933_dconst_concat-{0.0267956, -0.0218608, -0.0203065, -0.0339911, -0.028881, -0.0388802, -0.0164647, -0.0386342, ...}(f32[1,4096][4096,1]so[1,0], mem_prop=0, align=4)<entry>, [DYNAMIC_QUANTIZE]-[aten_ops.dynamic_blo
ERROR:torch_tensorrt [TensorRT Conversion Context]:IBuilder::buildEngineWithConfig: Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[net.4.weight_quantizer/dynamic_block_quantize_op_5_weights_fp4...[ELEMENTWISE]-[aten_ops.linear.default]-[net.4/linear_2_add_bias]]}. In computeCosts at /_src/optimizer/common/tactic/optimizer.cpp:4284)
Traceback (most recent call last):
File "/workspace/test_fp4_not_working.py", line 59, in <module>
trt_model = torch_tensorrt.compile(
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch_tensorrt/_compile.py", line 329, in compile
trt_graph_module = dynamo_compile(
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch_tensorrt/dynamo/_compiler.py", line 772, in compile
trt_gm = compile_module(
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch_tensorrt/dynamo/_compiler.py", line 1118, in compile_module
trt_module = convert_module(
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch_tensorrt/dynamo/conversion/_conversion.py", line 348, in convert_module
serialized_interpreter_result = interpret_module_to_result(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch_tensorrt/dynamo/conversion/_conversion.py", line 281, in interpret_module_to_result
interpreter_result = interpreter.run()
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch_tensorrt/dynamo/conversion/_TRTInterpreter.py", line 653, in run
assert cuda_engine
^^^^^^^^^^^
The error consistently references dynamic_block_quantize_op inserted by the input quantizer. Since FP4 weight-only compilation succeeds, this appears to be related to the dynamic activation quantization path rather than FP4 weight quantization in general.
Environment
TensorRT Version: 11.0.0.114
GPU Type: Jetson Thor Blackwell GPU
NVIDIA ModelOpt Version: 0.44.0
Nvidia Driver Version: 595.78
CUDA Version: 13.3
CUDNN Version: 9.23.0
Operating System + Version: Ubuntu 24.04.4
Python Version (if applicable): 3.12.3
PyTorch Version (if applicable): 2.13.0a0+8145d630e8.nv26.6.54250401
Baremetal or Container (if container which image + tag): nvcr.io/nvidia/pytorch:26.06-py3
Relevant Files
import torch
import torch.nn as nn
import torch_tensorrt
import modelopt.torch.quantization as mtq
from modelopt.torch.quantization.utils import export_torch_mode
device = "cuda"
class SimpleMLP(nn.Module):
def __init__(self):
super().__init__()
self.net = nn.Sequential(
nn.Linear(512, 4096),
nn.GELU(),
nn.Linear(4096,4096),
nn.GELU(),
nn.Linear(4096,512)
)
def forward(self, x):
return self.net(x)
model = SimpleMLP().to(device).eval()
quant_cfg = mtq.NVFP4_DEFAULT_CFG
calib_data = [
(torch.randn(128, 512, device=device),)
for _ in range(10)
]
def forward_loop(model):
with torch.no_grad():
for (x,) in calib_data:
model(x)
model = mtq.quantize(
model,
quant_cfg,
forward_loop,
)
# Verify the quantized model runs
x = torch.randn(128, 512, device=device)
with torch.no_grad():
y = model(x)
print(y.shape)
inputs = [
torch_tensorrt.Input(
shape=(128, 512),
dtype=torch.float32,
)
]
with export_torch_mode():
trt_model = torch_tensorrt.compile(
model,
ir='dynamo',
inputs=inputs,
min_block_size=1
)
print("Compiled successfully!")
Steps To Reproduce
In order to reproduce, run the docker image nvcr.io/nvidia/pytorch:26.06-py3. Then, you should run the above script.
docker run --runtime=nvidia -it --rm \
--shm-size=16g \
-v "$PWD":/workspace \
-w /workspace \
nvcr.io/nvidia/pytorch:26.06-py3
python test_fp4_not_working.py
Please include:
- Exact steps/commands to build your repro
- Exact steps/commands to run your repro
- Full traceback of errors encountered