Qwen3.6-27B-FP8 export failed in fuse_gdn_input_projections: Float8_e4m3fn and Half torch.cat promotion unsupported

Robotics & Edge Computing Jetson Systems Jetson Thor
1

Description

Qwen3.6-27B-FP8 export failed in fuse_gdn_input_projections: Float8_e4m3fn and Half torch.cat promotion unsupported

Environment

hardware: jetson thor
torch: 2.10.0+cu130

transformers: 5.3.0

cuda: 13.0

tensorrt-edge-llm: 0.7.1

model: Qwen/Qwen3.6-27B-FP8 · Hugging Face

Command

export MODEL_NAME=Qwen/Qwen3.6-27B-FP8

export WORKSPACE_DIR=$HOME/Projects/export_qwen3

mkdir -p $WORKSPACE_DIR

cd $WORKSPACE_DIR

python -m llm_loader.export_all_cli \

$MODEL_NAME \

$MODEL_NAME/onnx

Log

16:21:17 INFO llm_loader.export_all_cli: ============================================================ [00:00<?, ?it/s]
16:21:17 INFO llm_loader.export_all_cli: Model type : qwen3_5
16:21:17 INFO llm_loader.export_all_cli: Checkpoint : /home/wsc-noble/.cache/huggingface/hub/models–Qwen–Qwen3.6-27B-FP8/snapshots/e89b16ebf1988b3d6befa7de50abc2d76f26eb09
16:21:17 INFO llm_loader.export_all_cli: Output dir : Qwen/Qwen3.6-27B-FP8/onnx
16:21:17 INFO llm_loader.export_all_cli: thinker : yes
16:21:17 INFO llm_loader.export_all_cli: mtp_draft : no
16:21:17 INFO llm_loader.export_all_cli: talker : no
16:21:17 INFO llm_loader.export_all_cli: code_predictor : no
16:21:17 INFO llm_loader.export_all_cli: visual : yes
16:21:17 INFO llm_loader.export_all_cli: visual : no
16:21:17 INFO llm_loader.export_all_cli: audio : no
16:21:17 INFO llm_loader.export_all_cli: code2wav : no
16:21:17 INFO llm_loader.export_all_cli: action : no
16:21:17 INFO llm_loader.export_all_cli: FP8 embedding : no
16:21:17 INFO llm_loader.export_all_cli: MTP capable : yes
16:21:17 INFO llm_loader.export_all_cli: MTP export : no
16:21:17 INFO llm_loader.export_all_cli: Reduced vocab : no
16:21:17 INFO llm_loader.export_all_cli: ============================================================
16:21:17 INFO llm_loader.export_all_cli: [LLM] Loading checkpoint from /home/wsc-noble/.cache/huggingface/hub/models–Qwen–Qwen3.6-27B-FP8/snapshots/e89b16ebf1988b3d6befa7de50abc2d76f26eb09
Download complete: : 0.00B [00:00, ?B/s]
16:21:30 INFO llm_loader.checkpoint.loader: Stripping key prefix ‘model.language_model.’ from checkpoint keys (inserting ‘model.’)
16:21:30 INFO llm_loader.checkpoint.loader: Loaded 851 tensors, skipped 755 from /home/wsc-noble/.cache/huggingface/hub/models–Qwen–Qwen3.6-27B-FP8/snapshots/e89b16ebf1988b3d6befa7de50abc2d76f26eb09
16:21:30 INFO llm_loader.checkpoint.repacking: Repacked GPTQ weights
16:21:30 ERROR llm_loader.export_all_cli: [LLM] Failed to load checkpoint
Traceback (most recent call last):
File “/home/wsc-noble/Libraries/tensorrt-edge-llm/experimental/llm_loader/export_all_cli.py”, line 492, in _export_llm
model = AutoModel.from_pretrained(
File “/home/wsc-noble/Libraries/tensorrt-edge-llm/experimental/llm_loader/model.py”, line 167, in from_pretrained
fuse_gdn_input_projections(model)
File “/home/wsc-noble/Libraries/tensorrt-edge-llm/experimental/llm_loader/models/qwen3_5/modeling_qwen3_5_text.py”, line 720, in fuse_gdn_input_projections
fused_buffers[attr] = torch.cat(parts, dim=0)
RuntimeError: Promotion for Float8 Types is not supported, attempted to promote Float8_e4m3fn and Half

3

Hi @wsc921228, thanks for the full traceback and environment.

This is a TensorRT-Edge-LLM issue rather than a core TensorRT one - the failing call is in llm_loader.export_all_cli → fuse_gdn_input_projections inside the tensorrt-edge-llm package, which is the Thor-targeted C++ LLM/VLM stack and not part of libnvinfer. The Edge-LLM and Jetson Thor teams co-own that export tooling and the GDN fusion pass that’s tripping over the Float8_e4m3fn + Half torch.cat promotion.

I’m moving the thread over to the Jetson Thor category, which is where the Edge-LLM team and the Jetson software team both watch:

A quick technical note while the move happens: the torch.cat promotion error means the GDN input-projection fusion is trying to concatenate weights whose original dtypes were FP8 (the quantized o_proj/gate_proj style projections the GDN fuser is gathering) with weights still in FP16. Two ways that usually shows up:

  1. The Qwen3.6 FP8 checkpoint on Hugging Face has a mix of FP8-quantized layers and FP16-residual layers, and the fuser code path doesn’t have a dtype-bridge for the mixed case yet.

  2. The export tooling’s quantization metadata expects either an all-FP8 or all-FP16 input; partial mixing wasn’t on its tested matrix.

A pragmatic short-term workaround if you want to keep moving on Thor while the export tooling lands a fix: pull the FP16 (non-FP8) Qwen3.6-27B checkpoint and run the export with that, then re-quantize to FP8 with the Edge-LLM post-training quantization step. Slower, but it dodges the GDN fuser case until it’s patched.

Thanks, Atharva