We tried to integrate GPT-OSS-120B Eagle3 model which provided by NV on Hugging Face.
[Test condition]:
Inference runtime: TensorRT-LLM: 1.2.0rc1
LLM models: “openai/gpt-oss-120b” + “nvidia/gpt-oss-120b-Eagle3”
[TensorRT-LLM][WARNING] Attention workspace size is not enough, increase the size from 33559808 bytes to 33560320 bytes
[11/28/2025-10:40:51] [TRT-LLM] [W] Failed to parse harmony messages from tokens: %s Unexpected EOS while waiting for message header to complete
[11/28/2025-10:40:51] [TRT-LLM] [W] Failed to parse harmony output: %s. Raw output: %s Harmony parsing failed: Unexpected EOS while waiting for message header to complete <|channel|>!!!
[11/28/2025-10:40:51] [TRT-LLM] [W] ⚠️ Harmony parsing fell back to raw text decoding
Since NVIDIA’s developer site mentions that the GPT-OSS-12B “Eagle” model supports execution on the Blackwell architecture and delivers up to 3× performance improvement in GB200 testing,
check.txt (972 Bytes)
script.txt (1.4 KB)
log.txt (27.7 KB)
Please take a look at this and adapt to your scenario:
Run TensorRT on two DGX Spark servers
The image used is nvcr.io/nvidia/tensorrt-llm/release:1.2.0rc4
The GitHub case mentioned above is different from our original question.
We noticed a very similar issue on GitHub (https://github.com/NVIDIA/TensorRT-LLM/issues/8713 ), but that issue has not been resolved yet.
Anyone tried Eagle3 inference on GB10?
The issue got resolved on those 2 patches:
main ← jhaotingc:ampere_xqa_swa_1013
opened 06:45AM - 15 Oct 25 UTC
## Summary by CodeRabbit
- New Features
- Added sliding-window-aware atten… tion masking, with per-tile masking and early exits when masking is unnecessary.
- Performance
- Optimized mask computation and packing paths.
- Improved warmup accuracy by using the device memory clock rate.
- Updated memory prefetch to the newer CUDA API.
- Tests
- Updated warmup utilities to accept device clock rate and reflect it in metrics.
- Adjusted test harness and call sites to use the new warmup interface and prefetch flow.
### Before this PR
GPT-OSS Eagle3-one-model TP=2, H200/H100
```
[11/25/2025-03:48:07] [TRT-LLM] [I] lm-eval gsm8k exact_match,flexible-extract accuracy: 62.09
[11/25/2025-03:48:07] [TRT-LLM] [I] Hypothesis testing report:
===========================================================
= ACCURACY HYPOTHESIS TESTING
===========================================================
Alpha (Type I: False Positive): 0.050
Beta (Type II: False Negative): 0.200
Sigma (Standard deviation): 50.000
#Samples: 1319
Higher is better: True
Theta (Minimum detectable effect): 4.841
Reference accuracy: 90.300
Threshold: 87.097
===========================================================
Evaluated accuracy: 62.092
===========================================================
```
### After this PR
GPT-OSS Eagle3-one-model TP=2, H200/H100
```
[11/25/2025-19:39:07] [TRT-LLM] [I] lm-eval gsm8k results (scores normalized to range 0~100):
|Tasks|Version| Filter |n-shot| Metric | | Value | |Stderr|
|-----|------:|----------------|-----:|-----------|---|------:|---|-----:|
|gsm8k| 3|flexible-extract| 5|exact_match|↑ |88.8552|± |0.8668|
| | |strict-match | 5|exact_match|↑ |58.1501|± |1.3588|
[11/25/2025-19:39:07] [TRT-LLM] [I] lm-eval gsm8k exact_match,flexible-extract accuracy: 88.86
[11/25/2025-19:39:07] [TRT-LLM] [I] Hypothesis testing report:
===========================================================
= ACCURACY HYPOTHESIS TESTING
===========================================================
Alpha (Type I: False Positive): 0.050
Beta (Type II: False Negative): 0.200
Sigma (Standard deviation): 50.000
#Samples: 1319
Higher is better: True
Theta (Minimum detectable effect): 4.841
Reference accuracy: 90.300
Threshold: 87.097
===========================================================
Evaluated accuracy: 88.855
===========================================================
| Tasks |Version| Filter |n-shot| Metric | | Value | |Stderr|
|----------------------------|------:|------------|-----:|-----------|---|------:|---|-----:|
|gpqa_diamond_cot_zeroshot_aa| 1|strict-match| 0|exact_match|↑ |68.1818|± |3.3185|
```
GPT-OSS Eagle3-one-model TP=4, RTX PRO 6000
```
[11/25/2025-19:58:11] [TRT-LLM] [I] lm-eval gsm8k results (scores normalized to range 0~100):
|Tasks|Version| Filter |n-shot| Metric | | Value | |Stderr|
|-----|------:|----------------|-----:|-----------|---|------:|---|-----:|
|gsm8k| 3|flexible-extract| 5|exact_match|↑ |90.7506|± | 0.798|
| | |strict-match | 5|exact_match|↑ |63.3813|± | 1.327|
| Tasks |Version| Filter |n-shot| Metric | | Value | |Stderr|
|----------------------------|------:|------------|-----:|-----------|---|------:|---|-----:|
|gpqa_diamond_cot_zeroshot_aa| 1|strict-match| 0|exact_match|↑ |61.6162|± |3.4649|
```
Perf
<img width="1279" height="800" alt="image" src="https://github.com/user-attachments/assets/7f78e078-6543-4a20-8bfc-6f129a7fc43c" />
<img width="1279" height="800" alt="image" src="https://github.com/user-attachments/assets/b4f01676-fc9d-4a59-a66b-a54a150b5f5e" />
## Description
## Test Coverage
```
pytest -s integration/defs/accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_eagle3_2gpus[cutlass-one_model-overlap_scheduler] # Hopper
pytest -s integration/defs/accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_eagle3_4gpus[cutlass-one_model-overlap_scheduler] # RTX PRO 6000
```
## PR Checklist
Please review the following before submitting your PR:
- PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
- PR Follows [TRT-LLM CODING GUIDELINES](https://github.com/NVIDIA/TensorRT-LLM/blob/main/CODING_GUIDELINES.md) to the best of your knowledge.
- Test cases are provided for new code paths (see [test instructions](https://github.com/NVIDIA/TensorRT-LLM/tree/main/tests#1-how-does-the-ci-work))
- Any new dependencies have been scanned for license and vulnerabilities
- [CODEOWNERS](https://github.com/NVIDIA/TensorRT-LLM/blob/main/.github/CODEOWNERS) updated if ownership changes
- Documentation updated as needed
- The reviewers assigned automatically/manually are appropriate for the PR.
- [x] Please check this after reviewing the above items as appropriate for this PR.
## GitHub Bot Help
`/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...`
Provide a user friendly way for developers to interact with a Jenkins server.
Run `/bot [-h|--help]` to print this help message.
See details below for each supported subcommand.
<details>
`run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental)]`
Launch build/test pipelines. All previously running jobs will be killed.
`--reuse-test (optional)pipeline-id ` *(OPTIONAL)* : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.
`--disable-reuse-test ` *(OPTIONAL)* : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.
`--disable-fail-fast ` *(OPTIONAL)* : Disable fail fast on build/tests/infra failures.
`--skip-test ` *(OPTIONAL)* : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does **NOT** update GitHub check status.
`--stage-list "A10-PyTorch-1, xxx"` *(OPTIONAL)* : Only run the specified test stages. Examples: "A10-PyTorch-1, xxx". Note: Does **NOT** update GitHub check status.
`--gpu-type "A30, H100_PCIe"` *(OPTIONAL)* : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does **NOT** update GitHub check status.
`--test-backend "pytorch, cpp"` *(OPTIONAL)* : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does **NOT** update GitHub pipeline status.
`--only-multi-gpu-test ` *(OPTIONAL)* : Only run the multi-GPU tests. Note: Does **NOT** update GitHub check status.
`--disable-multi-gpu-test ` *(OPTIONAL)* : Disable the multi-GPU tests. Note: Does **NOT** update GitHub check status.
`--add-multi-gpu-test ` *(OPTIONAL)* : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.
`--post-merge ` *(OPTIONAL)* : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.
`--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx"` *(OPTIONAL)* : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx".
`--detailed-log ` *(OPTIONAL)* : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.
`--debug ` *(OPTIONAL)* : **Experimental feature**. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in the `stage-list` parameter to access the appropriate container environment. Note: Does **NOT** update GitHub check status.
For guidance on mapping tests to stage names, see `docs/source/reference/ci-overview.md`
and the `scripts/test_to_stage_mapping.py` helper.
### kill
`kill `
Kill all running builds associated with pull request.
### skip
`skip --comment COMMENT `
Skip testing for latest commit on pull request. `--comment "Reason for skipping build/test"` is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.
### reuse-pipeline
`reuse-pipeline `
Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.
</details>
main ← jhaotingc:fix_trtllm_spec_dec
opened 08:10PM - 08 Dec 25 UTC
## Summary by CodeRabbit
* **New Features**
* Added GPU-specific optimizat… ion detection to improve kernel performance and speculative decoding efficiency on compatible hardware configurations.
<sub>✏️ Tip: You can customize this high-level summary in your review settings.</sub>
## Description
Fix missing function in TrtllmAttentionWrapper introduced by #8383.
```
File "/scratch/TensorRT-LLM-dev-2/tensorrt_llm/_torch/modules/attention.py", line 450, in _attn_impl
attn_output = self.attn.forward(
^^^^^^^^^^^^^^^^^^
File "/scratch/TensorRT-LLM-dev-2/tensorrt_llm/_torch/attention_backend/trtllm.py", line 1611, in forward
output, output_sf = self.wrapper.run(
^^^^^^^^^^^^^^^^^
File "/scratch/TensorRT-LLM-dev-2/tensorrt_llm/_torch/attention_backend/trtllm.py", line 478, in run
if self.is_sm_version_trtllm_gen_kernel(sm=get_sm_version()):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'TrtllmAttentionWrapper' object has no attribute 'is_sm_version_trtllm_gen_kernel'
```
## Test Coverage
## PR Checklist
Please review the following before submitting your PR:
- PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
- PR Follows [TRT-LLM CODING GUIDELINES](https://github.com/NVIDIA/TensorRT-LLM/blob/main/CODING_GUIDELINES.md) to the best of your knowledge.
- Test cases are provided for new code paths (see [test instructions](https://github.com/NVIDIA/TensorRT-LLM/tree/main/tests#1-how-does-the-ci-work))
- Any new dependencies have been scanned for license and vulnerabilities
- [CODEOWNERS](https://github.com/NVIDIA/TensorRT-LLM/blob/main/.github/CODEOWNERS) updated if ownership changes
- Documentation updated as needed
- Update [tava architecture diagram](https://github.com/NVIDIA/TensorRT-LLM/blob/main/.github/tava_architecture_diagram.md) if there is a significant design change in PR.
- The reviewers assigned automatically/manually are appropriate for the PR.
- [x] Please check this after reviewing the above items as appropriate for this PR.
## GitHub Bot Help
`/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...`
Provide a user friendly way for developers to interact with a Jenkins server.
Run `/bot [-h|--help]` to print this help message.
See details below for each supported subcommand.
<details>
`run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental)]`
Launch build/test pipelines. All previously running jobs will be killed.
`--reuse-test (optional)pipeline-id ` *(OPTIONAL)* : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.
`--disable-reuse-test ` *(OPTIONAL)* : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.
`--disable-fail-fast ` *(OPTIONAL)* : Disable fail fast on build/tests/infra failures.
`--skip-test ` *(OPTIONAL)* : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does **NOT** update GitHub check status.
`--stage-list "A10-PyTorch-1, xxx"` *(OPTIONAL)* : Only run the specified test stages. Examples: "A10-PyTorch-1, xxx". Note: Does **NOT** update GitHub check status.
`--gpu-type "A30, H100_PCIe"` *(OPTIONAL)* : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does **NOT** update GitHub check status.
`--test-backend "pytorch, cpp"` *(OPTIONAL)* : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does **NOT** update GitHub pipeline status.
`--only-multi-gpu-test ` *(OPTIONAL)* : Only run the multi-GPU tests. Note: Does **NOT** update GitHub check status.
`--disable-multi-gpu-test ` *(OPTIONAL)* : Disable the multi-GPU tests. Note: Does **NOT** update GitHub check status.
`--add-multi-gpu-test ` *(OPTIONAL)* : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.
`--post-merge ` *(OPTIONAL)* : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.
`--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx"` *(OPTIONAL)* : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx".
`--detailed-log ` *(OPTIONAL)* : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.
`--debug ` *(OPTIONAL)* : **Experimental feature**. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in the `stage-list` parameter to access the appropriate container environment. Note: Does **NOT** update GitHub check status.
For guidance on mapping tests to stage names, see `docs/source/reference/ci-overview.md`
and the `scripts/test_to_stage_mapping.py` helper.
### kill
`kill `
Kill all running builds associated with pull request.
### skip
`skip --comment COMMENT `
Skip testing for latest commit on pull request. `--comment "Reason for skipping build/test"` is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.
### reuse-pipeline
`reuse-pipeline `
Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.
</details>
And they’re available since release Release v1.2.0rc6 · NVIDIA/TensorRT-LLM · GitHub
I intend to play with Eagle3 and gpt-oss-120b with vLLM on the spark at some time in the future.
My previous attempts showed some low acceptance rates which were made even lower after I quantized some layers.
Also, vLLM doesn’t have a tree-base speculative decoding - so it’s unable to take full advantage of Eagle3.
I have a note for myself to play with these: