TensorRT-Edge-LLM on Jetson AGX Thor: “No math backend found” although CUDA is installed

g.rado · June 30, 2026, 9:03pm

Title:
TensorRT-Edge-LLM High-Level API on Jetson AGX Thor fails: “No math backend found” although CUDA is installed

Body:
Hello NVIDIA team,

I am developing a robotics platform on Jetson AGX Thor with JetPack 7.2.

Our goal is to use TensorRT-Edge-LLM as the main local LLM runtime for a long-term robotics architecture.

Low-level status:

Jetson AGX Thor is running
JetPack 7.2 is installed
CUDA is installed and working
TensorRT is installed and working
TensorRT-Edge-LLM builds successfully
Engine generation works
The low-level llm_inference binary works

So the basic CUDA / TensorRT / low-level runtime path appears to be working.

However, we cannot use the High-Level Python API.

When trying to use the High-Level / Python layer, we run into the error:

“No math backend found”

This happens although CUDA is installed and available.

Because of this, we cannot reliably use the High-Level API / Python interface and cannot move to the intended production architecture.

Our desired architecture is:

Brain1
→ High-Level Python API
→ TensorRT-Edge-LLM
→ Brain2

At the moment we are forced to use:

Brain1
→ custom wrapper
→ llm_inference binary
→ Brain2

The wrapper approach works, but we would prefer to know whether the High-Level API is the officially recommended path for new Jetson Thor projects.

Questions:

Is the TensorRT-Edge-LLM High-Level Python API currently officially supported on Jetson AGX Thor with JetPack 7.2?
Is the “No math backend found” error a known issue on Jetson Thor?
Is there a missing backend, environment variable, build flag, or package required for the High-Level API on Thor?
Does NVIDIA currently recommend using the low-level llm_inference binary with a custom wrapper until the High-Level API is stable?
Is there a roadmap for production-ready High-Level Python API support on Jetson Thor?

We are not looking for a general installation guide.
We already have the low-level runtime working.

We mainly need architectural guidance so we can decide whether to continue with the wrapper architecture or invest more time into the High-Level Python API.

Thank you very much.

George Rado

whitesscott · July 1, 2026, 12:00am

What version; v0.8.0 is most recent release. Its release notes state

Server and API
    Expanded high-level Python API and server validation for LLM, VLM, and streaming flows

Following might help with the math backend part.

Are you using any of these existing python packages? The version numbers may be old in a few cases. This is on Jetpack 7.2 Thor in a venv where I’ve got a wide variety of packages. One of the main, I would think would be to search pypi.org for cu13 and get torch with cuda; pip install torch torchvision --index-url https://download.pytorch.org/whl/cu132. I’ve also attached a pip freeze of the packages that are installed in another venv for trtllmEdge; I modified its requirements.txt to install torch with Cuda.
trtllmEdge.txt (6.8 KB)

Here are the installable, or installed Cuda math libraries

pip list|grep math
mpmath                                   1.3.0
nvidia-libmathdx-cu13                    0.3.1.9
nvmath-python                            0.8.0

pip list|grep -i cu13
cutensor-cu13                            2.6.0
cvcuda-cu13                              0.16.0
holoscan-cu13                            4.3.0
nvidia-cudnn-cu13                        9.22.0.52
nvidia-cudss-cu13                        0.7.1.6
nvidia-cusparselt-cu13                   0.9.1
nvidia-libmathdx-cu13                    0.3.1.9
nvidia-nccl-cu13                         2.30.4
nvidia-nvimgcodec-cu13                   0.8.0.22
nvidia-nvimgcodec-tegra-cu13             0.6.0.32
nvidia-nvjpeg2k-tegra-cu13               0.9.0.43
nvidia-nvshmem-cu13                      3.6.5
torch                                    2.11.0+cu130
torchaudio                               2.11.0+cu130
torchvision                              0.26.0+cu130

pip list|grep -i nvidia
nvidia-cublas                            13.5.1.27
nvidia-cuda-cccl                         13.3.3.3.1
nvidia-cuda-crt                          13.3.33
nvidia-cuda-cupti                        13.3.35
nvidia-cuda-nvcc                         13.3.33
nvidia-cuda-nvrtc                        13.3.33
nvidia-cuda-runtime                      13.3.29
nvidia-cudnn-cu13                        9.22.0.52
nvidia-cudnn-frontend                    1.18.0
nvidia-cudss-cu13                        0.7.1.6
nvidia-cufft                             12.0.0.61
nvidia-cufile                            1.15.1.6
nvidia-curand                            10.4.0.35
nvidia-cusolver                          12.0.4.66
nvidia-cusparse                          12.6.3.3
nvidia-cusparselt-cu13                   0.9.1
nvidia-cutlass-dsl                       4.5.0
nvidia-cutlass-dsl-libs-base             4.5.0
nvidia-libmathdx-cu13                    0.3.1.9
nvidia-ml-py                             13.595.45
nvidia-nccl-cu13                         2.30.4
nvidia-nvfatbin                          13.2.78
nvidia-nvimgcodec-cu13                   0.8.0.22
nvidia-nvimgcodec-tegra-cu13             0.6.0.32
nvidia-nvjitlink                         13.0.88
nvidia-nvjpeg2k-tegra-cu13               0.9.0.43
nvidia-nvshmem-cu13                      3.6.5
nvidia-nvtx                              13.0.85
nvidia-nvvm                              13.3.33

AastaLLL · July 1, 2026, 3:23am

Hi,

Could you share more details about the high-level API you are using?
Is it TensorRT python API or TensorRT Edge LLM python API?

Thanks.

g.rado · July 1, 2026, 4:25am

Thank you very much for taking the time to help.

I already have these packages installed.

Here is the output from my environment:

pip list | grep math
mpmath 1.3.0
nvidia-libmathdx-cu13 0.3.1.9
nvmath-python 0.8.0

The CUDA packages are also installed:

torch 2.11.0+cu130
torchvision 0.26.0+cu130
nvidia-libmathdx-cu13
cutensor-cu13
cudnn-cu13
and the remaining CUDA 13 packages.

Despite this, the TensorRT-Edge-LLM high-level Python API still fails with:

“No math backend found”

The low-level runtime works correctly, CUDA is detected correctly, engine generation works, and inference works.

So at the moment it looks like the high-level Python API is unable to detect the installed math backend.

If you have any additional ideas, I would really appreciate them.

Thank you.

g.rado · July 1, 2026, 4:26am

Thank you.

We are using the TensorRT-Edge-LLM High-Level Python API, not the TensorRT Python API.

The low-level TensorRT-Edge-LLM runtime works correctly.

The issue only appears when using the High-Level Python API, which immediately reports:

“No math backend found”

CUDA, TensorRT and the low-level runtime all work correctly.

If needed, I can provide the complete Python example, stack trace and environment details.

g.rado · July 1, 2026, 4:39am

First: Thank you very much again!

To clarify:

My Jetson AGX Thor itself is running JetPack 7.2 with CUDA 13.2.

The important difference I noticed is not the system CUDA version, but the Python / PyTorch stack.

My current environment has:

torch==2.11.0+cu130
torchvision==0.26.0+cu130
torchaudio==2.11.0+cu130

while your working environment uses:

torch==2.12.1+cu132
torchvision==0.27.1+cu132

So my system CUDA is 13.2, but my current PyTorch wheel appears to be built for CUDA 13.0.

The low-level TensorRT-Edge-LLM path works correctly:

engine generation works
llm_inference works
CUDA itself works
TensorRT works

The failure only appears in the TensorRT-Edge-LLM High-Level Python API.

The error is:

“No math backend found”

What made us suspicious is that the relevant math packages appear to be installed:

mpmath==1.3.0
nvidia-libmathdx-cu13==0.3.1.9
nvmath-python==0.8.0

So it looks less like “math package not installed” and more like the High-Level Python stack is not resolving / loading the correct CUDA math backend.

Before I modify the whole environment:

Do you think the mismatch between JetPack/CUDA 13.2 and torch 2.11.0+cu130 could be the cause?

Or would you first recommend aligning my Python environment with your working setup, especially torch 2.12.1+cu132 and torchvision 0.27.1+cu132?

Thank you again for sharing your pip freeze. It is very helpful.

whitesscott · July 1, 2026, 5:20am

edit: first run pip freeze | tee requirements.txt so you can return to where you are now if needed.

You could then try this to see if it resolves problem.
pip install -U torch==2.12.0 torchvision==0.27.0 --index-url https://download.pytorch.org/whl/cu132

The https://github.com/NVIDIA/TensorRT-Edge-LLM/blob/main/requirements.txt for TensorRT Edge-LLM version 0.8.0 contains

torch==2.12.0
transformers==5.9.0
onnx==1.19.0
onnxscript==0.7.0
safetensors==0.7.0
numpy==2.4.6
onnx-graphsurgeon==0.6.1

g.rado · July 1, 2026, 7:37am

Hi,

I finally got the TensorRT-Edge-LLM Python High-Level Runtime working on Jetson Thor (JetPack 7.2).

For reference, my environment is:

Jetson Thor
JetPack 7.2
CUDA 13.2
TensorRT 10.16.2.10
TensorRT-Edge-LLM 0.8
Python 3.12

The initial problems were not related to TensorRT itself.

The Python environment required several additional packages that were not installed automatically in my setup:

nvmath-python
nvidia-modelopt
requests
pydantic
einops

After installing these packages, all TensorRT-Edge-LLM Python modules imported successfully.

The only remaining issue was building the experimental Python runtime (_edgellm_runtime). During the CMake build I consistently received:

fatal error: math.h: No such file or directory

I spent quite some time isolating the problem.

The following tests all succeeded:

/usr/include/math.h exists.
Standalone CUDA programs compile correctly.
CUDA + <cuda_runtime.h> + <cmath> compile correctly.
device_link_stub.cu compiles correctly when built manually with nvcc.

The failure only occurred during the CMake build of _edgellm_runtime.

After resolving that build issue, the runtime compiled successfully:

_edgellm_runtime.cpython-312-aarch64-linux-gnu.so

Verification:

runtime import OK
LLMRuntime OK

TEXT: Hello there!
IDS: 4

The High-Level Python Runtime is now working correctly on Jetson Thor.

I’m still investigating the exact root cause of the math.h build issue, but I wanted to share the successful result in case it helps others working with TensorRT-Edge-LLM on Thor.

Thanks!

whitesscott · July 2, 2026, 5:03am

I had encountered the same missing math.h error when I last built TensorRT-Edge-LLM on Thor. After your errors thought other people might get them too. Looked at it again and did 2 small pr as follows. This includes how I built TensorRT-Edge-LLM.

git clone https://github.com/NVIDIA/TensorRT-Edge-LLM.git
cd TensorRT-Edge-LLM

git submodule sync
git submodule update --init --recursive

# Install uv.
curl -fsSL https://astral.sh/uv/install.sh | sh

# Create fresh venv
uv venv trtedgellm -p 3.12.3 --seed
source trtedgellm/bin/activate

Edited

cpp/kernels/contextAttentionKernels/contextFMHARunner.cpp
-#include <math.h>      +#include <cmath>

cpp/kernels/contextAttentionKernels/fmhaParams_v2.h
-#include <math.h>      +#include <cmath>

Modified experimental/pybind/CMakeLists.txt

diff --git a/experimental/pybind/CMakeLists.txt b/experimental/pybind/CMakeLists.txt
index 855f8bc..daa94de 100644
--- a/experimental/pybind/CMakeLists.txt
+++ b/experimental/pybind/CMakeLists.txt
@@ -111,6 +111,31 @@ find_package(
   REQUIRED)
 find_package(pybind11 CONFIG REQUIRED)

+# ── Work around Debian/Ubuntu pybind11-dev exporting /usr/include ─────────
+# The apt-installed `pybind11-dev` package ships a CMake config whose
+# `pybind11::pybind11_headers` target sets
+#     INTERFACE_INCLUDE_DIRECTORIES = ${_IMPORT_PREFIX}/include
+# and `_IMPORT_PREFIX` resolves to `/usr`, so `/usr/include` gets propagated
+# to every consumer via `-isystem`. When nvcc then invokes host g++ on a CUDA
+# source, `-isystem /usr/include` reorders g++'s system header chain and
+# places `/usr/include` before `/usr/include/c++/<ver>`. libstdc++'s <cmath>
+# uses `#include_next <math.h>`, which now finds nothing after itself in the
+# search order, and compilation fails with:
+#   /usr/include/c++/13/cmath: fatal error: math.h: No such file or directory
+# Filter `/usr/include` out of the interface so the real pybind11 headers
+# (in a subdirectory under _IMPORT_PREFIX) are still exposed but the bogus
+# root include is not. No-op for pip-installed pybind11 (its _IMPORT_PREFIX
+# points at the venv's site-packages, not /usr).
+if(TARGET pybind11::pybind11_headers)
+  get_target_property(_pybind11_iface_inc pybind11::pybind11_headers
+                      INTERFACE_INCLUDE_DIRECTORIES)
+  if(_pybind11_iface_inc)
+    list(REMOVE_ITEM _pybind11_iface_inc "/usr/include")
+    set_property(TARGET pybind11::pybind11_headers PROPERTY
+                 INTERFACE_INCLUDE_DIRECTORIES "${_pybind11_iface_inc}")
+  endif()
+endif()
+
 # ── Header include paths (from source tree) ───────────────────────────────
 set(EDGELLM_INCLUDE_DIRS
     ${EDGELLM_ROOT}/cpp ${EDGELLM_ROOT}/examples/multimodal

Modified requirements.txt. No pr.

transformers==5.9.0
onnx==1.19.0
onnxscript==0.7.0
safetensors==0.7.0
numpy==2.4.6
onnx-graphsurgeon==0.6.1

--extra-index-url https://download.pytorch.org/whl/cu132
torch==2.12.0
torchvision

uv pip install -r requirements.txt
uv pip install -r requirements-server.txt

mkdir -p build
cd build

cmake .. \
    -DTRT_PACKAGE_DIR=/usr \
    -DCUDA_CTK_VERSION=13.2 \
    -DCMAKE_TOOLCHAIN_FILE=cmake/aarch64_linux_toolchain.cmake \
    -DEMBEDDED_TARGET=jetson-thor \
    -DENABLE_CUTE_DSL=ALL \
    -DENABLE_NVTX_PROFILING=ON \
    -DBUILD_PYTHON_BINDINGS=ON \
    -Dpybind11_DIR="$(python -m pybind11 --cmakedir)"

make -j10


uv build --wheel

uv pip install dist/tensorrt_edgellm-0.8.0-py3-none-any.whl

g.rado · July 2, 2026, 11:31am

Thanks a lot. That confirms the issue.

I also ended up with a working build, but only after isolating the pybind/device-link build path. Good to know that the math.h issue is reproducible on Thor and that you already opened PRs for it.

For now I will keep my current working build, but I’ll watch the PRs and rebuild from a clean checkout once the fixes are merged.

Thanks for taking the time to document your build steps.

Topic		Replies	Views
TensorRT Edge-LLM on the AGX Thor Jetson Thor tensorrt , generative_ai	10	1191	December 4, 2025
TensorRT LLM Jetson Thor tensorrt , generative_ai	5	953	October 27, 2025
Cuda 13+ needed for TensorRT-Edge LLM on Drive AGX Thor? DRIVE AGX Thor General	3	168	April 21, 2026
TensorRT-Edge-LLM DGX Spark / GB10	2	267	April 29, 2026
JetPack 7.0/Jetson Linux 38.2 for NVIDIA Jetson Thor is now live Jetson Thor cudnn , llama	19	3837	October 27, 2025
Jetson AGX Thor: official PyTorch 25.08 container works for Conv2d and ResNet18, but pip-installed PyTorch 2.12.0.dev+cu128 fails with "no kernel im Jetson Thor pytorch , cudnn , cublas	1	259	March 21, 2026
Inquiry on any updated support for tensorrt-llm support nvidia orin AGX? Jetson AGX Orin tensorrt , generative_ai , llama	3	362	June 3, 2025
Failed running build_wheel.py for TensorRT-LLM Jetson AGX Orin tensorrt , llm	11	223	January 13, 2026
TensorRT-LLM for jetson errors Jetson AGX Orin generative_ai , paligemma , kosmos-2 , llama	13	1112	December 19, 2024
tensorrt推理Qwen/Qwen3-VL-8B-Instruct不兼容 Jetson Thor tensorrt , llm	3	403	October 27, 2025

TensorRT-Edge-LLM on Jetson AGX Thor: “No math backend found” although CUDA is installed

Related topics