Jetson Orin Nano restarts while running building text-generation-webui (Loading exllamav2_ext extension (JIT)...)

[natarajnvidia]

post running jetson-containers build text-generation-webui the machine restarts and it hangs while loading the exllamav2_ext

the system was throttling the power and cpu was used 100% the exact error was “System throttled due to Over-current”
CPU 0 - 5 all at 100%
cpu-thermal 56.6 C
gpu-thermal 54.5 C
tj-thermal 56.6 C

The log file is as below

---

FlashAttention version 2.7.2.post1
– Building container text-generation-webui:r36.4.3-exllama

DOCKER_BUILDKIT=0 docker build --network=host --tag text-generation-webui:r36.4.3-exllama
–file /home/nataraj/aidata/jetson-containers/packages/llm/exllama/Dockerfile
–build-arg BASE_IMAGE=text-generation-webui:r36.4.3-flash-attention
–build-arg EXLLAMA_VERSION=“0.2.7”
–build-arg EXLLAMA_BRANCH=“0.2.7”
–build-arg FORCE_BUILD=“off”
/home/nataraj/aidata/jetson-containers/packages/llm/exllama
2>&1 | tee /home/nataraj/aidata/jetson-containers/logs/20250210_030831/build/text-generation-webui_r36.4.3-exllama.txt; exit ${PIPESTATUS[0]}

DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
environment-variable.

Sending build context to Docker daemon 19.46kB
Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
—> a1d4c6aa0e57
Step 3/5 : ARG EXLLAMA_VERSION EXLLAMA_BRANCH FORCE_BUILD=off
jetson-containers build text-generation-webui.txt (60.3 KB)

…
…
…
irement already satisfied: torch>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from exllamav2==0.2.7) (2.5.0)
Requirement already satisfied: safetensors>=0.3.2 in /usr/local/lib/python3.10/dist-packages (from exllamav2==0.2.7) (0.5.2)
Requirement already satisfied: sentencepiece>=0.1.97 in /usr/local/lib/python3.10/dist-packages (from exllamav2==0.2.7) (0.2.0)
Requirement already satisfied: pygments in /usr/local/lib/python3.10/dist-packages (from exllamav2==0.2.7) (2.19.1)
Collecting websockets (from exllamav2==0.2.7)
Downloading https://pypi.jetson-ai-lab.dev/root/pypi/%2Bf/f2e/53c72052f2596/websockets-14.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (169 kB)
Requirement already satisfied: regex in /usr/local/lib/python3.10/dist-packages (from exllamav2==0.2.7) (2024.11.6)
Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from exllamav2==0.2.7) (1.26.4)
Requirement already satisfied: rich in /usr/local/lib/python3.10/dist-packages (from exllamav2==0.2.7) (13.9.4)
Requirement already satisfied: pillow>=9.1.0 in /usr/local/lib/python3.10/dist-packages (from exllamav2==0.2.7) (11.1.0)
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch>=2.2.0->exllamav2==0.2.7) (3.17.0)
Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from torch>=2.2.0->exllamav2==0.2.7) (4.12.2)
Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=2.2.0->exllamav2==0.2.7) (3.4.2)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=2.2.0->exllamav2==0.2.7) (3.1.5)
Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch>=2.2.0->exllamav2==0.2.7) (2024.9.0)
Requirement already satisfied: sympy==1.13.1 in /usr/local/lib/python3.10/dist-packages (from torch>=2.2.0->exllamav2==0.2.7) (1.13.1)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy==1.13.1->torch>=2.2.0->exllamav2==0.2.7) (1.3.0)
Collecting cramjam>=2.3 (from fastparquet->exllamav2==0.2.7)
Downloading https://pypi.jetson-ai-lab.dev/root/pypi/%2Bf/8e0/c5d98a4e791f0/cramjam-2.9.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 114.8 MB/s eta 0:00:00
Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from fastparquet->exllamav2==0.2.7) (24.2)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas->exllamav2==0.2.7) (2.9.0.post0)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->exllamav2==0.2.7) (2025.1)
Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.10/dist-packages (from pandas->exllamav2==0.2.7) (2025.1)
Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich->exllamav2==0.2.7) (3.0.0)
Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py>=2.2.0->rich->exllamav2==0.2.7) (0.1.2)
Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil>=2.8.2->pandas->exllamav2==0.2.7) (1.16.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=2.2.0->exllamav2==0.2.7) (3.0.2)
Installing collected packages: websockets, cramjam, fastparquet, exllamav2
Successfully installed cramjam-2.9.1 exllamav2-0.2.7 fastparquet-2024.11.0 websockets-14.2

• python3 -c ‘import exllamav2; print(exllamav2.version);’
  Loading exllamav2_ext extension (JIT)…

---

Note: i do have uploaded the detailed log

It looks like the issue that i am facing is a little different than the one

Text-generation-webui install error

Hi when I try to install ``` text-generation-webui … I got error like below, anybody know what’s going on? louie001@localhost:~$ jetson-containers run $(autotag text-generation-webui) Namespace(packages=[‘text-generation-webui’], prefer=[‘local’, ‘registry’, ‘build’], disable=[‘’], user=‘dustynv’, output=‘/tmp/autotag’, quiet=False, verbose=False) – L4T_VERSION=36.4.2 JETPACK_VERSION=6.1.1 CUDA_VERSION=12.6 – Finding compatible container image for [‘text-generation-webui’] text-generati…

[natarajnvidia]

This error is consistently reproducible every time

I do have uploaded the relevant screenshots for the same

[AastaLLL]

Hi,

When an OC event occurs, the CPU/GPU clocks will be reduced to 50%.
But the average CPU clock in your use case remains > 1700 MHz. It looks like the OC throttling time is short.

Besides OC throttling, another reason that causes hang and reboot is out of memory.
Could you share the tegrastats output with us?

$ sudo tegrastats

Please also try to optimize the memory to see if this helps (will turn off the desktop GUI).

🔖 Memory optimization - NVIDIA Jetson AI Lab

Thanks.

[natarajnvidia]

Appreciate your response,

This time i did disable the GUI as per the recommendation (thanks for sharing) .The system did not hang and crash this time around.

There was RuntimeError: Error compiling objects for extension and the snippet is as below

The command '/bin/sh -c /tmp/exllama/install.sh || /tmp/exllama/build.sh' returned a non-zero code: 1
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/nataraj/aidata/jetson-containers/jetson_containers/build.py", line 122, in <module>
    build_container(args.name, args.packages, args.base, args.build_flags, args.build_args, args.simulate, args.skip_tests, args.test_only, args.push, args.no_github_api, args.skip_packages)
  File "/home/nataraj/aidata/jetson-containers/jetson_containers/container.py", line 147, in build_container
    status = subprocess.run(cmd.replace(_NEWLINE_, ' '), executable='/bin/bash', shell=True, check=True)  
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'DOCKER_BUILDKIT=0 docker build --network=host --tag text-generation-webui:r36.4.3-exllama --file /home/nataraj/aidata/jetson-containers/packages/llm/exllama/Dockerfile --build-arg BASE_IMAGE=text-generation-webui:r36.4.3-flash-attention --build-arg EXLLAMA_VERSION="0.2.7" --build-arg EXLLAMA_BRANCH="0.2.7" --build-arg FORCE_BUILD="off" /home/nataraj/aidata/jetson-containers/packages/llm/exllama 2>&1 | tee /home/nataraj/aidata/jetson-containers/logs/20250210_200319/build/text-generation-webui_r36.4.3-exllama.txt; exit ${PIPESTATUS[0]}' returned non-zero exit status 1.

the output of the log file snippet is below (sorry to post the long log message here) i do have uploaded the complete log file as well.

[quote="natarajnvidia, post:1, topic:323102"]
jetson-containers build text-generation-webui
[/quote]

Building ExLlamaV2 0.2.7
+ echo 'Building ExLlamaV2 0.2.7'
+ cd /opt/exllamav2
....

copying exllamav2/ext.py -> build/lib.linux-aarch64-cpython-310/exllamav2
copying exllamav2/__init__.py -> build/lib.linux-aarch64-cpython-310/exllamav2
copying exllamav2/module.py -> build/lib.linux-aarch64-cpython-310/exllamav2
copying exllamav2/headnorm.py -> build/lib.linux-aarch64-cpython-310/exllamav2
copying exllamav2/model.py -> build/lib.linux-aarch64-cpython-310/exllamav2
....
creating exllamav2.egg-info
....

!!
  check.warn(importable)
/usr/local/lib/python3.10/dist-packages/setuptools/command/build_py.py:212: _Warning: Package 'exllamav2.server' is absent from the `packages` configuration.
!!

        ********************************************************************************
        ############################
        # Package would be ignored #
        ############################
        Python recognizes 'exllamav2.server' as an importable package[^1],
        but it is absent from setuptools' `packages` configuration.

        This leads to an ambiguous overall configuration. If you want to distribute this
        package, please make sure that 'exllamav2.server' is explicitly added
        to the `packages` configuration field.

......
In file included from /opt/exllamav2/exllamav2/exllamav2_ext/cpp/sampling_avx2.cpp:15:
/opt/exllamav2/exllamav2/exllamav2_ext/cpp/avx_mathfun.h:7:10: fatal error: immintrin.h: No such file or directory
    7 | #include <immintrin.h>
      |          ^~~~~~~~~~~~~
compilation terminated.

.....



ile "/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py", line 868, in build_extensions
    build_ext.build_extensions(self)
  File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/command/build_ext.py", line 481, in build_extensions
    self._build_extensions_serial()
  File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/command/build_ext.py", line 507, in _build_extensions_serial
    self.build_extension(ext)
  File "/usr/local/lib/python3.10/dist-packages/setuptools/command/build_ext.py", line 264, in build_extension
    _build_ext.build_extension(self, ext)
  File "/usr/local/lib/python3.10/dist-packages/Cython/Distutils/build_ext.py", line 135, in build_extension
    super(build_ext, self).build_extension(ext)
  File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/command/build_ext.py", line 562, in build_extension
    objects = self.compiler.compile(
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py", line 681, in unix_wrap_ninja_compile
    _write_ninja_file_and_compile_objects(
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py", line 1784, in _write_ninja_file_and_compile_objects
    _run_ninja_build(
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py", line 2120, in _run_ninja_build
    raise RuntimeError(message) from e
RuntimeError: Error compiling objects for extension
The command '/bin/sh -c /tmp/exllama/install.sh || /tmp/exllama/build.sh' returned a non-zero code: 1
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/nataraj/aidata/jetson-containers/jetson_containers/build.py", line 122, in <module>
    build_container(args.name, args.packages, args.base, args.build_flags, args.build_args, args.simulate, args.skip_tests, args.test_only, args.push, args.no_github_api, args.skip_packages)
  File "/home/nataraj/aidata/jetson-containers/jetson_containers/container.py", line 147, in build_container
    status = subprocess.run(cmd.replace(_NEWLINE_, ' '), executable='/bin/bash', shell=True, check=True)  
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'DOCKER_BUILDKIT=0 docker build --network=host --tag text-generation-webui:r36.4.3-exllama --file /home/nataraj/aidata/jetson-containers/packages/llm/exllama/Dockerfile --build-arg BASE_IMAGE=text-generation-webui:r36.4.3-flash-attention --build-arg EXLLAMA_VERSION="0.2.7" --build-arg EXLLAMA_BRANCH="0.2.7" --build-arg FORCE_BUILD="off" /home/nataraj/aidata/jetson-containers/packages/llm/exllama 2>&1 | tee /home/nataraj/aidata/jetson-containers/logs/20250210_200319/build/text-generation-webui_r36.4.3-exllama.txt; exit ${PIPESTATUS[0]}' returned non-zero exit status 1.

The oom-kill did kick in couple of times and the log is as below (from dmesg)

[  892.973699] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=wpa_supplicant.service,mems_allowed=0,global_oom,task_memcg=/system.slice/docker-8ec65c4db65dd7d8ffaa42752cb6ce763d7b522aaaefe64460ca193e82486a04.scope,task=cc1plus,pid=5008,uid=0
[  892.973724] Out of memory: Killed process 5008 (cc1plus) total-vm:1390404kB, anon-rss:860364kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:2648kB oom_score_adj:0
[  918.090107] cc1plus invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[  918.090124] CPU: 4 PID: 5005 Comm: cc1plus Tainted: G           O      5.15.148-tegra #1
[  918.090129] Hardware name: NVIDIA NVIDIA Jetson Orin Nano Engineering Reference Developer Kit Super/Jetson, BIOS 36.4.3-gcid-38968081 01/08/2025
[  918.090132] Call trace:
[  918.090133]  dump_backtrace+0x0/0x1d0
[  918.090147]  show_stack+0x34/0x50
[  918.090152]  dump_stack_lvl+0x68/0x8c
[  918.090158]  dump_stack+0x18/0x3c
[  918.090160]  dump_header+0x50/0x208
[  918.090165]  oom_kill_process+0x248/0x260
[  918.090170]  out_of_memory+0x1f4/0x560
[  918.090172]  __alloc_pages+0xd3c/0xe10
[  918.090176]  alloc_pages+0xa0/0x1a0
[  918.090179]  __page_cache_alloc+0xb8/0xe0
[  918.090184]  pagecache_get_page+0x1e0/0x560
[  918.090188]  filemap_fault+0x434/0x9f0
[  918.090191]  __do_fault+0x48/0x1e0
[  918.090196]  __handle_mm_fault+0x8e8/0x1020
[  918.090199]  handle_mm_fault+0xf8/0x240
[  918.090202]  do_page_fault+0x148/0x480
[  918.090206]  do_translation_fault+0xf4/0x110
[  918.090209]  do_mem_abort+0x58/0xe0
[  918.090212]  el0_ia+0x5c/0xf0
[  918.090216]  el0t_64_sync_handler+0x12c/0x130
[  918.090219]  el0t_64_sync+0x1a4/0x1a8
[  918.090221] Mem-Info:
[  918.090224] active_anon:1361968 inactive_anon:194609 isolated_anon:160



 918.090671] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=docker-8ec65c4db65dd7d8ffaa42752cb6ce763d7b522aaaefe64460ca193e82486a04.scope,mems_allowed=0,global_oom,task_memcg=/system.slice/docker-8ec65c4db65dd7d8ffaa42752cb6ce763d7b522aaaefe64460ca193e82486a04.scope,task=cc1plus,pid=5005,uid=0
[  918.090696] Out of memory: Killed process 5005 (cc1plus) total-vm:1471544kB, anon-rss:878104kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:2836kB oom_score_adj:0
[ 1009.442243] cpufreq: cpu0,cur:985000,set:729600,delta:255400,set ndiv:57
[ 1022.637274] cpufreq: cpu0,cur:1004000,set:729600,delta:274400,set ndiv:57
[ 1126.097142] cpufreq: cpu0,cur:1108000,set:729600,delta:378400,set ndiv:57
[ 1159.563732] cpufreq: cpu0,cur:932000,set:729600,delta:202400,set ndiv:57
[ 1408.107088] cpufreq: cpu0,cur:1067000,set:729600,delta:337400,set ndiv:57
[ 1502.432774] cpufreq: cpu0,cur:1062000,set:729600,delta:332400,set ndiv:57
[ 1525.768453] cpufreq: cpu0,cur:1062000,set:729600,delta:332400,set ndiv:57
[ 1557.208283] cpufreq: cpu0,cur:1061000,set:729600,delta:331400,set ndiv:57
[ 1565.321494] cpufreq: cpu0,cur:1039000,set:729600,delta:309400,set ndiv:57
[ 1575.465688] cpufreq: cpu0,cur:1047000,set:729600,delta:317400,set ndiv:57

Note : Uploaded the complete error log jetson-containers build text-generation-webui-error-1.txt (138.6 KB)

Note: As requested i do have uploaded tegrastats.txt (324.4 KB)

[AastaLLL]

Hi,

Do you flash and install all the components from the JetPack 6.2?
Or you have upgraded from other version like JetPack 6.1?

We have confirmed that the text-generation-webui container can work on a flash JetPack 6.2 environment.
Please check the below link to see if the info helps:

Couldn't find a compatible container for text-generation-webui

Hi, We test the container with the latest JetPack 6.2 and it can work as expected. Please try if the below command can also work on your side: $ jetson-containers build text-generation-webui ... -- Done building container text-generation-webui:r36.4.3 $ sudo docker run -it --rm --runtime=nvidia --network=host --volume /home/nvidia/jetson-containers/packages/llm/auto_awq:/test --volume /home/nvidia/jetson-containers/data:/data --workdir /test text-generation-webui:r36.4.3 /usr/local/lib/python…

Thanks.

[natarajnvidia]

The firmware for this box came with an older version and i had to update the firmware so i had to install jetpack 5.3 upgrade the firmware and then deployed 6.2.

[AastaLLL]

Hi,

You can find the details to setup the Orin Nano in the below guide:

🚀 Initial Setup Guide - Jetson Orin Nano - NVIDIA Jetson AI Lab

Thanks.

[natarajnvidia]

Thank you very much for pointing yes, i followed the exact same guide

1. JetPack 5.1.3 SD first and then moved to
2. Updated QSPI Updater
3. Then updated with Jeson 6.2

[AastaLLL]

Hi,

Just want to double-check, have you applied the memory optimization step after reflashing to JetPack 6.2?

Since your error is related to out of memory. Memory optimization might help.

[  892.973724] Out of memory: Killed process 5008 (cc1plus) total-vm:1390404kB, anon-rss:860364kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:2648kB oom_score_adj:0

Thanks.

[natarajnvidia]

@AastaLLL

Just want to double-check, have you applied the memory optimization step after reflashing to JetPack 6.2?

After applying the memory optimization the hardware does not hang and crash any more hence closing this issue.