TensorRT 10.8.0.43 and older Incompatible with new glibc 2.41+r2+g0a7c7a3e283a

Description

Hello,

the TensorRT 10.8.0.43 and also older version (tested with 10.7.0.23) is incompatible with newest glibc version

We are using TRT to inference and build TRT engines from mainly Yolo networks.

Inference is working but when attempted to create a TRTEngine from a ONNX model I got this error (exception)

[nvidialogger]: createInferBuilder: Error Code 6: API Usage Error (Unable to load library: libnvinfer_builder_resource.so.10.8.0:
libnvinfer_builder_resource.so.10.8.0: cannot enable executable stack as shared object requires: Invalid argument)
  1/2 FAIL : DetectorTest::imageDetectionTest()
             [EXCEPTION] Unexpected exception thrown. Error message: Could not build trt engine : Could not intialize TRT Builder

Downgrading to glbc 2.40+r66+g7d4b6bcae91f Fixes the issue, but downgrading should not be a solution since downgrading a system package can introduce instability to a system.

It is working with glibc version 2.40+r66+g7d4b6bcae91f

Environment

TensorRT Version: 10.8.0.43
GPU Type: RTX2070
Nvidia Driver Version: 570.86.16
CUDA Version: 12.8.0
CUDNN Version: 9.5.1.17
Operating System + Version: Arch Linux + Linux 6.13.1
Glibc: 2.41+r2+g0a7c7a3e283a

Steps To Reproduce

Update to latest glibc (2.41) and try to build any trtengine with createInferBuilder

@HowP, can you provide more information on this issue? For example where is glibc 2.41 coming from, such as what is the path for where it’s installed? If you can provide a Dockerfile for example to help reproduce the environment that would help. glibc 2.41 is very new so we can’t be certain it’s not a regression or known issue in glibc.

cannot enable executable stack as shared object requires: Invalid argument

This almost looks like a loader issue, but I can’t confirm that with any sort of certainty since I’ve never seen anything like this before. The only link I found that might be sort of related is this: rhel - chcon fails to fix "cannot enable executable stack as shared object requires" err under RHEL6 - Unix & Linux Stack Exchange. This assume SELinux is enabled though.

@ework It is a arch distribution with official packages, so glibc is coming from Arch Linux - glibc 2.41+r2+g0a7c7a3e283a-1 (x86_64) the package is directly at https://archlinux.org/packages/core/x86_64/glibc/download/

As far as I know, the Ubuntu is also preparing to update to new glibc glibc package : Ubuntu ( A newer version of glibc is available for packaging: GLibC 2.41)

The path of the installed files is Arch Linux - glibc 2.41+r2+g0a7c7a3e283a-1 (x86_64) - File List

The SELinux is not enabled by default on arch system. My current kernel cmdline is

initrd=\amd-ucode.img initrd=\initramfs-linux.img root=/dev/nvme0n1p3 rw nvidia_drm.modeset=1  nvidia-drm.fbdev=0 nowatchdog

Here you can find more information about the glibc package Commits · main · Arch Linux / Packaging / Packages / glibc · GitLab

The TensorRT is taken directly from your website and installed correctly.

We have been using TRT for quite a while and I can safely say that this is a problem against new glibc

I will try to do a Dockerfile in a spare time.

This appears to be an intended change in glibc 2.41 after doing some more searching, adding the version to the search context. Here are a few other examples where this same issue is happening.

The particular release note item that seems relevant from https://lists.gnu.org/archive/html/info-gnu/2025-01/msg00014.html is this:

* dlopen and dlmopen no longer make the stack executable if a shared
  library requires it, either implicitly because of a missing GNU_STACK
  ELF header (and default ABI permission having the executable bit set)
  or explicitly because of the executable bit in GNU_STACK, and the
  stack is not already executable.  Instead, loading such objects will
  fail.

One workaround mentioned could be the following:

execstack -c libnvinfer_builder_resource.so.10.8.0

If this works we’ll need to ensure the library is built with this flag cleared. If not then the solution could be more complicated.

1 Like

@ework I confirm that executing the

execstack -c libnvinfer_builder_resource.so.10.8.0

fixes the issue. I do not know if the problem is relevant to other libraries as well, but I stumbled on libnvinfer_builder_resource only.

Thank you for your help.

I hope that the new TRT package will be available soon.


On the side note (for me not relevant issue):

Also I noticed that TRT when engine

could not open /sys/fs/cgroup/memory/memory.limit_in_bytes

This file is not present on arch linux but I believe it is not present on Ubuntu also since this location corresponds to cgroup v1 and Ubuntu and also Arch is using cgroupv2

@HowP thanks for confirming the workaround fixed your issue. We will work on including this fix in our next monthly release. I don’t think it will harm the other libraries to change the linker flags to also clear the execstack flag since they shouldn’t need it based on your results so far, and are likely already clear.

Regarding,

could not open /sys/fs/cgroup/memory/memory.limit_in_bytes

I believe this is a recent change and most important for embedded systems with less memory. Thanks for pointing out we should add support for cgroupsv2 as well.

@ework Hi I am still facing this issue, am using python 3.12.9 + cuda 12.8 + arch linux with gblic 2.41 (same as his)

[02/12/2025-15:19:32] [TRT] [E] createInferBuilder: Error Code 6: API Usage Error (Unable to load library: libnvinfer_builder_resource.so.10.7.0: libnvinfer_builder_resource.so.10.7.0: cannot enable executable stack as shared object requires: Invalid argument)
[vapoursynth] Script evaluation failed:
[vapoursynth] Python exception: pybind11::init(): factory function returned nullptr
[vapoursynth] 
[vapoursynth] Traceback (most recent call last):
[vapoursynth]   File "src/cython/vapoursynth.pyx", line 3387, in vapoursynth._vpy_evaluate
[vapoursynth]   File "src/cython/vapoursynth.pyx", line 3388, in vapoursynth._vpy_evaluate
[vapoursynth]   File "/home/syntist/.config/mpv/rife.py", line 16, in <module>
[vapoursynth]     builder = trt.Builder(logger)
[vapoursynth]               ^^^^^^^^^^^^^^^^^^^
[vapoursynth] TypeError: pybind11::init(): factory function returned nullptr
[vapoursynth] 
[vapoursynth] could not init VS
Disabling filter vapoursynth.00 because it has failed.

Also i tried doing

(.venv) ╰─❯ execstack -q /home/syntist/Documents/vs-rife/.venv/lib/python3.12/site-packages/tensorrt_libs/libnvinfer_builder_resource.so.10.7.0

X /home/syntist/Documents/vs-rife/.venv/lib/python3.12/site-packages/tensorrt_libs/libnvinfer_builder_resource.so.10.7.0
(.venv) ╰─❯ execstack -c /home/syntist/Documents/vs-rife/.venv/lib/python3.12/site-packages/tensorrt_libs/libnvinfer_builder_resource.so.10.7.0

execstack: /home/syntist/Documents/vs-rife/.venv/lib/python3.12/site-packages/tensorrt_libs/libnvinfer_builder_resource.so.10.7.0: section file offsets not monotonically increasing

So i can’t really disable it. I am using AUR (en) - execstack
this for execstack.

╰─❯ python --version
Python 3.12.9
pip list
Package                  Version
------------------------ ------------
certifi                  2025.1.31
charset-normalizer       3.4.1
filelock                 3.13.1
fsspec                   2024.6.1
idna                     3.10
Jinja2                   3.1.4
MarkupSafe               2.1.5
mpmath                   1.3.0
networkx                 3.3
numpy                    2.1.2
nvidia-cublas-cu12       12.6.4.1
nvidia-cuda-cupti-cu12   12.6.80
nvidia-cuda-nvrtc-cu12   12.6.77
nvidia-cuda-runtime-cu12 12.6.77
nvidia-cudnn-cu12        9.5.1.17
nvidia-cufft-cu12        11.3.0.4
nvidia-curand-cu12       10.3.7.77
nvidia-cusolver-cu12     11.7.1.2
nvidia-cusparse-cu12     12.5.4.2
nvidia-cusparselt-cu12   0.6.3
nvidia-nccl-cu12         2.21.5
nvidia-nvjitlink-cu12    12.6.85
nvidia-nvtx-cu12         12.6.77
packaging                24.2
pillow                   11.0.0
pip                      25.0.1
requests                 2.32.3
setuptools               75.8.0
sympy                    1.13.1
tensorrt                 10.7.0.post1
tensorrt_cu12            10.7.0.post1
tensorrt-cu12-bindings   10.7.0.post1
tensorrt-cu12-libs       10.7.0.post1
torch                    2.6.0+cu126
torch_tensorrt           2.6.0+cu126
torchvision              0.21.0+cu126
tqdm                     4.67.1
triton                   3.2.0
typing_extensions        4.12.2
urllib3                  2.3.0
VapourSynth              70
vsrife                   5.6.0
wheel                    0.45.1

For others looking at this thread later, see also: [Linux] [gblic 2.41] Unable to load library: libnvinfer_builder_resource.so.10.7.0

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.