Description
I’m following below NVIDIA document and trying to run TensorRT-LLM container and getting below error
pip3 install tensorrt_llm==0.9.0 -U --extra-index-url https ://pypi.nvidia.com --verbose
Using pip 21.3.1 from /usr/lib/python3.9/site-packages/pip (python 3.9)
Looking in indexes: https: //pypi.org/simple, https ://pypi.ngc.nvidia.com, https ://pypi.nvidia.com
Collecting tensorrt_llm==0.9.0
Downloading tensorrt_llm-0.9.0.tar.gz (1.1 kB)
Running command /usr/bin/python3 /tmp/pip-standalone-pip-s6c23vg6/env_pip.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-yymw5ml7/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https ://pypi.org/simple --extra-index-url https ://pypi.ngc.nvidia.com --extra-index-url https ://pypi.nvidia.com --trusted-host pypi.ngc.nvidia.com – nvidia-stub
Looking in indexes: https ://pypi.org/simple, https ://pypi.ngc.nvidia.com, https ://pypi.ngc.nvidia.com, https ://pypi.nvidia.com
Collecting nvidia-stub
Downloading nvidia_stub-0.1.8-py3-none-any.whl (66 kB)
Installing collected packages: nvidia-stub
Successfully installed nvidia-stub-0.1.8
WARNING: Running pip as the ‘root’ user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https ://pip.pypa.io/warnings/venv
Installing build dependencies … done
Running command /usr/bin/python3 /usr/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpnexg8pj0
Getting requirements to build wheel … done
Running command /usr/bin/python3 /usr/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpf_jwkn5c
INFO:nvidia-stub:Testing wheel tensorrt_llm-0.9.0-cp310-cp310-linux_x86_64.whl against tag cp310-cp310-linux_x86_64
INFO:nvidia-stub:Testing wheel tensorrt_llm-0.9.0-cp310-cp310-win_amd64.whl against tag cp310-cp310-win_amd64
File “/tmp/pip-build-env-yymw5ml7/overlay/lib/python3.9/site-packages/nvidia_stub/wheel.py”, line 177, in download_wheel
return download_manual(wheel_directory, distribution, version)
File “/tmp/pip-build-env-yymw5ml7/overlay/lib/python3.9/site-packages/nvidia_stub/wheel.py”, line 144, in download_manual
raise RuntimeError(f"Didn’t find wheel for {distribution} {version}")
Traceback (most recent call last):
File “/usr/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py”, line 156, in prepare_metadata_for_build_wheel
hook = backend.prepare_metadata_for_build_wheel
AttributeError: module ‘nvidia_stub.buildapi’ has no attribute ‘prepare_metadata_for_build_wheel’
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “/tmp/pip-build-env-yymw5ml7/overlay/lib/python3.9/site-packages/nvidia_stub/wheel.py”, line 177, in download_wheel
return download_manual(wheel_directory, distribution, version)
File “/tmp/pip-build-env-yymw5ml7/overlay/lib/python3.9/site-packages/nvidia_stub/wheel.py”, line 144, in download_manual
raise RuntimeError(f"Didn’t find wheel for {distribution} {version}")
RuntimeError: Didn’t find wheel for tensorrt-llm 0.9.0
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “/usr/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py”, line 363, in
main()
File “/usr/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py”, line 345, in main
json_out[‘return_val’] = hook(**hook_input[‘kwargs’])
File “/usr/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py”, line 160, in prepare_metadata_for_build_wheel
whl_basename = backend.build_wheel(metadata_directory, config_settings)
File “/tmp/pip-build-env-yymw5ml7/overlay/lib/python3.9/site-packages/nvidia_stub/buildapi.py”, line 29, in build_wheel
return download_wheel(pathlib.Path(wheel_directory), config_settings)
File “/tmp/pip-build-env-yymw5ml7/overlay/lib/python3.9/site-packages/nvidia_stub/wheel.py”, line 179, in download_wheel
report_install_failure(distribution, version, exception_context)
File “/tmp/pip-build-env-yymw5ml7/overlay/lib/python3.9/site-packages/nvidia_stub/error.py”, line 63, in report_install_failure
raise InstallFailedError(
nvidia_stub.error.InstallFailedError:
The installation of tensorrt-llm for version 0.9.0 failed.
This is a special placeholder package which downloads a real wheel package
from https ://pypi.nvidia.com. If https ://pypi.nvidia.com is not reachable, we
cannot download the real wheel file to install.
You might try installing this package via
$ pip install --extra-index-url https ://pypi.nvidia.com tensorrt-llm
Here is some debug information about your platform to include in any bug
report:
Python Version: CPython 3.9.20
Operating System: Linux 5.10.233-224.894.amzn2.x86_64
CPU Architecture: x86_64
Driver Version: 550.144
CUDA Version: 12.4
Preparing metadata (pyproject.toml) … error
WARNING: Discarding https ://files.pythonhosted.org/packages/9c/47/7a0edc16c983f65f7eb0d276583fc428a1e23423131a94818f0a3589706d/tensorrt_llm-0.9.0.tar.gz#sha256=559b1298edfced68fc34847e10ba260f7541afab46962e0215528835751c08bd (from https ://pypi.org/simple/tensorrt-llm/) (requires-python:<4,>=3.7).Command errored out with exit status 1: /usr/bin/python3 /usr/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpf_jwkn5c Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement tensorrt_llm==0.9.0 (from versions: 0.5.0, 0.5.0.post1, 0.6.1, 0.7.0, 0.7.1, 0.8.0.dev20240116, 0.8.0.dev20240123, 0.8.0.dev2024011601, 0.8.0.dev2024012301, 0.8.0.dev2024012302, 0.8.0.dev2024013000, 0.8.0, 0.9.0.dev2024020600, 0.9.0.dev2024022000, 0.9.0.dev2024022700, 0.9.0.dev2024030500, 0.9.0.dev2024031200, 0.9.0.dev2024031900, 0.9.0.dev2024032600, 0.9.0.dev2024040200, 0.9.0.dev2024040900, 0.9.0, 0.10.0.dev2024041600, 0.10.0.dev2024042300, 0.10.0.dev2024043000, 0.10.0.dev2024050700, 0.10.0, 0.11.0.dev2024051400, 0.11.0.dev2024052100, 0.11.0.dev2024052800, 0.11.0.dev2024060400, 0.11.0.dev2024061100, 0.11.0.dev2024061800, 0.11.0.dev2024062500, 0.11.0, 0.12.0.dev2024070200,0.12.0.dev2024070900, 0.12.0.dev2024071600, 0.12.0.dev2024072300, 0.12.0.dev2024072301, 0.12.0.dev2024072302, 0.12.0.dev2024073000, 0.12.0.dev2024080600, 0.12.0, 0.13.0.dev2024081300, 0.13.0.dev2024082000, 0.13.0.dev2024082700, 0.13.0.dev2024090300, 0.13.0, 0.14.0.dev2024091000, 0.14.0.dev2024091700, 0.14.0.dev2024092400, 0.14.0.dev2024092401, 0.14.0.dev2024100100, 0.14.0.dev2024100800, 0.14.0, 0.15.0.dev2024101500, 0.15.0.dev2024102200, 0.15.0.dev2024102900, 0.15.0.dev2024110500, 0.15.0.dev2024111200, 0.15.0, 0.16.0.dev2024111900, 0.16.0.dev2024112600, 0.16.0.dev2024120300, 0.16.0.dev2024121000, 0.16.0, 0.17.0.dev2024121700, 0.17.0, 0.17.0.post1)
ERROR: No matching distribution found for tensorrt_llm==0.9.0
Environment
GPU Type: NVIDIA A10G
Nvidia Driver Version: 550.144.03
CUDA Version: 12.4
Operating System + Version:
NAME=“Amazon Linux”
VERSION=“2”
ID=“amzn”
ID_LIKE=“centos rhel fedora”
VERSION_ID=“2”
PRETTY_NAME=“Amazon Linux 2”
ANSI_COLOR=“0;33”
CPE_NAME=“cpe:2.3:o:amazon:amazon_linux:2”
HOME_URL=“https ://amazonlinux.com/”
SUPPORT_END=“2026-06-30”
Python Version (if applicable): Python 3.7.16
Steps To Reproduce
Using Amazon EC2 g5.8xlarge instance type and following below URL