Bitsandbytes on Nvidia Jetson AGX Orin

Robotics & Edge Computing Jetson Systems Jetson AGX Orin
1

I am trying to install bitsandbytes on my Nvidia Jetson AGX Orin(64 GB) GPU with compute compatibility 8.7 and cuda version 12.6 from source because the pip package didn’t work well in my case. I used these commands

git clone https://github.com/TimDettmers/bitsandbytes.git
cd bitsandbytes
cmake ..
CUDA_VERSION=126 python setup.py install

and to test if the bitsandbytes module is installed and working well I run this command
python -m bitsandbytes but I got the following errors

bitsandbytes library load error: Configured CUDA binary not found at /mnt/storage/LogLLM/bitsandbytes/bitsandbytes/libbitsandbytes_cuda126.so
 If you are using Intel CPU/XPU, please install intel_extension_for_pytorch to enable required ops
Traceback (most recent call last):
  File "/mnt/storage/LogLLM/bitsandbytes/bitsandbytes/cextension.py", line 319, in <module>
    lib = get_native_library()
  File "/mnt/storage/LogLLM/bitsandbytes/bitsandbytes/cextension.py", line 282, in get_native_library
    raise RuntimeError(f"Configured {BNB_BACKEND} binary not found at {cuda_binary_path}")
RuntimeError: Configured CUDA binary not found at /mnt/storage/LogLLM/bitsandbytes/bitsandbytes/libbitsandbytes_cuda126.so
================ bitsandbytes v0.47.0.dev0 =================
Platform: Linux-5.15.148-tegra-aarch64-with-glibc2.35
  libc: glibc-2.35
Python: 3.10.18
PyTorch: 2.6.0+cu126
  CUDA: 12.6
  HIP: N/A
  XPU: N/A
Related packages:
  accelerate: 1.8.1
  diffusers: 0.34.0
  numpy: 2.2.6
  pip: 25.1
  peft: 0.15.2
  safetensors: 0.5.3
  transformers: 4.53.0
  triton: not found
  trl: not found
============================================================
PyTorch settings found: CUDA_VERSION=126, Highest Compute Capability: (8, 7).
Library not found: /mnt/storage/LogLLM/bitsandbytes/bitsandbytes/libbitsandbytes_cuda126.so. Maybe you need to compile it from source?
Checking that the library is importable and CUDA is callable...
Traceback (most recent call last):
  File "/mnt/storage/miniconda3/envs/logllm/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/mnt/storage/miniconda3/envs/logllm/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/mnt/storage/LogLLM/bitsandbytes/bitsandbytes/__main__.py", line 4, in <module>
    main()
  File "/mnt/storage/LogLLM/bitsandbytes/bitsandbytes/diagnostics/main.py", line 107, in main
    raise e
  File "/mnt/storage/LogLLM/bitsandbytes/bitsandbytes/diagnostics/main.py", line 96, in main
    sanity_check()
  File "/mnt/storage/LogLLM/bitsandbytes/bitsandbytes/diagnostics/main.py", line 35, in sanity_check
    p1 = p.data.sum().item()
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.```
2

*** Please note that this reply is generated by LLM automatically ***

I’d be happy to help you troubleshoot the issue with installing bitsandbytes on your Nvidia Jetson AGX Orin (64 GB) GPU.

First, let’s break down the information you provided:

  • You’re using an Nvidia Jetson AGX Orin (64 GB) GPU with compute compatibility 8.7 and CUDA version 12.6.
  • You’re trying to install bitsandbytes from source because the pip package didn’t work well for you.
  • You’ve run some commands to install bitsandbytes, but unfortunately, you didn’t provide the exact commands you used.
  • You’re testing the installation by running a command, but again, you didn’t provide the exact command.
  • You’re encountering errors, but you didn’t provide the exact error messages.

To better assist you, I’ll provide some general guidance on installing bitsandbytes on a Jetson AGX Orin device.

Prerequisites

Before installing bitsandbytes, make sure you have the following prerequisites:

  1. CUDA 12.6 installed on your Jetson AGX Orin device. You can verify this by running nvcc --version in your terminal.
  2. Python 3.8 or later installed on your device. You can verify this by running python3 --version in your terminal.

Installing bitsandbytes from source

To install bitsandbytes from source, you’ll need to clone the repository, install the dependencies, and then build the library. Here are the general steps:

  1. Clone the bitsandbytes repository: git clone https://github.com/bitsandbytes/bitsandbytes.git
  2. Install the dependencies: pip3 install -r requirements.txt
  3. Build the library: python3 setup.py install

Troubleshooting

If you’re encountering errors during the installation process, here are some potential issues to check:

  1. CUDA version: Make sure your CUDA version is compatible with the bitsandbytes library. You can check the CUDA version required by bitsandbytes in the setup.py file.
  2. Python version: Ensure you’re using a compatible Python version. bitsandbytes supports Python 3.8 and later.
  3. Dependencies: Verify that all dependencies are installed correctly. You can check the dependencies required by bitsandbytes in the requirements.txt file.
  4. Build errors: If you’re encountering build errors, check the error messages for any clues about the issue. You can also try cleaning the build directory and rebuilding the library.

Comparison with other Jetson devices

If you’re experiencing issues with installing bitsandbytes on your Jetson AGX Orin device, you may want to compare the installation process with other Jetson devices. For example, you can check the installation instructions for the Jetson Orin Nano or Jetson TX2 devices to see if there are any differences in the installation process.

In summary, to troubleshoot the issue with installing bitsandbytes on your Nvidia Jetson AGX Orin (64 GB) GPU, I recommend verifying the prerequisites, checking the installation commands, and troubleshooting any errors that occur during the installation process. If you’re still encountering issues, feel free to provide more details about the errors you’re experiencing, and I’ll do my best to assist you further.

4

cuda-compile

git clone https://github.com/bitsandbytes-foundation/bitsandbytes.git && cd bitsandbytes/

cmake -DCOMPUTE_BACKEND=cuda -S .
make
pip install .

following from github_bitsandbytes

" bitsandbytes has the following minimum requirements for all platforms:
Python 3.9+
PyTorch 2.2+ "