Couldn't find a compatible container for text-generation-webui

[hardik13]

Hello everyone,
While running the commands to setup text generation webui from LLaVA - NVIDIA Jetson AI Lab by @dusty_nv, I am getting the following error message.

Error: Couldn’t find a compatible container for text-generation-webui

I am attaching the configurations of the Jetson AGX Orin, that I am using. Could someone please help me out with this.

Thank you very much!

[carolyuu]

Hi,
Here are some suggestions for the common issues:

1. Performance

Please run the below command before benchmarking deep learning use case:

$ sudo nvpmodel -m 0
$ sudo jetson_clocks

2. Installation

Installation guide of deep learning frameworks on Jetson:

• TensorFlow: Installing TensorFlow for Jetson Platform - NVIDIA Docs
• PyTorch: Installing PyTorch for Jetson Platform - NVIDIA Docs
  We also have containers that have frameworks preinstalled:
  Data Science, Machine Learning, AI, HPC Containers | NVIDIA NGC

3. Tutorial

Startup deep learning tutorial:

• Jetson-inference: Hello AI World guide to deploying deep-learning inference networks and deep vision primitives with TensorRT and NVIDIA Jetson
• TensorRT sample: Jetson/L4T/TRT Customized Example - eLinux.org

4. Report issue

If these suggestions don’t help and you want to report an issue to us, please attach the model, command/step, and the customized app (if any) with us to reproduce locally.

Thanks!

[AastaLLL]

Hi,

Could you share the complete log with us?
Suppose the script will start to build a container based on your environment information.

Thanks.

[hardik13]

Sure, please find the command and its associated logs in the images below.

Thanks!

[hardik13]

and this too!

[AastaLLL]

Hi,

Could you help to check the CUDA version in your environment?

$ ll /usr/local/cuda*

JetPack 6.1 should include CUDA 12.6 but it’s CUDA 12.1 detected in your environment.

Thanks.

[hardik13]

Hello,

Yes, I have manually linked the “/usr/local/cuda” to CUDA 12.1, since the model( haotian-liu/LLaVA: [NeurIPS’23 Oral] Visual Instruction Tuning (LLaVA) built towards GPT-4V level capabilities and beyond.) i am trying to run requires CUDA 12.1. I have linked it back to CUDA 12.6.

Now, while running the command:
jetson-containers run --workdir=/opt/text-generation-webui $(autotag text-generation-webui) *
** python3 download-model.py --output=/data/models/text-generation-webui *
** TheBloke/llava-v1.5-13B-GPTQ**

I am getting the following error:

Is there anything I can do?

[AastaLLL]

Hi,

Do you have custom Python version?

JetPack 6.1 should by default use Python 3.10.
But the error seems related to the Python3.6?

Thanks.

[hardik13]

No, I am using Python version 3.10.

[AastaLLL]

Hi,

We test the container with the latest JetPack 6.2 and it can work as expected.
Please try if the below command can also work on your side:

$ jetson-containers build text-generation-webui
...
-- Done building container text-generation-webui:r36.4.3
$ sudo docker run -it --rm --runtime=nvidia --network=host --volume /home/nvidia/jetson-containers/packages/llm/auto_awq:/test --volume /home/nvidia/jetson-containers/data:/data --workdir /test text-generation-webui:r36.4.3
/usr/local/lib/python3.10/dist-packages/transformers/utils/hub.py:128: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead.
  warnings.warn(
08:01:21-253741 INFO     Starting Text generation web UI                                                                                                                                                           
08:01:21-259695 WARNING  The --triton flag has been deprecated and will be removed soon. Please remove that flag.                                                                                                  
08:01:21-261235 WARNING                                                                                                                                                                                            
                         You are potentially exposing the web UI to the entire internet without any access password.                                                                                               
                         You can create one with the "--gradio-auth" flag like this:                                                                                                                               
                                                                                                                                                                                                                   
                         --gradio-auth username:password                                                                                                                                                           
                                                                                                                                                                                                                   
                         Make sure to replace username:password with your own.                                                                                                                                     
08:01:21-263644 INFO     Loading settings from "settings.yaml"                                                                                                                                                     

Running on local URL:  http://0.0.0.0:7860

The open the http://[IP]:7860 with a browser can see:

Thanks.

[hardik13]

Thank you very much for the solution. I am able to run it now.