Can't loading "TheBloke_llava-v1.5-13B-GPTQ" with AGXorin 32GB

I try to practice LLaVA tutorial from LLaVA - NVIDIA Jetson AI Lab with my AGX orin 32GB devkit but it returns “ERROR The model could not be loaded because its checkpoint file in .bin/.pt/.safetensors format could not be located.”.
Detailed log as bellow:
aaa@aaaadmin:~$ jetson-containers run --workdir=/opt/text-generation-webui $(autotag text-generation-webui) \

python3 server.py --listen
–model-dir /data/models/text-generation-webui
–model TheBloke_llava-v1.5-13B-GPTQ
–multimodal-pipeline llava-v1.5-13b
–loader autogptq
–disable_exllama
–verbose
Namespace(disable=[‘’], output=‘/tmp/autotag’, packages=[‘text-generation-webui’], prefer=[‘local’, ‘registry’, ‘build’], quiet=False, user=‘dustynv’, verbose=False)
– L4T_VERSION=35.1.0 JETPACK_VERSION=5.0.2 CUDA_VERSION=11.4
– Finding compatible container image for [‘text-generation-webui’]
dustynv/text-generation-webui:r35.4.1-cp310

  • sudo docker run --runtime nvidia -it --rm --network host --volume /tmp/argus_socket:/tmp/argus_socket --volume /etc/enctune.conf:/etc /enctune.conf --volume /etc/nv_tegra_release:/etc/nv_tegra_release --volume /tmp/nv_jetson_model:/tmp/nv_jetson_model --volume /var/run /dbus:/var/run/dbus --volume /var/run/avahi-daemon/socket:/var/run/avahi-daemon/socket --volume /var/run/docker.sock:/var/run/docker.so ck --volume /home/qsiadmin/jetson-containers/data:/data --device /dev/snd --device /dev/bus/usb --device /dev/video0 --device /dev/vide o1 --device /dev/video2 --device /dev/video3 --device /dev/video4 --device /dev/video5 --device /dev/i2c-0 --device /dev/i2c-1 --device /dev/i2c-2 --device /dev/i2c-3 --device /dev/i2c-4 --device /dev/i2c-5 --device /dev/i2c-6 --device /dev/i2c-7 --device /dev/i2c-8 --d evice /dev/i2c-9 -v /run/jtop.sock:/run/jtop.sock --workdir=/opt/text-generation-webui dustynv/text-generation-webui:r35.4.1-cp310 pyth on3 server.py --listen --model-dir /data/models/text-generation-webui --model TheBloke_llava-v1.5-13B-GPTQ --multimodal-pipeline llava- v1.5-13b --loader autogptq --disable_exllama --verbose
    /usr/local/lib/python3.10/dist-packages/transformers/utils/hub.py:124: FutureWarning: Using TRANSFORMERS_CACHE is deprecated and will be removed in v5 of Transformers. Use HF_HOME instead.
    warnings.warn(
    07:55:09-964020 INFO Starting Text generation web UI
    07:55:09-969277 WARNING
    You are potentially exposing the web UI to the entire internet without any access password.
    You can create one with the “–gradio-auth” flag like this:

                       --gradio-auth username:password
    
                       Make sure to replace username:password with your own.
    

07:55:09-971591 INFO Loading settings from “settings.yaml”
07:55:09-976041 INFO Loading “TheBloke_llava-v1.5-13B-GPTQ”
07:55:10-022885 ERROR The model could not be loaded because its checkpoint file in .bin/.pt/.safetensors format could not be
located.
07:55:10-024882 INFO Loading the extension “multimodal”
07:55:11-719097 INFO LLaVA - Loading CLIP from openai/clip-vit-large-patch14-336 as torch.float16 on cuda:0…
07:55:14-307723 INFO LLaVA - Loading projector from liuhaotian/llava-v1.5-13b as torch.float16 on cuda:0…
07:55:14-722383 INFO LLaVA supporting models loaded, took 3.00 seconds
07:55:14-725033 INFO Multimodal: loaded pipeline llava-v1.5-13b from pipelines/llava (LLaVA_v1_5_13B_Pipeline)

Running on local URL: http://0.0.0.0:7860

I’m using AGX orin 32GB and my L4T is R35.1.0with JP v5.0.2.

I had issue first instruction “jetson-containers run --workdir=/opt/text-generation-webui $(autotag text-generation-webui)
python3 download-model.py --output=/data/models/text-generation-webui
TheBloke/llava-v1.5-13B-GPTQ”. it shows model downloaded but I can’t find it. as bellow:

aaa@aaaadmin:~$ jetson-containers run --workdir=/opt/text-generation-webui $(autotag text-generation-webui) \

python3 download-model.py --output=/data/models/text-generation-webui
TheBloke/llava-v1.5-13B-GPTQ
Namespace(disable=[‘’], output=‘/tmp/autotag’, packages=[‘text-generation-webui’], prefer=[‘local’, ‘registry’, ‘build’], quiet=False, user=‘dustynv’, verbose=False)
– L4T_VERSION=35.1.0 JETPACK_VERSION=5.0.2 CUDA_VERSION=11.4
– Finding compatible container image for [‘text-generation-webui’]
dustynv/text-generation-webui:r35.4.1-cp310

  • sudo docker run --runtime nvidia -it --rm --network host --volume /tmp/argus_socket:/tmp/argus_socket --volume /etc/enctune.conf:/etc/enctune.conf --volume /etc/nv_tegra_release:/etc/nv_tegra_release --volume /tmp/nv_jetson_model:/tmp/nv_jetson_model --volume /var/run/dbus:/var/run/dbus --volume /var/run/avahi-daemon/socket:/var/run/avahi-daemon/socket --volume /var/run/docker.sock:/var/run/docker.sock --volume /home/qsiadmin/jetson-containers/data:/data --device /dev/snd --device /dev/bus/usb --device /dev/video0 --device /dev/video1 --device /dev/video2 --device /dev/video3 --device /dev/video4 --device /dev/video5 --device /dev/i2c-0 --device /dev/i2c-1 --device /dev/i2c-2 --device /dev/i2c-3 --device /dev/i2c-4 --device /dev/i2c-5 --device /dev/i2c-6 --device /dev/i2c-7 --device /dev/i2c-8 --device /dev/i2c-9 -v /run/jtop.sock:/run/jtop.sock --workdir=/opt/text-generation-webui dustynv/text-generation-webui:r35.4.1-cp310 python3 download-model.py --output=/data/models/text-generation-webui TheBloke/llava-v1.5-13B-GPTQ
    Downloading the model to /data/models/text-generation-webui
    aaa@aaaadmin:~$ ls -ll jetson-containers/data/models/text-generation-webui/
    total 11075212
    -rw-r–r-- 1 root root 1905 九 2 14:20 config.json
    -rw-r–r-- 1 root root 154 九 2 14:20 generation_config.json
    -rw-r–r-- 1 root root 288 九 2 16:05 huggingface-metadata.txt
    -rw-r–r-- 1 root root 7020 九 2 14:20 LICENSE.txt
    -rw-r–r-- 1 root root 4081004224 九 2 11:39 llama-2-7b-chat.Q4_K_M.gguf
    -rw-r–r-- 1 root root 7259435072 九 2 14:30 model.safetensors
    -rw-r–r-- 1 root root 134 九 2 14:20 quantize_config.json
    -rw-r–r-- 1 root root 19579 九 2 14:20 README.md
    -rw-r–r-- 1 root root 438 九 2 14:20 special_tokens_map.json
    -rw-r–r-- 1 root root 748 九 2 14:20 tokenizer_config.json
    -rw-r–r-- 1 root root 499723 九 2 14:21 tokenizer.model
    -rw-r–r-- 1 root root 4766 九 2 14:20 USE_POLICY.md
    aaa@aaaadmin:~$

Hi,

Could you flash JetPack 6 and try it again?
Thanks.

Due to some reason, I have to try it on JP5. May I know the reason why to try it on JP6?

Hi,

Jetson AI Lab is based on JetPack 6 so it will be better to run it on the JetPack 6.

Based on the log you shared, could you search the entire folder to see if the model is downloaded to other paths?

Thanks

I tried again and upgrade L4T to Jetson linux R36.3 with JetPack6.0. but still failed to run. I can pull required images but just failed to loading the model. detailed log as bellow:

qsiadmin@ubuntu:~$ jetson-containers run $(autotag stable-diffusion-webui)
Namespace(packages=[‘stable-diffusion-webui’], prefer=[‘local’, ‘registry’, ‘build’], disable=[‘’], user=‘dustynv’, output=‘/tmp/autotag’, quiet=False, verbose=False)
– L4T_VERSION=36.3.0 JETPACK_VERSION=6.0 CUDA_VERSION=12.2
– Finding compatible container image for [‘stable-diffusion-webui’]

Found compatible container dustynv/stable-diffusion-webui:r36.2.0 (2024-02-02, 8.9GB) - would you like to pull it? [Y/n] Y
dustynv/stable-diffusion-webui:r36.2.0

  • sudo docker run --runtime nvidia -it --rm --network host --volume /tmp/argus_socket:/tmp/argus_socket --volume /etc/enctune.conf:/etc/enctune.conf --volume /etc/nv_tegra_release:/etc/nv_tegra_release --volume /tmp/nv_jetson_model:/tmp/nv_jetson_model --volume /var/run/dbus:/var/run/dbus --volume /var/run/avahi-daemon/socket:/var/run/avahi-daemon/socket --volume /var/run/docker.sock:/var/run/docker.sock --volume /home/qsiadmin/jetson-containers/data:/data --device /dev/snd --device /dev/bus/usb --device /dev/i2c-0 --device /dev/i2c-1 --device /dev/i2c-2 --device /dev/i2c-3 --device /dev/i2c-4 --device /dev/i2c-5 --device /dev/i2c-6 --device /dev/i2c-7 --device /dev/i2c-8 --device /dev/i2c-9 -v /run/jtop.sock:/run/jtop.sock dustynv/stable-diffusion-webui:r36.2.0
    Unable to find image ‘dustynv/stable-diffusion-webui:r36.2.0’ locally
    r36.2.0: Pulling from dustynv/stable-diffusion-webui
    bfbe77e41a78: Pulling fs layer
    ef8924b3a5a5: Pulling fs layer
    a1644e8aa54d: Pulling fs layer
    61787b9cb382: Waiting
    9eed1314ef49: Waiting
    b20423b6e2ce: Waiting
    a20029456556: Pull complete
    c2b4fe089356: Pull complete
    9f7f75397d9a: Pull complete
    95fe6423d877: Pull complete
    998d899205bf: Pull complete
    e36b078c70b9: Pull complete
    996b3cb613ea: Pull complete
    8484becd05f1: Pull complete
    327b4c452f86: Pull complete
    e68ae1f7f272: Pull complete
    fe49d845af9b: Pull complete
    1443fe09d99f: Pull complete
    53429c90fca6: Pull complete
    371a5f71e467: Pull complete
    137b5e764b3d: Pull complete
    27dfafc2927c: Pull complete
    890266ed231f: Pull complete
    a4c23da899b0: Pull complete
    2104a4c1a8ee: Pull complete
    03efdbaf00b7: Pull complete
    76fba44ddee1: Pull complete
    69d0b1919cbb: Pull complete
    c178a8bfbb22: Pull complete
    8cc9d76941be: Pull complete
    6152e6a48d9c: Pull complete
    673eee54d8b1: Pull complete
    3cca7a36333c: Pull complete
    2f280795aa6f: Pull complete
    e0dec9a37fa8: Pull complete
    c043dbbe7527: Pull complete
    2bb056c77e7d: Pull complete
    620108db8dd7: Pull complete
    014139db0bb7: Pull complete
    f97bb16810e0: Pull complete
    c4cbf4c95073: Pull complete
    01e3f9a7555c: Pull complete
    fced774b2df4: Pull complete
    cbdf0d5b1efe: Pull complete
    c63fbdf9de5d: Pull complete
    aed884d5736b: Pull complete
    73673215b6e5: Pull complete
    ddff7db10bf7: Pull complete
    5499cb80b471: Pull complete
    22b297097c57: Pull complete
    b2b152fa707e: Pull complete
    e9c9240b7968: Pull complete
    022fd4331b37: Pull complete
    5ed0fe170e1c: Pull complete
    121c36bc5778: Pull complete
    f1f74198a753: Pull complete
    f8631e4c3622: Pull complete
    8df846002c58: Pull complete
    baacb0ea5506: Pull complete
    3948db201840: Pull complete
    67df534cabb2: Pull complete
    fae4ef99d1a6: Pull complete
    2d1c1134ae52: Pull complete
    Digest: sha256:1b394259129abd865d5ee9dc4882bfb7bf6890552a5fdcaf0147e075725330eb
    Status: Downloaded newer image for dustynv/stable-diffusion-webui:r36.2.0
    Python 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0]
    Version: v1.7.0
    Commit hash: cf2772fab0af5573da775e7437e6acdca424f26e
    Launching Web UI with arguments: --data=/data/models/stable-diffusion --enable-insecure-extension-access --xformers --listen --port=7860
    Style database not found: /data/models/stable-diffusion/styles.csv
    Downloading: “https://huggingface.co/runwayml/stable-diffusion-v1-5/resolve/main/v1-5-pruned-emaonly.safetensors” to /data/models/stable-diffusion/models/Stable-diffusion/v1-5-pruned-emaonly.safetensors

loading stable diffusion model: FileNotFoundError
Traceback (most recent call last):
File “/usr/lib/python3.10/threading.py”, line 973, in _bootstrap
self._bootstrap_inner()
File “/usr/lib/python3.10/threading.py”, line 1016, in _bootstrap_inner
self.run()
File “/usr/lib/python3.10/threading.py”, line 953, in run
self._target(*self._args, **self._kwargs)
File “/opt/stable-diffusion-webui/modules/initialize.py”, line 147, in load_model
shared.sd_model # noqa: B018
File “/opt/stable-diffusion-webui/modules/shared_items.py”, line 128, in sd_model
return modules.sd_models.model_data.get_sd_model()
File “/opt/stable-diffusion-webui/modules/sd_models.py”, line 531, in get_sd_model
load_model()
File “/opt/stable-diffusion-webui/modules/sd_models.py”, line 602, in load_model
checkpoint_info = checkpoint_info or select_checkpoint()
File “/opt/stable-diffusion-webui/modules/sd_models.py”, line 224, in select_checkpoint
raise FileNotFoundError(error_message)
FileNotFoundError: No checkpoints found. When searching for checkpoints, looked at:

  • file /opt/stable-diffusion-webui/model.ckpt
  • directory /data/models/stable-diffusion/models/Stable-diffusionCan’t run without a checkpoint. Find and place a .ckpt or .safetensors file into any of those locations.

Stable diffusion model failed to load
Applying attention optimization: xformers… done.
/opt/stable-diffusion-webui/extensions-builtin/stable-diffusion-webui-tensorrt/ui_trt.py:64: GradioDeprecationWarning: The style method is deprecated. Please set these arguments in the constructor instead.
with gr.Row().style(equal_height=False):
Running on local URL: http://0.0.0.0:7860

Hi,

The stable diffusion model v1.5 is somehow removed. Please try 2.0 or 3.0 instead.

For more details, please find below link:

Thanks.

Could you share more information to change the model to 2.0? just download model from huggingfface and replace it to /data/models/stable-diffusion/models/Stable-diffusion/?
thanks.

Hi,

Please try the patch below:

Thanks