I am trying this tutorial: LLaVA - NVIDIA Jetson AI Lab
Using a Xavier AGX.
Downloading the model works fine:
pit@pit-desktop:~/jetson-containers$ jetson-containers run --workdir=/opt/text-generation-webui $(autotag text-generation-webui) \ > python3 download-model.py --output=/data/models/text-generation-webui \ > TheBloke/llava-v1.5-13B-GPTQ Namespace(disable=[''], output='/tmp/autotag', packages=['text-generation-webui'], prefer=['local', 'registry', 'build'], quiet=False, user='dustynv', verbose=False) -- L4T_VERSION=35.6.0 JETPACK_VERSION=5.1 CUDA_VERSION=12.4 -- Finding compatible container image for ['text-generation-webui'] dustynv/text-generation-webui:r35.4.1-cp310 V4L2_DEVICES: ### DISPLAY environmental variable is already set: ":0" localuser:root being added to access control list + sudo docker run --runtime nvidia -it --rm --network host --shm-size=8g --volume /tmp/argus_socket:/tmp/argus_socket --volume /etc/enctune.conf:/etc/enctune.conf --volume /etc/nv_tegra_release:/etc/nv_tegra_release --volume /tmp/nv_jetson_model:/tmp/nv_jetson_model --volume /var/run/dbus:/var/run/dbus --volume /var/run/avahi-daemon/socket:/var/run/avahi-daemon/socket --volume /var/run/docker.sock:/var/run/docker.sock --volume /home/pit/jetson-containers/data:/data -v /etc/localtime:/etc/localtime:ro -v /etc/timezone:/etc/timezone:ro --device /dev/snd -e PULSE_SERVER=unix:/run/user/1000/pulse/native -v /run/user/1000/pulse:/run/user/1000/pulse --device /dev/bus/usb -e DISPLAY=:0 -v /tmp/.X11-unix/:/tmp/.X11-unix -v /tmp/.docker.xauth:/tmp/.docker.xauth -e XAUTHORITY=/tmp/.docker.xauth --device /dev/i2c-0 --device /dev/i2c-1 --device /dev/i2c-2 --device /dev/i2c-3 --device /dev/i2c-4 --device /dev/i2c-5 --device /dev/i2c-6 --device /dev/i2c-7 --device /dev/i2c-8 -v /run/jtop.sock:/run/jtop.sock --name jetson_container_20250304_183126 --workdir=/opt/text-generation-webui dustynv/text-generation-webui:r35.4.1-cp310 python3 download-model.py --output=/data/models/text-generation-webui TheBloke/llava-v1.5-13B-GPTQ Downloading the model to /data/models/text-generation-webui
### Start Web UI with Multimodal Extension
pit@pit-desktop:~/jetson-containers$ jetson-containers run --workdir=/opt/text-generation-webui $(autotag text-generation-webui) python3 server.py --listen --model-dir ~/data/models/text-generation-webui --model TheBloke_llava-v1.5-13B-GPTQ --multimodal-pipeline llava-v1.5-13b --loader autogptq --disable_exllama --verbose
Namespace(disable=[''], output='/tmp/autotag', packages=['text-generation-webui'], prefer=['local', 'registry', 'build'], quiet=False, user='dustynv', verbose=False)
-- L4T_VERSION=35.6.0 JETPACK_VERSION=5.1 CUDA_VERSION=12.4
-- Finding compatible container image for ['text-generation-webui']
dustynv/text-generation-webui:r35.4.1-cp310
V4L2_DEVICES:
### DISPLAY environmental variable is already set: ":0"
localuser:root being added to access control list
+ sudo docker run --runtime nvidia -it --rm --network host --shm-size=8g --volume /tmp/argus_socket:/tmp/argus_socket --volume /etc/enctune.conf:/etc/enctune.conf --volume /etc/nv_tegra_release:/etc/nv_tegra_release --volume /tmp/nv_jetson_model:/tmp/nv_jetson_model --volume /var/run/dbus:/var/run/dbus --volume /var/run/avahi-daemon/socket:/var/run/avahi-daemon/socket --volume /var/run/docker.sock:/var/run/docker.sock --volume /home/pit/jetson-containers/data:/data -v /etc/localtime:/etc/localtime:ro -v /etc/timezone:/etc/timezone:ro --device /dev/snd -e PULSE_SERVER=unix:/run/user/1000/pulse/native -v /run/user/1000/pulse:/run/user/1000/pulse --device /dev/bus/usb -e DISPLAY=:0 -v /tmp/.X11-unix/:/tmp/.X11-unix -v /tmp/.docker.xauth:/tmp/.docker.xauth -e XAUTHORITY=/tmp/.docker.xauth --device /dev/i2c-0 --device /dev/i2c-1 --device /dev/i2c-2 --device /dev/i2c-3 --device /dev/i2c-4 --device /dev/i2c-5 --device /dev/i2c-6 --device /dev/i2c-7 --device /dev/i2c-8 -v /run/jtop.sock:/run/jtop.sock --name jetson_container_20250304_183539 --workdir=/opt/text-generation-webui dustynv/text-generation-webui:r35.4.1-cp310 python3 server.py --listen --model-dir /home/pit/data/models/text-generation-webui --model TheBloke_llava-v1.5-13B-GPTQ --multimodal-pipeline llava-v1.5-13b --loader autogptq --disable_exllama --verbose
/usr/local/lib/python3.10/dist-packages/transformers/utils/hub.py:124: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead.
warnings.warn(
18:35:55-020871 INFO Starting Text generation web UI
18:35:55-032096 WARNING
You are potentially exposing the web UI to the entire
internet without any access password.
You can create one with the "--gradio-auth" flag like
this:
--gradio-auth username:password
Make sure to replace username:password with your own.
18:35:55-039757 INFO Loading settings from "settings.yaml"
NOW THE ERROR:
18:35:55-050759 INFO Loading "TheBloke_llava-v1.5-13B-GPTQ"
18:35:55-145301 ERROR The model could not be loaded because its checkpoint
file in .bin/.pt/.safetensors format could not be
located.
18:35:55-149351 INFO Loading the extension "multimodal"
18:35:58-281075 INFO LLaVA - Loading CLIP from
openai/clip-vit-large-patch14-336 as torch.float16 on
cuda:0...
18:36:02-952300 INFO LLaVA - Loading projector from
liuhaotian/llava-v1.5-13b as torch.float16 on cuda:0...
18:36:03-332203 INFO LLaVA supporting models loaded, took 5.05 seconds
18:36:03-336693 INFO Multimodal: loaded pipeline llava-v1.5-13b from
pipelines/llava (LLaVA_v1_5_13B_Pipeline)
Running on local URL: http://0.0.0.0:7860
There was a similar problem here:
dusty_nvModerator
Hi @michtw , I recall in the oobabooga webui, there is a “use safetensors” option in the model loader section, so it may need a similar flag if done from the command line. You do not need jp5 git branch of jetson-containers anymore, that was from long ago. Also the Multimodal Chat could still working with JetPack 5, I had recently rebuild the container for it on L4T R35.2.1.
But I can’t wrap my brain around “use safetensors”.
Any hint appreciated.
-Pete