NVIDIA Jetson Generative AI Lab is great, but only able to run services on local host

Robotics & Edge Computing Jetson Systems Jetson AGX Orin
1

There is some great tutorials on this text-generation-webui - NVIDIA Jetson Generative AI Lab

I have used oobabooga’s text-generaton-webui on PC before and manage to share the service on LAN using the --share option:
python server.py --model mosaicml_-7b-chat --share --trust-remote-code --chat

But following the instruction on text-generation-webui - NVIDIA Jetson Generative AI Lab

cd jetson-containers
./run.sh $(./autotag text-generation-webui)

The webpage is working and in the terminal I see:
Running on local URL: http://0.0.0.0:7860

But the service is not available on LAN

The ufw firewall is disabled

Any advice?

2

Hi @pederhomelab, you can supply your own launch command to text-generation-webui when you start the container like this:

./run.sh --workdir=/opt/text-generation-webui $(./autotag text-generation-webui) \
  python3 server.py --model-dir=/data/models/text-generation-webui --listen --verbose

By default, it should already be accessible over LAN because --listen is included in the default RUN command, and indeed Running on local URL: http://0.0.0.0:7860 shows that it is listening for connections on all network interfaces (that’s what 0.0.0.0 means)

Do you have a firewall on your PC side that’s blocking it? Can you view the webpage from the Jetson itself?