LLaMa 2 LLMs w/ NVIDIA Jetson and textgeneration-web-ui

Hi, @dusty_nv, LlamaSpeak project is cool and fast. Am I correct in saying that it can also run on x86 standalone PCs with Nvidia graphics card? This project has a bright future, especially it can be useful for people with visual impairments.

Here are the two optimizations I performed on my NVIDIA Jetson AGX Orin 64GB Developer Kit:

  • I disabled the graphical interface to free up RAM memory.
  • I turned on the MAXN mode.

See the video below

The usage of RAM

Thank you!