Description
I’ve created a web based GUI to manage inference and run a OpenAI endpoint on the spark. It uses llama.cpp or transformers and makes it easy to develop against the spark and or thor. I’m using this to power a few robots and local chat.
athkumar
3
Hii @matthew.rogers ,
Excellent Work ! Thanks for posting to the community.
I highly recommend posting this here as well for wider community engagement: DGX Spark / GB10 Projects - NVIDIA Developer Forums
Well Done !
Related topics
| Topic | Replies | Views | Activity | |
|---|---|---|---|---|
| Web Based Inference GUI Server for Easy Model Deployment on the Spark | 2 | 250 | February 17, 2026 | |
| Optimizing Inference on Large Language Models with NVIDIA TensorRT-LLM, Now Publicly Available | 8 | 2135 | January 25, 2024 | |
| Support tensorrt llm and tensorrt llm on triton inference server backend | 1 | 147 | June 30, 2025 | |
| Easier. Faster. Open. TensorRT LLM 1.0 | 0 | 139 | September 25, 2025 | |
| Easier. Faster. Open. TensorRT LLM 1.0 is here | 0 | 365 | September 25, 2025 | |
| Model Orchestration and Deployment | 4 | 991 | November 24, 2025 | |
| NVIDIA TensorRT-LLM 및 NVIDIA Triton Inference Server로 Meta Llama 3 성능 강화 | 0 | 394 | May 3, 2024 | |
| How to serve TensorRT-LLM engines with Triton Inference Server on Jetson Thor and compare inference speed with vLLM container? | 5 | 469 | February 2, 2026 | |
| Thor Inference Server with webGUI | 2 | 152 | February 16, 2026 | |
| Is it possible to deploy the Llama-70b model with TensorRT LLM on an L40S GPU? | 2 | 751 | May 30, 2024 |