Inference Stack Choice for Mixed Jetson Orin + DGX Spark Environment (Ollama vs vLLM?)

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

I’ve been experimenting with LLM inference for some time and gradually expanded my hardware footprint. What started as a single Jetson Orin NX 16GB turned into:

  • 2× Jetson Orin NX 16GB

  • 1× Jetson Orin AGX 64GB

  • 1× DGX Spark 128GB (recent addition)

My use case is inference only (no training).

On the software side, we’ve been running everything bare metal with minimal overhead and built our own lightweight “LLM traffic manager.” Applications send requests to this layer, which routes inference calls across the available machines based on capacity and model constraints. This setup has worked very well so far.

However, now that we’ve added the DGX Spark, I’m questioning whether Ollama is still the right runtime, especially for that class of hardware.

From what I’ve read, Ollama seems well suited for Jetson-class devices (simplicity, easy model management, quick deployment). But for DGX-class GPUs, it appears that frameworks like vLLM (or possibly NVIDIA-optimized stacks) might better exploit memory bandwidth, tensor parallelism, batching, and overall throughput.

So I’d appreciate input from people who have experience with mixed Jetson + DGX environments:

  1. Is Ollama a reasonable choice on DGX Spark for high-performance inference?

  2. Would you recommend moving to vLLM (or something else) on DGX specifically?

  3. If so, would it make sense to standardize the entire fleet (Jetsons included) on a different runtime?

  4. Are there NVIDIA-optimized inference stacks that would be a better long-term direction?

I’m especially interested in practical tradeoffs: performance, GPU utilization, batching efficiency, operational complexity, and maintainability. Thanks in advance for any help.

3

Ollama is not a good choice for anything right now, better use llama.cpp directly (the runtime ollama was built upon).

For DGX Spark, it’s almost always better to use vLLM. Here is a simple decision tree:

image
image2380×984 173 KB

4

I have a Jetson Orin Nano Super available. It’s far less performant than the Spark. Nonetheless, it is usable to run a quantized ~3-4B model for occasional domain expertise in multi-agent workflows.

The best path is likely compiling llama.cpp with CUDA enabled from scratch on the device. I found this post helpful: Reddit - The heart of the internet

Edit: Also suggest cross posting to the Jetson forums. There are vLLM containers from Nvidia for IOT and other insights over there. Pick the appropriate forum and search for vLLM or llama.cpp or TensorRT-LLM; for example Performance Inquiry: Optimizing Qwen3-VL 2B Inference for 2 QPS Target on Orin Nano Super

5

For DGX Spark, it’s almost always better to use vLLM. Here is a simple decision tree

Any preference for multi-modals models such as Qwen3-VL?

6

It depends, I guess I’d follow pretty much the same path.

For my specific use cases, I prefer llama.cpp to run Qwen3-VL-8B, but only because I don’t want to keep it loaded in memory all the time, but when I do need it, I want it to load very quickly. I use llama-swap to load models on demand, including ones served by vLLM.

7

ollama should really not be in the cards in 2026 - stick with llama.cpp, vllm, and SGLang for your inferencing needs.

ollama is just too far behind.