I am really looking forward to benchmarking multimodal models on the DGX Spark. I am really curious as to how it compares to the same model running on a RTX 5090 with 32 GB of memory and an RTX 6000 Pro with 96GB. It gives me an excuse to buy the RTX cards. (If I can find them for a reasonable price, or get Nvidia to send them to me.)
I too am waiting for the DGX Spark like the rest of you. My plan was to replace my aging i5 desktop. When I placed the reservation, I really didn’t think we’d be waiting until Christmas though. They probably could have just waited until next year’s conference to announce these. Anyway, in the meantime I picked up a pair of RTX Pro 6000 Blackwell Workstation Edition. They’re strictly running LLM (not on my desktop), mostly under Ollama or llama.cpp. Which models are you interested in, maybe I can give you some quick stats using 1 or both 6000.
Hi Aaron, yeah if you have time can you benchmark dots.OCR? pull the non quantized version from huggingface and compare it to any other RTX that you have and the RTX 6000 pro. You should be able to run it all in VRAM on pretty much anything. and have it process a PDF form. I just search for forms and pull the I-131 form from online because it has a bunch of fields on it.
Thanks!
I tried for a few hours to get it running Garry, but was unsuccessful. I tried bringing it up using github with no luck. Then tried using their docker image, which doesn’t appear to be compatible with Blackwell GPU. I get the error
RuntimeError: CUDA error: no kernel image is available for execution on the device
I’ll give it another shot over the weekend. It’s unfortunate so many projects have not yet made their pre-built containers compatible yet. Blackwell GPUs have been in the wild for 8 months now.
Thanks Aaron, That is really interesting results. I didn’t think there was anything special about dots.ocr. That runtime error almost sounds like it couldn’t find the cuda library or resources.
if your model fits in 32GB incl. context, than this is very likely the best performance/price option.
RTX PRO 6000 should only be ~10% faster than RTX5090, so would only pay 4x if you REALLY need the memory.
Some bechmark here testing:
NVIDIA DGX Spark
NVIDIA RTX PRO™ 6000 Blackwell Workstation Edition
NVIDIA GeForce RTX 5090 Founders Edition
NVIDIA GeForce RTX 5080 Founders Edition
Apple Mac Studio (M1 Max, 64 GB unified memory)
Apple Mac Mini (M4 Pro, 24 GB unified memory)
From NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference | LMSYS Org