Hi,
I am the owner of a NVIDIA DGX Spark system and have been working intensively with it for two weeks. I encountered a significant issue with the missing official native ARM64 NIM images for essential AI models such as Llama 2 70b. Because of this, the basic setup was not feasible without complicated workarounds.
The marketing and communications around DGX Spark raised high expectations for immediate use of local AI models on ARM architectures. However, the current software support appears incomplete, which created some confusion and delays.
Could someone please clarify the current status and expected timeline for full native ARM64 NIM image support? Any guidance or updates would be greatly appreciated.
Thank you in advance for your help.
Best Peter
It looks like the NIM is just for the smaller models – but you should be able to use the TRT-LLM playbook? What about just running LMStudio with Llama 70B?
Hi Alan and NVIDIA Team,
Thank you for your response. However, your suggestion to use the TRT-LLM playbook or LMStudio for Llama 70B essentially means relying on complex, manual workarounds and community-driven improvisation instead of having a ready-to-use solution.
My main question remains this. When will NVIDIA provide a fully tested, stable, and officially supported native ARM64 NIM image for large models such as Llama 2 70B that can be used without prolonged experimentation or trial and error?
It is essential for users and developers to know this timeline to plan their projects and investments properly. The current situation where highly marketed hardware lacks fully seamless native software support is causing frustration and wasted time.
I kindly ask NVIDIA to provide clearer information or a roadmap for making these essential software images available on ARM64 hardware.
Thank you for your attention. I look forward to your detailed answer.
Best Peter
Think the problem is that if you bought a spark, you’re using it to test real cloud distributed inferencing model or you’re using it for fine-tuning smaller LLMs or other ML models and deploy them to cloud later in your experiment phase, reducing your cloud costs. So, LMStudio is not in the picture and SGLang, TRT-LLM and vLLM are.
Do they have a consolidated list anywhere on what NIM’s are currently supported on ARM/DGX spark? Have been struggling to find which models are actually supported
@Nvidia Team,
I’m currently using a cluster of two DGX Sparks, allowing me to use up to 256GB of memory.
For those using two DGX Sparks, please also deploy AI models with large parameter counts for DGX Spark.
@Nvidia team how do I download nvfp4 version of GLM-5?