Proxmox VE on DGX Spark

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

Are we forced to use the DGX OS on DGX Spark as host; or we can install things like a Proxmox VE (arm64 version) without manually hunting drivers around?

2

Proxmox VE lack offical ARM64 support, so even if I’m wondering the same, I do believe it’s better don’t mess around the OS trying to run Proxmox VE on this kind of machine.

3

Hi Wingenlit, DGX OS is the tested and supported operating system for Spark. You’re welcome to try other packages, but if you need support or hardware triage later, you’ll likely need to revert to DGX OS so we have a known baseline.

4

@wingenlit Pay everyone else no mind. It won’t be as seamless as the official support x86 Proxmox, but you can most definitely install something like GitHub - jiangcuo/pxvirt: A fork of Proxmox VE for ARM and LoongArch architectures , then run ARM64 VMs – even a DGX OS VM.

Before you take that journey, you should see if the pre-installed DGX OS can fulfill your needs. I have not used it, but it is a fork of Ubuntu, so I imagine there is native virtualization functionality, such as Canonical’s LXD (Ubuntu’s play on LXC, such as in Proxmox), or something similar.

Regardless, just about anything is doable on a Linux OS., it’s up to you to decide if it’s worth the inevitable pain in your butt it’s going to be to achieve it.

5

Like @fakebizprez said, there is an ARM64-based fork of Proxmox that you can run, but it’s a bit tedious to set up (I found). However, there is another project that makes this a lot easier, namely dockur/proxmox. Running this via Docker (which is by default installed in DGX OS 7) you can get yourself a running instance of Proxmox VE since it automatically uses the pxvirt fork under the hood when it is ran on ARM-based architectures.

An example Docker Compose I’ve used (with some extra config to get DHCP to work in our subnet)

services:
  proxmox:
    hostname: pve-dgxspark-01
    image: dockurr/proxmox
    container_name: proxmox
    environment:
      PASSWORD: "changeme123"
      NETWORK: "N"
    ports:
      - 8006:8006
    devices:
      - /dev/kvm
      - /dev/net/tun
      - /dev/vhost-net
    device_cgroup_rules:
      - 'c *:* rwm'
    cap_add:
      - NET_ADMIN
    volumes:
      - ./data:/var/lib/vz
      - ./config:/var/lib/pve-cluster
    configs:
      - source: interfaces
        target: /etc/network/interfaces
    networks:
      vlan:
        ipv4_address: 10.100.100.201
    dns:
      - 10.100.100.1
    restart: always
    privileged: true
    stop_grace_period: 2m

configs:
  interfaces:
    content: |
      auto lo
      iface lo inet loopback

      auto eth0
      iface eth0 inet manual

      auto vmbr0
      iface vmbr0 inet static
          address 10.100.100.201/24
          gateway 10.100.100.1
          bridge-ports eth0
          bridge-stp off
          bridge-fd 0

      source /etc/network/interfaces.d/*

networks:
  vlan:
    external: true

# docker network create -d macvlan \
#     --subnet=10.100.100.0/24 \
#     --gateway=10.100.100.3 \
#     --ip-range=10.100.100.201/29 \
#     -o parent=br0 vlan

Though, like @fakebizprez and @NVES warned; these things aren’t declared to be supported in any way and I’ve had issues getting virtual machines to boot. My gut feeling is that there is a lot of UEFI shenanigans on ARM that doesn’t work / virtualizes well.