Cheaper way to go 1T vram(170hx + 4 sparks)

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
,
1

I am not sure if I can talk about this on Nvidia forum, but at least I do not see rule or announcement about not to, and saw other Nvidia linux forum talk about this unlocked cards, but no mention on this forum.

My intention is sharing info that there is some side step to achieve extra missing (alot of ) Vram that 2 x sparks or 4 x sparks to run upcoming getting bigger llm models, which is current trend.

So, basically getting 8 x Nvidia CMP 170hx 64GB HBM2e unlockable crytomining card. Well, fully going 16 cards is another option but this is Spark forum and I already invested 4 sparks + switch, then I come up with mixing them together. (and probably getting 1 or 2 cards for dense model or sub 120B model would be great option since it has 1.49T/s memory bandwidth)

I have bought 10 cards and paying about 1100 USD per piece ( I had been hesitated for a week and pulled trigger, so I was late to this party) and still in the middle of shipping.

I was planning to benchmark but I found recent one by done someone on GLM5.2 4 bit.
At the moment, 8 x 170hx cards can do 30 tg/s, 2600+ pp/s even with pcie 1 x 4 condition. Yet you can get pcie 2 x 16 max, but there is still chance to see 3 x 16 or more later.

So my plan is simple, 478ish UMA + 471 Vram

4 sparks - mikrotic switch - qsfp - 8 x 170hx single server

I do not have any expertise on how to optimize but later later I will just let Sol does job, I guess.

But on 5.6 pro’s simulation based on our forum’s benchmark and 170hx’s benchmark, 1.13T model (virtual 1.5x GLM5.2 but A48B) is fine.
like.. 128k ctx, c1: 1600 pp/s, 17 tg/s, c2: 2700 pp/s, 28 tg/s
1.49T model (virtual 2 x GLM5.2 but A56B) is barely doable TG/s.
like.. 128k ctx, c1: 1270 pp/s, 13 tg/s, c2: 2100 pp/s, 21 tg/s .
It’s just assumption + simulation, so nothing promising.

Nvlink might not be implemented even later, but hoping for TP possibility by faster pcie speed. then concurrency might be less problem.

So I am kinda hoping + gambling on it, but I see it’s about half price of adding another 4 sparks with trade off power consumption + less concurrency + noise vs speed + cheaper price.

original benchmark text below.

# CMP 170HX (unlocked, 64GB) — LLM + Diffusion Benchmark Results

8× unlocked 170HX, driver 610.43.02, sm_*80 (GA100), PCIe **Gen1 x4 (~1 GB/s)**, stock (no cap mod / no Gen2). Rented, July 2026.

---

## Raw compute (per-card, torch GEMM)

| precision | TFLOPS/TOPS | note |
|---|---|---|
| FP16 | **162.7** | tensor, unlocked |
| BF16 | **171.4** | tensor, unlocked |
| TF32 | 79.0 | tensor |
| FP32 | 12.6 | FMA restored (locked = 0.39) |
| FP64 | 11.6 | |
| **INT8**|**44.1 TOPS**|**STILL GATED** (3.7× slower than FP16; A100 = 2× faster) |
| HBM bandwidth | **1600 GB/s** | measured, above 1493 datasheet |
| VRAM integrity | 8/8 PASS | full byte-compare |

Per-card spread <2.5% across 8 cards. INT8/IMMA gated → **use W4A16, not W8A8**.

---

## GLM-5.2 (744B, 40B active, glm_moe_dsa) — THE headline

### vLLM + `TRITON_MLA_SPARSE` (PR #38476), PP8, W4A16 — **WORKS**
| ctx | prefill t/s |
|---|---|
| 4k | 665 |
| 32k | 1,497 |
| 65k | 2,342 |
| **131k**|**2,675** |

- **Decode (PP8, no MTP): 30.2 t/s**
- **KV capacity @0.92 util: 438,107 tokens** (BF16 KV; ~88–100 KB/token, MLA)
- Model load: ~440–620 s
- 315k prefill: **~2 min (extrapolated from 131k, not directly measured)**
- Prefill RISES with context (chunked prefill + sparse attn)

### llama.cpp (Q4_K_S GGUF, 8 cards, `-sm layer`) — **UNUSABLE**
| ctx | prefill t/s |
|---|---|
| 512 | 141 |
| 4k | 162 |
| 16k | 124 (degrades) |

- Decode: 17.2 t/s
- Cause: **no DSA — dense-attention fallback** (llama.cpp issue #24730). ~42 min for 315k prompt.
- `-sm layer` runs 1 GPU at a time (no chunked-prefill pipelining).

### TP8 (QuantTrio Int4-Int8Mix, enforce*_eager) — **BAD**
- Prefill 4k/16k/32k: 382 / 435 / 629 t/s (**~4× worse than PP8**)
- Decode: 3.4 t/s (eager) — OR crashes in CUDA-graph capture (vLLM #48285)
- MTP=ON: OOM

**GLM-5.2 verdict: PP8 only. PP prefill excellent, decode 30 t/s, no MTP.**

---

## Qwen3-Coder-480B-A35B (35B active, GQA) — vLLM PP, AWQ
| ctx | prefill t/s |
|---|---|
| 1k | 745 |
| 4k | 1,511 |
| 16k | 1,853 |
| 32k | 2,452 |

- Decode: **46.6 t/s** (untuned fused_*moe, no MTP). Standard GQA → runs on stock vLLM, no patched backend.

---

## Qwen2.5-72B dense (AWQ) — TP vs PP
| config | pp1k | pp4k | pp16k | decode |
|---|---|---|---|---|
| 1 card | 839 | 1,092 | 960 | 27.3 |
| PP2 | 829 | 1,084 | 1,167 | 29.1 |
| **TP2**|**316**|**420**|**416**|**33.7** |

**TP wrecks prefill (2.3–2.8× worse) for +23% decode. On Gen1 x4, PP mandatory.**

---

## Diffusion — 170HX vs RTX 3090 (identical script)
| test | 170HX | RTX 3090 | 170HX advantage |
|---|---|---|---|
| SDXL 1024² 30-step | 4.73 s (6.35 it/s), 10.5 GB | 7.59 s (3.95 it/s) | **1.60×** |
| Wan2.1-T2V 81f@480p | 73.4 s (0.91 s/f), 18.5 GB | 132.8 s (1.64 s/f) | **1.81×** |
| LTX-Video 81f | 11.0 s (0.14 s/f), 15.9 GB | 20.0 s (0.25 s/f) | **1.82×** |
| Wan2.1 129f@720p | 1,485 s, **33.3 GB**|**OOM (24 GB)** | runs vs can't |

---

## Best quants (sm_80)
- **USE:** W4A16 / AWQ / GPTQ (INT4 weights + BF16 activations → unlocked fast path)
- **AVOID:** W8A8 / INT8 (gated IMMA, 44 TOPS), FP8 / NVFP4 (need sm_89+/sm_120 tensor cores)
- **GLM-5.2 on vLLM: SYMMETRIC quant REQUIRED** (MoE kernels reject asymmetric):
- ✅ `lowbitcoffee/GLM-5.2-W4A16` (sym g128, 388 GB)
- ✅ `QuantTrio/GLM-5.2-Int4-Int8Mix`
- ❌ `cyankiwi/GLM-5.2-AWQ-INT4` (asym g32 — **FAILS**, and it's the one most guides cite)
- **KV: BF16 only** — FP8 KV unsupported by TRITON_MLA_SPARSE on sm_80

---

## Working GLM-5.2 vLLM recipe
```
pip install vllm==0.20.2 # release, for compiled* _C
# apply PR #38476 python files as a DIFF onto site-packages
**# (VLLM_USE_PRECOMPILED editable install ships NO vllm._C — will fail)
pip install -U transformers # need 5.x; 4.57 doesn't know glm_moe_dsa
export VLLM_ATTENTION_BACKEND=TRITON_MLA_SPARSE
export VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0
vllm ... --pipeline-parallel-size 8 --gpu-memory-utilization 0.90
# --block-size 64 auto-set by DEEPSEEK_V32_INDEXER
# max_model_*len MUST fit KV capacity or engine won't boot
```

---

## MTP
- MTP-head checkpoints **refuse PP** ("not SupportsPP") → MTP needs TP.
- TP8 crashes graph capture (#48285) → `enforce*_eager` → decode 3.4 t/s.**
- **No usable MTP on PP today.** General vLLM bug, fix in RFC #44697 (unmerged). If it lands: ~1.7× decode (30→~50 t/s) on PP8, no hardware mod.

---

## Stability
- **0 hard faults** across full session on a healthy host (hundreds of 60 s health samples).
- Transient "GPU requires reset" only at `gpu-mem-util 0.95` + huge ctx — **self-recovered**. Keep util ≤ 0.90.
- Repeatedly hard-killing multi-GPU jobs → ~32 zombie CUDA procs → **host CUDA runtime wedges** (`cuInit`=999 for all frameworks while nvidia-smi stays healthy). Unfixable inside a container; needs host reboot. Don't `kill -9` live multi-GPU jobs.
- Peak 60 °C @ 254 W under load.

---

## One-line summary
GLM-5.2 (744B DSA) **runs on 8× unlocked 170HX via vLLM PR #38476**: **2,675 t/s prefill @131k, 30 t/s decode, 438k ctx, 0 faults** — PP only, W4A16 symmetric quant, no MTP yet. Prefill is A100-class; decode is bandwidth-limited by PP. INT8 stays gated (use W4A16). Diffusion beats a 3090 by 1.6–1.8× with 64 GB headroom.

3

If rumors are correct, the cards have more memory, than officially reported.

Could it be we have similar software locks on DGX Sparks hardware? Like those highly anticipated tcgen05 kernels?

4

Was $200 before the hack was announced. :-D , $600 the day that it was. :-)

5

yep, the party closed early. I have heard many buyers who paid 500~600USD on Alibaba at that time, got message to pay double or cancel by sellers. So not many ppl actually got in that price range.

6

Could it be we have similar software locks on DGX Sparks hardware? Like those highly anticipated tcgen05 kernels?

According to the information online, the 170HX were ‘crippled’ A100 cards (probably cards that didn’t pass hardware checks). Makes sense they’d want to recoup the cost by rebadging and sell them for crypto, where VRAM wasn’t a concern rather than destroy them.
You’re still gambling on hitting the ‘silicon lottery’ as far as to how much of the HBM in each 170HX will actually work properly when unlocked.

Unfortunately, that’s definitely not the case in our Sparks; we definitely do not have extra ‘locked’ memory in our systems.

7

averaging $1300 on ebay or more now

8

It is more about the part, that was flying around for months before:

They have very particular trust problem and it was not properly addressed.

9

not really useful for LLM the PCI-e is too limited a V100 runs circles around them

these were $150 before the unlock went publlic …

128GB-Class Inference Comparison: V100 vs CMP 170HX vs A100 vs Ryzen AI Max+ 395 vs NVIDIA DGX Spark (Trimmed)

Excluded: CMP 170HX 40GB and 80GB modes. Only stable configurations are listed.

Primary Comparison Table

Platform Memory FP32 est 8B Q4 tok/s 30B/34B Q4 tok/s 70B Q4 tok/s PCIe / host bandwidth
Ryzen AI Max+ 395 128GB unified ~59.4 TFLOPS 37-47 66-72 4.5-5.0 16x PCIe Gen4, ~64 GB/s
V100 32GB x4 128GB VRAM ~56 TFLOPS ~104-108 - ~16.9 16x PCIe Gen3 each, ~16 GB/s each
A100 PCIe 80GB 80GB VRAM ~19.5 TFLOPS ~138 - ~22.1 16x PCIe Gen4, ~64 GB/s Best single-card NVIDIA balance of speed and capacity
A100 SXM 80GB 80GB VRAM ~19.5 TFLOPS ~133 - ~24.3 NVLink / SXM, ~600+ GB/s
CMP 170HX 8GB→64GB 64GB HBM2e ~6.3 TFLOPS unlocked ~12-18 est ~8-12 est ~2.5-4.0 est PCIe Gen1 x4 shipped, ~1 GB/s; Gen2 x16 mod possible, ~4 GB/s
NVIDIA DGX Spark 128GB unified ~1 PFLOPS FP4 est ~55-75 est ~28-38 est ~8-12 est Unified internal, ~128 GB/s

The PCIe Lane Problem

Platform Stock PCIe Practical host bandwidth Model load time for 70B Q4 (~40GB)
V100 32GB x4 16x PCIe Gen3 each ~16 GB/s each ~2.5 seconds
A100 PCIe 80GB 16x PCIe Gen4 ~64 GB/s ~0.6 seconds
NVIDIA DGX Spark Unified internal ~128 GB/s ~0.3 seconds
CMP 170HX 8GB→64GB PCIe Gen1 x4 shipped ~1 GB/s shipped ~40 seconds shipped; ~10 seconds if fully modded

Key constraint: Even a fully modded 170HX has 4-16x less host bandwidth than V100/A100/DGX Spark.

Compute Power

Platform FP32 est FP64 est AI TOPS / est Notes
V100 32GB x4 ~56 TFLOPS ~28 TFLOPS ~480 INT8 TOPS est Aggregate
A100 PCIe 80GB ~19.5 TFLOPS ~9.8 TFLOPS ~624 INT8 TOPS Ampere GA100; 7nm
A100 SXM 80GB ~19.5 TFLOPS ~9.8 TFLOPS ~624 INT8 TOPS Same die; NVLink interconnect
CMP 170HX 8GB→64GB ~6.3 TFLOPS unlocked ~3.2 TFLOPS unlocked ~400-500 INT8 TOPS est GA100 cut-down; HBM2e; unlock restores some SMs
NVIDIA DGX Spark ~1 PFLOPS FP4 est ~15-20 TFLOPS FP64 est ~2000+ INT8/FP4 TOPS GB10 Grace Blackwell; 128GB unified

Power Efficiency (70B Q4)

Platform Typical TDP 70B Q4 tok/s 70B est watts/token
A100 PCIe 80GB 300W ~22.1 ~13.6W
A100 SXM 80GB 400W ~24.3 ~14.1W
NVIDIA DGX Spark ~250W est ~8-12 est ~21-31W est
V100 32GB x4 ~1200W ~16.9 ~71W
CMP 170HX est ~300W est ~2.5-4.0 est ~75-120W est

What Fits Where: 128GB-Class Models

Model Q4 Size V100 x4 128GB A100 80GB x2 160GB DGX Spark 128GB 170HX 64GB
8B dense ~4.6GB Yes Yes Yes Yes
13B dense ~7.5GB Yes Yes Yes Yes
30B dense ~14.5GB Yes Yes Yes Yes
70B dense ~39.6GB Yes Yes Yes Yes
100B MoE ~20-40GB Yes Yes Yes Tight

Real-World Generation Speeds

Platform 8B Q4 30B Q4 70B Q4 Usable?
V100 32GB x4 ~104-108 - ~16.9 Yes, but slow for 70B
A100 PCIe 80GB x2 ~276 - ~44.2 est Yes
NVIDIA DGX Spark ~55-75 est ~28-38 est ~8-12 est Yes
CMP 170HX 8GB→64GB ~12-18 est ~8-12 est ~2.5-4.0 est Marginal
CMP 170HX 8GB→64GB x2 ~20-28 est ~12-16 est ~3-5 est Worse scaling; PCIe bottleneck compounds across cards

Mod References

Important: the x16 width fix is separate from the memory unlock. You need both for best results, and the Gen2 speed unlock is still unreleased as of mid-2026.

Buyer’s Checklist: Used CMP 170HX

  • Verify PCI ID: must be 10de:20c2 for 8GB→64GB unlock
  • Confirm it is NOT 10de:20b0 — that is not a 170HX
  • Check if the card is from an early production batch (pre-laser-cutting that cannot be software unlocked)
  • Verify the unlock tool actually works on this specific card before buying
  1. PCIe Gen1 x4 is a hard bottleneck for LLM inference
  2. V100 32GB x4 gives 128GB aggregate with full x16 PCIe for less total cost
  3. A100 80GB gives better single-card performance for similar or less money
  4. The 170HX has lower raw compute than A100/V100 even when unlocked
  5. No ECC, no display outputs, no official NVIDIA support
  6. The 64GB mode is unproven for 24/7 stability

Bottom Line

The CMP 170HX is an experiment, not a production inference platform. it is a bad value compared to V100, A100, or DGX Spark — all of which have proper PCIe lanes, official support, and proven stability.

10

Hey mate, thanks for reply. but from these very generic model size comparison, I am confused that real human has written. since now a days, not really comparing this way? rather comparing popular model such as dsv4 flash 0731 or qwen 3.6 27b etc.

I think this comparision is mixing a few very different things, tbh.

First, the cmp numbers in that table are estimates, and they are already contradicted by measured results. The table says 2.5 to 4 tok/s for a 70B Q4 model, while a single unlocked 170hx actually measured 27.3 tok/s on qwen2.5-72b awq. It also lists ~6.3 fp32 tflops and 400–500 int8 tops, while measured results are roughly 12.6 fp32 tflops and only 44 int8 tops. So I dont think the estimates are a useful basis for the conclusion.

Also, generic “8B / 30B / 70B Q4” buckets are not really how I’m evaluating this setup. Modern moe models behave very differently depending on active parameters, quant format, attention backend and topology.

These are actual results:

Hardware Model / topology Measured result
1x 170HX Qwen2.5 72B AWQ 27.3 tok/s
4x 170HX, Gen2 x4, no P2P DeepSeek V4 Flash 0731, PP4 + DSpark 98.1 tok/s, ~5,300 prefill tok/s
8x 170HX, Gen1 x4 GLM5.2 744B/40B active, PP8 30.2 tok/s, 2,675 prefill tok/s at 131K

deepseek result and reproducible setup:
https://github.com/allover326/deepseek-v4-cmp170hx

The pcie link is obviously bad for TP, but these measurements also show why it does not make the cards useless for llm. PP only moves a hidden state between stages. deepseek PP4 works very well even with gen2 x4 and no p2p, while glm 5.2 PP8 already works on gen1 x4.

The memory comparison is also not equivalent. Four v100 32GB cards are only 128GB. Eight v100s are 256GB, while eight unlocked 170hx cards are 512GB. You would need sixteen V100 32GB cards to match the capacity of one 8×170hx server.

power-wise, the post did keep four v100s in the table, not one, but it used 1200w while also describing pcie cards. V100 PCIe is rated at 250w/card, so that should be 1000w; 1200w is the 300w/card sxm figure.
In the current llm tests the 170hx cards are normally being run around 150–180w, though proper wall-power testing is still needed.

There is a software generation gap too. volta has no native bf16 and current vLLM does not support Volta for awq or marlin, while ampere does. Flashattention-2 is also ampere or newer.
v100 can still run plenty of models, of course, but it is not a drop-in replacement for the sm80 paths used by these glm and deepseek results.

My rough view of the 8 gpu & 4 gpu topology are below.

Rough topology outlook

These are projections, not measured results.

Assumptions:

  • Unlocked 64 GB CMP 170hx cards
  • Working gpu p2p
  • Stable cuda graphs and optimized kernels
  • Roughly 150 W per card (120w to 180w by setup)
  • pcie bandwidth measured as real p2p throughput, not just link speed

GLM-5.2 — 8x CMP 170HX

Measured baseline:

  • PP8
  • PCIe Gen1 x4
  • MTP disabled
  • 30.2 tok/s raw decode
  • 665 prefill tok/s at 4K
  • 1,497 prefill tok/s at 32K
  • 2,342 prefill tok/s at 65K
  • 2,675 prefill tok/s at 131K

The following projections use the same assumptions for both
decode and prefill:

  • 120 W per card (since i like to reduce power usage on x8 cards)
  • TP2 x PP4
  • Working GPU P2P
  • Working CUDA Graphs
  • MTP disabled
Metric Gen2 x16 Gen3 x16
C1 raw decode ~43 tok/s ~45 tok/s
4K prefill 900–1,000 tok/s 940–1,010 tok/s
32K prefill 1,450–1,550 tok/s 1,490–1,580 tok/s
65K prefill 1,950–2,150 tok/s 2,050–2,200 tok/s
131K prefill 2,000–2,220 tok/s 2,120–2,300 tok/s

TP2 x PP4 is expected to improve short- and medium-context
prefill because it reduces the pipeline from eight stages to four.

At very long context, however, PP8 is already well filled and may
remain equally fast or faster. The measured full-power PP8 result
is already 2,675 prefill tok/s at 131K.

So faster pcie mainly improves raw decode and shorter prefill.
It is not required for the already strong long-context PP8 result.

DeepSeek V4 Flash 0731 — 4x CMP 170HX

Measured baseline:

  • PP4
  • pcie Gen2 x4
  • No GPU P2P
  • dspark enabled
  • 180 W per card
  • 98.1 tok/s single-stream decode
  • ~5.2–5.3K prefill tok/s

The existing PP4 result already exceeds the usual
2x DGX Spark decode figures as expected, but according to repo owner, this recipe is made by a single non coder who has just given task to claude in a few days, unlikely here many talented ppl has input quite of time and effort ( thanks again).

Expected hybrid-topology potential

Link Best decode topology Estimate
Gen2 x16 + P2P TP2 x PP2 125–155 tok/s
Gen3 x16 + P2P TP2 x PP2 145–180 tok/s

Central estimates at 180 W/card:

  • Gen2 x16: ~140 tok/s
  • Gen3 x16: ~160 tok/s

For long-context prefill, PP4 may remain the best topology:

Configuration Prefill estimate
PP4, measured Gen2 x4 ~5.2–5.3K tok/s
TP2 x PP2, Gen2 x16 ~4.0–5.2K tok/s
TP2 x PP2, Gen3 x16 ~4.7–5.8K tok/s

Faster pcie is not required for the already strong PP4 result.

Its main value is enabling a practical TP2 x PP2 topology,
which may substantially improve single-request decode while
keeping collective overhead far below TP4.

_______________

so…Im not claiming the 170hx is a production replacement for v100 or A100. It is an unofficial unlock, has hardware risk, no normal warranty/support and needs some very specific software work. v100 is clearly better on those points. but honestly havent seen anyone failed unlocked on discord.

But saying “v100 runs circles around it” just doesnt match the llm measurements we have now. PP already performs surprisingly well, and faster p2p/pcie would be an additional path for hybrid TP+PP later, not a requirement for the current results.

11

I got a cmp 170hx with x16 built in, and unlocked it. It is real. Currently attached to my strix halo machine running qwen3.8 at 120tps

12

Oh nice this is my plan with my 10gb CMP thats on the way; attach to the USB4 EGPU for my strix halo instead of putting it in the workstation. Did you get the 8gb and unlock it or are you using the 10gb unlocked?

13

Sorry for the delay! I got the cards with the capacitor mod so they can do 2x16 and peer to peer (being data center cards) can be enabled. They will sit on a 2 GPU switch (PEX 88096) and feed 16 lanes to the main PCIE slot to a new build (that I need to build…I got the components in…) on a ryzen 9 with 64GB DDR5. I’m trying to balance price and capability, so I also snagged one of those connectx6 cards with ocp for 130 that can be flashed back to stock firmware (facebook clearance cards) to sit on the pcie slot that is 4x4 on a consumer AM5 board. I’ll lose pcie5 but gain 16 lane traffic on the CMPs, plus cx6 port witj 64GbE cap (pcie 4x4). I call it the ghetto sparkie.

My half-baked idea is to somehow create a sidecar on the worker spark, so that pipeline parallelism can do head spark —> worker spark —> dual CMP. I don’t know if it will work but we will try. I’m not a comp engineer or software person, I do this for learning and fun, but goal is to be able to either run a larger model in this weird side car, run qwen3.8-122 on tje CMPs with GLM5.3 Flash on the dual sparks, deepseek on dwarfstar on rhe M2 ultra mac and Ling 3.0Flash on the Strix for a 4 agent network. I have this “tower of Babble” pun intended networked with 10Gb but maybe the CX6/CX7 connection would be better across the board — Strix eGPU pcie will get a CX6 card as well, and can hook on to the head spark, making a long sausage of VRAM with the mellanox cards.

BUT to your point/Question: I plugged in on strix to verify the dual GPU switch (works), unlock the card (works), use 170tune to run a benchmark for the memory stability (works; all 64GiBs are intact, no faults) and now it’s running Qwen3.8-27b at massive speed. Let me draw a graph for the models tested so far.

My second card is in Illinois now and heading here, fingers crossed it works as well and enables the Frankenstein data center I’m considering.

IMG_1306
IMG_13061498×824 89.2 KB