Instructions are in the readme on the GitHub.
Used Colibri base to create a rust port and have the GLM 5.2 model running on one spark.
It is not fast, it is not optimized, but its a start.
Disclaimer: I just started learning this stuff like 2 months ago so forgive my ignorance when reading through the code. I am working on getting the expert load down and multi spark use setup
Nice. I thought about trying something like this when I heard about Colibri. Glm 5.2 seems pretty ambitious though, i imagine its pretty slow.
That version is about 1 token per second, working on some changes that brought it up to 4 tokens /second and started using fp8 to have minimal quality loss.
I am working on setting up multiple sparks now, and trying to hit 8 tokens / second for two sparks.
I think it will show some big problems for the data centers. Frontier performance at less than a month of development cost seems like the shift of computers to personal devices.
cost and cost of speed - if you get it on api for cheap and your system is 10x more expensive in power and 10x as slow .. you paying twice just saying .. its surely nice to be able to run stuff local .. but there is a price to it too specially if you use it different then as a hobby / toy
It is really just for learning and proof on concept, I wouldn’t suggest anyone use this as their inference machine, but it could look for bugs and things at night. Working on speeding it up to make is usable
By itself, probably not that useful. But it’s a very interesting if you can apply this to other models. Imagine running a larger model without REAP, just keep the unused layers in 2nd tier memory. In order to be really useful, it needs to support TP or at least PP or EP.
It does use reap. The update I’m pushing today can use the fp8 and nvfp4 full models. It is expert offloading so the more sparks the better so you’re not fighting the 10GB/s load from disk to RAM
Not really interested in REAP or crazy enough to think I can reasonably run GLM on a DGX Spark, even 15 token/s won’t be practical. Proof of concepts will be interesting when we are more realistic, new large models with 200B parameters or more but only 3-10B active.
sorry, doesn’t *. No reap
The main Colibri project isn’t really looking at the Spark, unfortunately, but it seems like the ideal platform for this. Especially the Gen5 SSD models.
What it needs is GDS direct storage access a la Instanttensor plus support for expert parallel over CX-7 to both enbiggen the RAM pool (keeping more in residence) and linearly scale aggregate SSD read bandwidth.
I’d like to see this sort of thing get more mature, because the total active parameters in say K3 isn’t much higher than GLM-5.2 - it’s just bigger overall. Smart sharding and EP should allow scaling to K3 with over 5 tok/s on 2x Sparks. Maybe up to around 10.
I am working on that. I have it downloading on one spark’s docker container, detecting other connected sparks, splitting up the layers to blobs and loading the blobs to each server rather than the entire model.
Currently each speed test takes about 40 mins which is really a pain. Best run so far was 8 tokens/second on two sparks which is almost usable. I need more data for better predictions for expert loading and I have a bug limiting my upload speed.