Hello everyone,
I’m the author of deep-fold, an experimental open source LLM inference stack. I’d appreciate technical feedback on the CUDA implementation, memory handling, and portability.
My goal is to make local inference more feasible on consumer hardware and to get a sense of the engineering of it. This is also educational: the source, architecture notes, experiment history, benchmark artifacts are public.
ImplementationThe stack includes:
- A Go compressor that packs NF4 weights into a custom container, CHR0;
- A tensor-core NF4 GEMM path for prefill, reconstructing BF16 weight fragments on the fly during multiplication without materializing full dense linear-weight matrices;
- An N=1 CUDA-core NF4 GEMV path for resident decoding;
- A full greedy-step CUDA graph, with token position and valid KV length maintained in device buffers;
- Packed embedding lookup that reconstructs only the requested rows;
- An overflow path, CopyRing, streaming selected packed matrices from pinned host memory through two reusable GPU slots;
- CLI for generation and chat.
One recent improvement was removing a dense embedding table from the resident decode path. Coupled with changes to initialization, this resolved a serious first-request slowdown in the published 20B test.
Published results
My test machine is an RTX 3080 12GB, running Windows/WDDM.
The results below use the same 12-question fixture, a 2,048-token context limit, and up to 256 generated tokens. Average decode rate across requests is used as throughput.
| Model | deep-fold NF4 | Ollama Q4_K | Correct answers: deep-fold / Ollama |
|---|---|---|---|
| Qwen2.5-3B-Instruct | 190.9 tok/s | 197.2 tok/s | 8/12 / 7/12 |
| Qwen2.5-14B-Instruct | 54.8 tok/s | 66.2 tok/s | 11/12 / 10/12 |
| InternLM2.5-20B-Chat | 39.0 tok/s | 13.2 tok/s | 9/12 / 9/12 |
The comparison at 20B reflects different memory placement: deep-fold stays on GPU while the recorded Ollama configuration offloads. It’s not intended as a pure kernel comparison, and the 3B and 14B numbers show Ollama is faster on those rows, and that twelve questions is not enough for an overall quality comparison.
Qwen2.5-32B also runs through the overflow path. A dedicated 64-token benchmark reports about 2.5 tok/s, comparable to Ollama and llama.cpp auto-fit (for that test).
These are single-server measurements. The 3 B and 14 B numbers predate the packed-embedding enhancement; independent replication and updated numbers would be appreciated.
Benchmark protocols and documentation
Hardware support
Current hardware and benchmarks focus on Ampere, particularly sm_86. Paths for alternative acceptably exotic architectures are experimental. I’ve added an experimental sm_120 path for RTX 50 series GPUs but haven’t published any figures for it.
I’d like to coordinate documenting and validating on other GPU architectures, and classifying workloads across additional operating systems.
Where the feedback would be helpful
I’m especially interested in:
- GEMV profiling: what test results would best clarify packed-weight bandwidth, reconstruction overhead, instruction ordering, and thread usage in the N=1 path?
- Memory handling: requests on resident-memory overhead and CopyRing operation, particularly on Windows/WDDM versus Linux.
- Portability: validation and performance results on other GPU architectures.
- Reproducibility: independent tests, install reports, benchmark tooling improvements.
Relevant source code:
AI involvement and contributions
The implementation and documentation were heavily aided by AI agents. I’d like to be transparent about that; a rigorous independent audit would be especially beneficial.
Contributors are welcome, including those able to run on other GPU hardware without modifying code. Helpful information would be checkout commit, deepfold doctor output, GPU, OS, driver/toolkit versions, full benchmark artifacts.
The project is open source under the MIT license. Source and installation instructions.
Thinking back on your CUDA quantized inference experience: what would you test or comment on first, and what evidence would help you interpret this implementation?