Project Showcase: NexaQuant v1 - Zero-Copy C++ Inference Engine for 1.58-bit Ternary LLMs

Accelerated Computing CUDA CUDA Programming and Performance
1

Hi everyone,

I wanted to share a project I am developing called NexaQuant v1. It’s a high-performance C++ inference engine written from scratch, specifically designed to bypass huge RAM bottlenecks and run Large Language Models on legacy or low-resource hardware.

Core Technical Features:

  • 1.58-bit Ternary Weights: Specifically engineered to leverage ternary quantization mathematics to drastically reduce the model’s memory footprint.
  • Zero-Copy & Memory Mapping (mmap): Built with custom memory management using mmap to stream weights directly from storage, bypassing high RAM requirements and eliminating execution copy overhead.
  • Pure C++ Architecture: Written close to the metal with zero unnecessary runtime bloat.

Development Stack & Environment:

  • Language: Pure C++ (CMake build system)
  • OS: Windows / Linux
  • Target: Low-RAM environments and edge hardware

Looking for Feedback:

I am currently working on optimizing the compute kernels. I would love to get feedback or advice from the NVIDIA community on how to best align these ternary weight structures with hardware acceleration principles (like custom CUDA kernels or exploiting Tensor Cores for non-standard bit widths).

GitHub Repository: GitHub - Nexa1nc/NexaQuant · GitHub

Looking forward to hearing your thoughts and technical suggestions!

2

Hi @ristorim013, thanks for sharing the project.

This is interesting kernel-design work, but it is not a TensorRT support issue yet. The TensorRT category is best for engine build, parser, plugin integration, and runtime behavior once you are actually putting the model through TensorRT.

For the feedback you are asking for - bit packing, custom CUDA kernels, non-standard ternary compute, memory mapping, and whether any Tensor Core path makes sense - CUDA Programming and Performance is the better queue:

My high-level suggestion before you move deeper into TensorRT: first prove the core ternary compute kernel outside TensorRT with a minimal CUDA benchmark. Once the op contract and kernel are stable, a TensorRT plugin can wrap it so the op can live inside a TensorRT engine.

I’m moving the thread over there. You do not need to repost, but adding one minimal kernel or benchmark target in a follow-up will make it much easier for the CUDA folks to give concrete advice.

Thanks,
Atharva