I recently published an architectural research note on using INT2 not as
a replacement precision, but as a predictive head nested inside an
FP8/packed-FP4 precision hierarchy.
## Hypothesis
A stabilized FP4 representation can provide a bounded target surface on
which an INT2 predictor learns a coarse estimate, leaving the full model
body to compute only the residual that the predictor cannot reliably
supply.
The native C++/CUDA implementation is currently training on Hopper.
Because Hopper does not natively execute the complete intended INT2/FP4
compute hierarchy, the low-bit representations are software-managed and
expanded into supported compute formats. The current work therefore
focuses on representational learnability and the control model:
1. Preserving mutable delayed-scaling state across activation-checkpoint
recomputation.
2. Enforcing per-layer ownership of packed-FP4 state histories.
3. Training an INT2 coarse predictor inside that stabilized envelope.
4. Measuring future delivered coverage rather than trusting an
in-window confidence value.
5. Using a governor to grant limited and reversible authority only after
external calibration.
6. Measuring actual residual-work reduction before making a throughput
claim.
The broader idea is to treat extreme low precision as a division of
labor—coarse prediction versus residual correction—rather than as
uniform compression of the full operation.
The article includes the low-bit state invariants, the Lion learning-rate
correction for the predictive head, the nested INT2 → FP4 → FP8 model,
and explicit falsification gates.
**Full research note:**
[From FP4 Stability to INT2 Prediction]( From FP4 Stability to INT2 Prediction )
I am particularly interested in how others are handling temporal
quantizer-state mutation inside activation-checkpoint recomputation in
native CUDA training systems.