TL;DR
We scanned three Q4_K_M GGUFs (Qwen2.5-1.5B, Qwen3-8B, Meta Llama-3-8B) and found that llama.cpp’s per-layer quantization choice — driven purely by weight-space L2 error — independently converges on the same three-phase depth structure reported by Elie Bakouch after analyzing 38 models with CKA and Jacobian lens tools. Three independent measurement signals (activation geometry, weight quantization error, and standard mechanistic-interpretability lenses) converge on the same “sensitive endpoints, tolerant middle” macro structure across two independent training lineages. This is not a coincidence; it is an inherent property of decoder-only next-token language models.
Background — Bakouch’s observation
On 2026-07-08, Elie Bakouch (PrimeIntellect, ex-HuggingFace) shared analysis results covering 38 open-source LLMs including Llama, Qwen, OLMo and more. The work revealed all tested models naturally form a consistent three-phase depth structure:
- Phase A (0–46.5% depth): perception / detokenization
- Phase B (46.5–64.1% depth): workspace / feature engineering
- Phase C (64.1–100% depth): prediction / output preparation
The dividing thresholds 46.5% and 64.1% relative depth remain stable regardless of model size. The analysis relies on Jacobian lens and global workspace interpretability frameworks from Anthropic.
This observation echoes earlier research from Lad et al. (2024), which outlined four distinct inference stages: detokenization → feature engineering → prediction ensembling → residual sharpening. Bakouch’s three-phase framework merges the two middle stages into a unified workspace segment.
Our independent signal
The function llama_tensor_get_type in llama.cpp decides whether each tensor uses Q4_K (4-bit) or Q6_K (6-bit) quantization based on three inputs:
- Predefined high-importance tensors list (embedding layer, output projection, specific layers in certain quantization presets)
- Per-layer L2 quantization error estimation
- Depth-based hard rules (e.g., the last 1/8 layers adopt Q6_K under default presets)
Notably, this logic only processes raw weight values, with no access to activation data, CKA similarity metrics or interpretability analysis. It is a practical heuristic refined by the llama.cpp community through long-term empirical tuning.
If this weight-only quantization scheme automatically reserves higher precision at exactly the same layer boundaries identified via activation-space CKA measurement, we obtain two fully independent validation channels pointing to identical structural patterns.
Method
We built a Python script to parse GGUF model files via the official gguf library. The script groups weight tensors by layer index and module type, counts the proportion of Q4_K and Q6_K quantization per component, calculates relative depth as layer index / total layers - 1, and categorizes each layer into the three structural phases defined by Bakouch.
Results — three models, two independent training lineages
Qwen2.5-1.5B-Q4_K_M (28 layers)
表格
| Role | Phase A (<46.5%) | Phase B (46.5–64.1%) | Phase C (>64.1%) |
|---|---|---|---|
| attn_v | 8/13 = 62% | 2/5 = 40% ⬇ | 4/10 = 40% |
| ffn_down | 8/13 = 62% | 2/5 = 40% ⬇ | 4/10 = 40% |
Qwen3-8B-Q4_K_M (36 layers)
表格
| Role | Phase A (<46.5%) | Phase B (46.5–64.1%) | Phase C (>64.1%) |
|---|---|---|---|
| attn_v | 8/17 = 47% | 2/6 = 33% ⬇ | 8/13 = 62% ⬆ |
| ffn_down | 8/17 = 47% | 2/6 = 33% ⬇ | 8/13 = 62% ⬆ |
Tail lock: Layers 30–35 (85.7–100% depth) use Q6_K for all attn_v and ffn_down weights.
Head lock: Layers 0–3 (0–8.6% depth) fully adopt Q6_K quantization.
Llama-3-8B-Instruct-Q4_K_M (32 layers) — cross-architecture verification
表格
| Role | Phase A (<46.5%) | Phase B (46.5–64.1%) | Phase C (>64.1%) |
|---|---|---|---|
| attn_v | 7/15 = 47% | 2/5 = 40% ⬇ | 7/12 = 58% ⬆ |
| ffn_down | 7/15 = 47% | 2/5 = 40% ⬇ | 7/12 = 58% ⬆ |
Tail lock: Layers 27–31 (87.1–100% depth) are fully Q6_K.
Head lock: Layers 0–3 (0–9.7% depth) are fully Q6_K.
Why Llama-3 matters
Qwen2.5-1.5B and Qwen3-8B are developed by the same team using shared training infrastructure, so matching structural patterns could be attributed to internal quantization tuning preferences.
Meta Llama-3 represents an entirely independent training pipeline with fundamental differences:
- Training corpus: Meta proprietary web & synthetic data vs Alibaba multilingual corpus
- Tokenizer: 128k BPE vs Qwen 152k vocabulary
- RoPE base frequency: 500k vs 1M
- GQA configuration: 8 KV heads out of 32 total attention heads (same ratio but different absolute dimensions)
- Alignment pipeline: RLHF vs Qwen multi-stage SFT + DPO
Even with completely distinct weight distributions, the unified GGUF quantizer prioritizes high precision at identical head/tail layers and reduces precision in the middle workspace segment. The 46.5% and 64.1% depth boundaries align within ±3 percentage points across all three test models.
Why this pattern cannot be coincidental
Three separate analytical approaches converge on the identical layer decomposition rule:
- Activation space analysis: CKA feature similarity + Jacobian lens gradient measurement
- Weight space analysis: llama.cpp layer-wise quantization error heuristic
- Mechanistic interpretability research: Layer probing and logit lens validation from Lad et al. 2024
Consistent structural boundaries hold across two unrelated model training families. This cross-method consistency confirms a universal inherent architectural pattern, stable regardless of measurement tools and training pipelines.
Implications for inference engineering
We recently completed a two-day decode optimization tuning on NVIDIA Tesla V100 for batch size 1 scenarios. For Qwen3-8B, single-token latency (TPOT) dropped from 34.53 ms to 22.39 ms, a 1.54x speedup closing roughly half the performance gap compared to llama.cpp. Previously, our analysis concluded uniform INT8 KV Cache compression was not worthwhile for short context on V100, as the 4%–5% accuracy loss failed to offset minor memory gains.
This new structural finding revises that conclusion. Since Phase B (46.5%–64.1% depth) is the noise-tolerant middle workspace region validated by both interpretability tools and quantization behavior, we implement layer-aware KV Cache quantization: INT4 compression for Phase B layers, FP16 retention for all other layers. This design delivers three key advantages:
- Less output quality degradation than global INT8 quantization, as numerical noise is confined to the model’s most robust layers
- Comparable memory reduction for long context workloads — Phase B accounts for 15%–20% of all transformer layers, matching the memory savings of uniform INT8 compression
- Superior bit-per-quality efficiency against global low-precision KV schemes
We have finished Stage 1 implementation in xLLM: a static kv_layer_format[] configuration hardcoded with Bakouch’s phase thresholds. The engine prints a human-readable layer partition map on initialization as shown below:
plaintext
[engine] kv_layer_format: n_layers=36 phase_B=[16,23) (7 layers, 19.4%) enabled=1
[engine] kv_layer_format map: AAAAAAAAAAAAAAAA|BBBBBBB|AAAAAAAAAAAAA
We verified exact token-level consistency with full FP16 baseline outputs for Qwen2.5-1.5B and Qwen3-8B greedy generation. Next development stages include dedicated INT4 KV cache CUDA kernels and integration into the main forward inference loop, with detailed design notes documented internally.
A second practical optimization direction emerges: same-model self-speculative decoding with early exit at the A/B phase boundary. Existing LayerSkip research proposes early layer exit for draft tokens with full-stack verification, which previously faced fairness concerns when paired with external draft models. Using the principled 46.5% depth split as the exit point eliminates the need for auxiliary small draft models entirely.
Caveats — avoid overgeneralization
- Only three decoder-only models are tested; architectures including Granite, Mixtral, Phi, Gemma and SSM remain unvalidated. Our scanning script supports all GGUF-formatted models, and community test results are welcome.
- Partial “tail lock” high-precision layers stem from hard quantization rules in llama.cpp, not emergent model properties. Embedding, output projection and final layers are forced to higher bitwidth by preset definitions.
- Bakouch’s layer partitioning is measured at activation neuron granularity, while our quantization analysis operates on full weight tensors. The two datasets agree on global macro structure but not per-neuron fine-grained patterns.
- The most reliable unbiased evidence is the reduced Q6_K proportion within Phase B and the gradual precision shift from full Q6_K mixed quantization in the initial 10%–15% layers, as these trends are not enforced by quantization hard rules.
- Bakouch’s original structural observation is a community technical post rather than peer-reviewed literature, though the underlying CKA and Jacobian lens tooling are widely accepted standard interpretability methods.
Closing remarks
Advanced interpretability analysis and years of empirical quantization tuning within llama.cpp independently discovered the identical three-phase layer structure of decoder LLMs. Interpretability probing conclusions align with practical inference engine weight quantization behavior.
Different measurement pipelines, different mathematical signals, identical global structural pattern — this consistency defines a genuine universal intrinsic property of next-token transformers. As inference engineers, our core work is to leverage this natural model characteristic to build more memory-efficient, faster low-precision inference pipelines on hardware like Tesla V100.