FHE - Fully homomorphic encryption means computing directly on encrypted data: the server evaluates the model without ever decrypting the inputs, the activations, or the outputs, and it could not read them if it wanted to, because it never holds the secret key.
Two results, measured this month on a single DGX Spark at 128-bit security, on real model
weights and real tokens:
- Interactive protocol: 1.05 s/token, with output byte-identical to plaintext, 49 of 49 tokens.
- Non-interactive, every layer under encryption, attention and KV cache included, zero
decryptions inside a token: ~6 minutes/token, 5 of 6 tokens identical to the plaintext reference.
No latency or fidelity number below is projected, simulated, or extrapolated from a single
layer. I am posting numbers because published FHE-LLM figures are hard to compare: they
often describe very different systems, and the differences are usually larger than the
speed gaps between them.
Why the interaction model is the headline, not the latency
Most published encrypted-LLM work is one of two shapes. Either only a fraction of the
network is evaluated under encryption, or the client decrypts and re-encrypts activations
at every layer boundary. Both are legitimate engineering. Both get reported under the same
phrase, “encrypted inference”, as if the server never held plaintext at any point. Those
are different claims, so I measured both ends of the spectrum on the same hardware with the
same weights and report them separately.
About the model. I am keeping its identity and dimensions out of this post. Two
concessions do matter to the numbers and I would rather state them than have them assumed
away: the weights are quantized to a small discrete set with a per-tensor scale, and the
MLP activation is replaced with a low-degree polynomial. Both are standard moves for FHE.
Retraining the model to absorb those changes, rather than rounding after the fact, is what
makes the output worth reading: rounding alone put perplexity at 979 against the parent
model’s 4.2, and retraining brought it to 5.8. The encrypted output being bit-identical to
that model is a separate fact with a separate cause: the encrypted linear algebra lands at
rel-L2 1.3e-13 against a double-precision reference, far inside the argmax margin. Every
fidelity number below is against that same retrained model run in the clear.
Security, stated the way I would want to read it: HEStd_128_classic, with the
library’s HE-standard table enforcing the ring and modulus choice rather than prose
asserting it. In the interactive configuration the ciphertext modulus sits at well under
half the standard’s cap for that ring and secret distribution, so the margin is large and
independently checkable rather than marginal.
1. Interactive: the server evaluates the encrypted linear algebra
The server holds no secret key and sees only ciphertext. The key holder computes the
nonlinearities and returns re-encrypted activations.
| Measured | |
|---|---|
| Latency | 1.05 s/token, single stream |
| Throughput | 9.6 tokens/s aggregate at batch 16 |
| Fidelity | 49/49 tokens byte-identical to the same model in plaintext |
| Encrypted matvec | rel-L2 1.3e-13 vs a double-precision reference |
| Wire | tens of MB per token at batch 1; batching cuts it roughly 8x |
| Parameters | ring 2^14 at batch 1, 2^15 at batch 16; no bootstrapping in either |
On how the fidelity number was earned, because “matches plaintext” only means something if
the reference could have disagreed: the plaintext twin is not a separate reimplementation
that might have been tuned into agreement. It reads the same padded weight blocks the
encrypted server evaluates, with the encryption removed, and the loading path is checked
independently against a stock inference library. Same token ids, same order. Not “similar
output”, not a perplexity delta.
Caveats I would want stated if I were the reader: the client and server are roles in one
process on one box here. The split is enforced in code and the server path never touches a
secret key, but the network is not yet between them, and at tens of MB per token the wire
rather than the compute is what stands between this and a remote deployment. And in this
protocol the key holder keeps the KV cache on its own side in the clear, as part of the
nonlinear work it already owns.
2. Non-interactive: every layer under encryption
Both projections, the activation, the normalisations, the attention scores, the softmax,
and the KV cache, all evaluated on ciphertext. The KV cache is never decrypted at any point
in the run. There are zero decryptions inside a token: the server computes a complete token
with no interaction whatsoever, and the key owner comes online only at token boundaries, to
decrypt the logits and feed back the next token’s embedding. An encrypted argmax and an
encrypted embedding lookup would remove even that; I have not attempted them, so I am not
claiming zero interaction full stop.
| Measured | |
|---|---|
| Latency | ~346 s/token best of run; mean 389 s on a memory-contended box |
| Fidelity | 5 of 6 generated tokens identical to the plaintext greedy reference |
| Bootstraps in-pipeline | 0.69 s best, ~1.05 s mean, ring 2^17 |
| Prefill | every prompt position is a full encrypted step, so time-to-first-token scales with prompt length |
The one divergent token is an accuracy limit, not a bug, and it was predicted before the
run: sweeping an FHE-faithful simulator with the bootstrap’s measured error injected at
every site the real pipeline refreshes reproduced both the exact continuation and the exact
index of divergence. The encrypted run then produced precisely that. Closing it needs a
materially more precise bootstrap than this GPU path delivers. That gap, not the packing
and not the calibration, is the ceiling.
I would genuinely like to be pointed at comparable published end-to-end numbers. Everything
I have found either encrypts part of the network or keeps the client in the loop for the
nonlinearities, which is exactly why the two configurations above are reported separately
instead of blended into one flattering figure.
Why this box, specifically
Two properties of the GB10 did real work here. The 128 GB unified pool is what makes the
fully-packed bootstrapping configuration measurable at all: at ring 2^17 with every slot
in use, key material alone holds 106 to 113 GB, which does not fit on an 80 GB card and
leaves almost nothing beside it on a 141 GB one. The generation run in section 2 is packed
at half that and holds 60 GB of keys. And nearly every hot kernel in CKKS is memory-bound
rather than compute-bound: measured by byte counting against the achievable bandwidth on
this machine, the bootstrap pipeline sits at roughly three quarters of the physics floor,
with the keyswitch kernels above 95% of it. Tensor cores are of no use here, and I say
that having measured it: CKKS has almost no data reuse, so the win is bandwidth-optimal
streaming on plain CUDA cores, not GEMM throughput.
I learned that the hard way, and the lesson is about profiling on this chip rather than
about kernels. A profile of the bootstrap suggested its largest kernel bucket, the NTT, had
nearly 3x of headroom, so I rewrote it as a two-pass shared-memory kernel tuned for this
chip. The rewrite is bit-exact against the stock transform, forward and inverse, on a real
ciphertext, and measured head to head on the same buffers in the same stream it is exactly
as fast: 1.00x. The promised headroom never existed. The profile’s per-kernel number was a
rollup averaged over a heterogeneous population of launches, roughly 3x slower than the
kernel’s true per-launch time, and a speedup measured against an average like that is not a
speedup. The sharper trap: GB10 currently exposes no dram__ metrics, so Nsight Compute’s
Speed-of-Light memory percentage is not meaningful here. It reported ~10% memory throughput
for a keyswitch kernel that byte counting places at 97% of the roofline. On this box, count
bytes and divide by measured bandwidth; the tools that usually do this for you cannot, yet.
Three findings that cost me time, so they may save you some
1. CKKS bootstrapping precision degrades badly on a message with a large DC offset.
In my pipeline, a clamp that pins unused lanes to the low edge of the exponent’s Chebyshev
domain handed the bootstrap a message with peak amplitude around 29 and a mean around -12.
That refresh came back with 7.3e-1 absolute error. Recentring the same message to zero mean
first improves it by roughly 33x, which was still not good enough; what actually fixed it
was reordering, so the refresh happens before the operation that introduces the offset.
The failure is quiet: no error is raised, and the garbage it produces looks exactly like a
packing or calibration bug. I spent days looking in the wrong place. If your pipeline masks,
clamps, or pads lanes before a refresh, this is worth checking on your own stack.
2. On the GPU CKKS stack I use, the bootstrap’s precision and iteration arguments are
silently ignored on the device branch. Asking for a higher-precision bootstrap returns
the single-precision result, no warning, no error. Found by measurement, confirmed by
reading the source. Probe your own stack before you trust a precision knob.
3. A Chebyshev domain can be mathematically correct and numerically unusable. At the
same degree, a fit over a generous, safe-looking interval measured 7.5e-1 relative error
where the same degree over the range a real forward pass actually visits gave 4.0e-3, no
bootstrap involved in either. The evaluation loses precision long before the polynomial
does, so my conservative safety margin on the domain actively caused a failure.