TP2 vs TP2+PP2 on 4×DGX Spark — looking for advice

Looking for advice: 2×TP2 vs TP2+PP2 on 4×DGX Spark

We’re running a small game studio with around 20 programmers, and we recently put together a local inference setup using 4× NVIDIA DGX Spark.

Our current setup looks like this:

                LiteLLM
               /      \
              /        \
        Endpoint A    Endpoint B
            TP2          TP2
          Spark 1-2    Spark 3-4

LiteLLM handles routing and sticky sessions so that requests from the same conversation tend to stay on the same endpoint. This has been working well, especially for prefix/KV cache locality and cached-token reuse.

The main issue we’re seeing is that the KV cache pool is only around 1.2M tokens per endpoint, and I’m wondering whether we’re using the 4 Sparks efficiently.

I’m considering changing the topology to something like:

                LiteLLM
                   |
              One endpoint
               TP2 + PP2
                   |
          +--------+--------+
          |                 |
       PP Stage 0        PP Stage 1
        TP2                 TP2
      Spark 1-2           Spark 3-4

My understanding is that PP could reduce the amount of model weights each GPU needs to hold, potentially giving us substantially more room for KV cache.

There is also another reason I’m interested in this architecture:

We may want to serve zai-org/GLM-5.3-Flash locally.

With 4×DGX Spark, I’m wondering whether TP2 + PP2 is a better way to fit and serve this kind of model than maintaining two separate TP2 replicas.

There’s another architectural advantage that seems interesting:

With two separate TP2 endpoints, we need sticky routing to preserve cached-token locality:

conversation A → endpoint A
conversation B → endpoint B

If we turn the whole 4-node setup into a single TP2+PP2 instance, then in theory there is no longer an endpoint-level choice to make. The KV cache is part of one serving instance, so we wouldn’t have to worry about a conversation being routed to the “wrong” node/replica and losing its cached tokens.

That sounds attractive to us because a lot of our workload is long-running coding/agent sessions, where prefix reuse matters.

However, I’m not sure how much we would be giving up by moving from 2 independent TP2 replicas → 1 TP2+PP2 pipeline.

We’re not trying to maximize benchmark numbers. Our priority is good interactive latency + long-context coding sessions + high cache reuse + making the most out of these 4 machines.

Would really appreciate advice from anyone who has actually run pipeline parallel inference across DGX Spark / GB10-class systems.

Thanks!