The MPS documentation names terminate_client as the way to remove a client cleanly: When to Use MPS — Multi-Process Service
We see it fail in two ways on the same GPU, with the same procedure — terminates issued one at a time, each response checked:
1. It damages clients it never touched. Reclaiming a few of the clients leaves untouched neighbours with cudaErrorIllegalAddress (also seen as cudaErrorIllegalInstruction and CUDNN_STATUS_EXECUTION_FAILED). Neighbours are only counted when they were verifiably still making progress five seconds before the reclaim fired. Reproduced in 16 of 17 such cells.
2. It does not come back. One call in a sequence stops responding. The control daemon itself blocks in unix_seqpacket_recvmsg while the MPS server it fronts sits idle in pipe_read, so this is not an unfinished GPU operation:
nvidia-cuda-mps-control -d main: __skb_wait_for_more_packets <- the daemon is stuck
nvidia-cuda-mps-control thr2: __skb_wait_for_more_packets <- CLI awaiting a reply
nvidia-cuda-mps-server pipe_read <- idle
Once in that state the daemon answers nothing at all: a fresh ps on a new connection returns nothing, and a new client attaching does not get error 807 — it hangs too. Killing the pending request does not release it. Only restarting the daemon does. Observed four times independently, hanging at the 15th, 35th, 40th and 40th terminate of a sequence; the longest ran five hours before we gave up on it. No Xid is logged.
These are the same procedure, and the durations are a continuum. The repo below captures the permanent case, but partial stalls are more common in practice: individual terminate_client calls taking 2.6 s and 21.5 s, and one cut off by our own 30 s budget with no answer. The tens-of-seconds case is what we normally hit in production — the caller times out, escalates to SIGKILL, and SIGKILL on a client holding a resident queue then leaves unreclaimable entries in the server until it is restarted.
It is not only a stress artefact. The shapes in the repos (18 and 40 clients) were chosen to raise the rate so the defect could be studied, but the same failures occur in ordinary sharing — around half a dozen tenants on one GPU — which is a normal configuration, not an abusive one.
Reproducers, controls and full measurements:
Measured on B200, driver 580.126.20, CUDA MPS with a private control daemon, tenants in separate containers. Both reproduce in bare docker with nothing but PyTorch and MPS.
Question: is this probabilistic error propagation and unresponsiveness something that can be fixed? Since terminate_client is the documented clean way to remove a client, an operator has nothing else to fall back on — and if it cannot be relied on, that limits what MPS can be used for in a shared, multi-tenant deployment. We would like to know whether a fix is possible, or whether there is a supported way to remove a client that avoids both failure modes.