Well 20 people might be too much for the CPU. Remember, that kind of animation runs through the CPU first for processing. So it does not matter about the Blackwell Pro 6000, if the cpu is the limiter. If 1 person runs fine, and 5 people runs fine, eventually there will be a point that it starts to slow down. The FPS on the GPU is nothing to do with the cpu.
If you don’t think it is limited by the cpu, then crank up the fps for the stage from 60fps to 120fps and see if they literally double in speed. If so, then it is not system limited. It may be something else. Increase the stage FPS until you get the desired speed and see if you can get everything smooth. Start with 120fps, and see.
Also, as I said earlier, rather than jumping from 1 person to 20, try 5 people, 10 people etc. Maybe there is a hard point of inflection, where 9 people are fine and 10 people really drops off.
That is really great news ! Not sure what the problem is, but really happy to hear it is working well. Let me ask you this. Are those characters does with an FBX file and “baked” keyframed animation or are they all driven with USDSkel animation data?