There are some patches that have not made it into vLLM release which will stop the crashes and most failed responses to requests. Also you should check out this guy’s post. Using the cuda coredump flag will stop the crashes due to race conditions
Apply #49876 + #51391 + #50528 to stop tool call and thinking leakages
Apply #49227 to stop 500 errors related to 200028 at the end of structured outputs
After these patches were applied, the model stops the undesirable behaviors which can ruin the agent’s performance.
Since you’re running it text only the vision problems aren’t an issue. The model was said to be trained for financial firms which are funding the company, so I’d guess its strengths should be in that area