Has anyone had any luck using a spark-vllm-docker model as a drop in replacement in langchain’s ChatAnthropic implementation? We’re looking to run evals faster and cheaper on some agents using smaller local models.
Our agents use streaming, tool calling, citations and structured responses via ChatAnthropic which should resolve to vllm messages endpoint. We’ve gotten close with qwen3.5 variants but hitting tool call response shapes not matching so the agent loops or hangs retrying.
I was able to root out the issue that was blocking our agent to vllm not supporting search_result content blocks. However, like all the runtimes, the vllm messages endpoint implementation is pretty rudimentary. So, sticking with basic features and content block types is probably wise at this point.