Tool call arguments are not streamed: 300 s of silence closes the stream with no finish_reason (integrate.api.nvidia.com)

AI & Data Science NVIDIA NIM NIM on RTX AI PCs and Workstations
, , ,
1

Hello,

I use integrate.api.nvidia.com/v1/chat/completions from a local OpenAI-compatible proxy that drives an agentic coding client. That workload is dominated by tool calls with large arguments, for example writing a file in a single call. I have measured five behaviours of the gateway that make this workload unreliable. All timings below are from my own probes, and I include nvcf-reqid values so you can look the requests up.

Environment: public endpoint https://integrate.api.nvidia.com/v1, a personal nvapi- API key, models minimaxai/minimax-m3 and z-ai/glm-5.2, plain HTTPS from a single client, no proxy in between other than my own local process.

1. Tool call arguments are never streamed, so the stream goes silent and dies

This is the main issue and fixing it would make the other four much less painful.

While the model assembles a tool call, the gateway sends nothing at all. It appears to parse the call and release it only when it is complete. If assembling the call takes longer than the idle limit, the connection is closed and the client gets nothing, even though the model worked the whole time.

Probe from 2026-07-31, minimaxai/minimax-m3, streaming, one tool declared (Write with file_path and content), prompt asking for a 400 line file in a single call:

nvcf-reqid: 3581fff9-f034-4dcc-90fd-c617bafd5383
first byte:            53.0 s
chunks received:       21 (100 characters of assistant text, then nothing)
tool call arguments:   0 characters received
longest silence:       300.0 s
total time:            354.1 s
stream ended with:     no finish_reason, no error object, no [DONE]

The same prompt with the tools field removed, same model, same settings, on the same day:

nvcf-reqid: 270c9d0c-8fc8-4d5a-b1c5-ec3fc2f2155f
first byte:            42.7 s
chunks received:       239
longest silence after first byte: under 1 s
finish_reason:         stop
total time:            60.5 s

So the model streams normally. Only the tool call path is silent. Earlier probes on 2026-07-30 showed the same pattern at larger scale: z-ai/glm-5.2 released about 4000 tokens of tool call arguments in one piece after 138.6 s of complete silence, and minimaxai/minimax-m3 without the tools field streamed continuously for 1513 s with a longest gap of 62.8 s before stopping on max_tokens.

In practice this caps a single tool call at roughly 8000 tokens of arguments, which is about 250 lines of file content. Anything larger cannot be returned at all.

Request: stream delta.tool_calls[].function.arguments in fragments as they are produced, the way the OpenAI API does. Partial JSON in the arguments field is expected by every OpenAI compatible client.

2. No heartbeat while a request is queued

Queue time before the first byte is routinely tens of seconds, even for trivial requests. A request asking only for the word OK with max_tokens: 32:

nvcf-reqid: 8da4693b-e443-4fee-b04e-e4de8f726161
model: z-ai/glm-5.2, non streaming
total time: 34.7 s

The streaming probes above waited 42.7 s and 53.0 s for the first byte. During that window nothing is sent, so any client with an idle timeout has to treat a healthy queued request as a dead connection. My own client library defaults to aborting after 300 s of silence, and the gateway can consume all of that budget before the model even starts.

Request: emit periodic SSE comment lines (for example : keepalive) or empty chunks while a request is queued or generating. This costs nothing on your side and fixes clients you do not control.

3. Streams are closed without any terminal event

In the failure above the stream simply ends. There is no finish_reason, no error object inside the stream and no [DONE] sentinel. A client cannot tell a complete answer from a truncated one, so it either shows a truncated result as final or retries work that may have partly succeeded.

The non streaming path behaves the same way from the caller’s point of view: a bare HTTP 504 after about 302 s, with no indication of what happened.

Request: always terminate a stream with a final chunk carrying a finish_reason, or with an error event describing the reason, before closing the connection.

4. The idle limit is undocumented and cannot be raised

The limit is exactly 300 s of silence between bytes, reproducible across models and across streaming and non streaming modes. I could only establish that by measuring it. I could not find it in the documentation, and there is no way to raise it for tool heavy workloads.

Request: document the limit, and allow raising it per request through a header or a request field.

5. No rate limiting or retry metadata in responses

A successful response carries only these headers:

date, content-type, transfer-encoding, connection, access-control-expose-headers, nvcf-reqid, nvcf-status, vary

There is no Retry-After on 429 or 503, and no X-RateLimit-Remaining or X-RateLimit-Reset. Clients therefore have to guess backoff timing. My client currently guesses: a 15 s cooldown after a 429, and 2 to 8 s escalating backoff when the model is busy. With a real Retry-After this would be exact instead of a heuristic, which is better for your capacity as well as for me.

Request: send Retry-After on 429 and 503, and expose remaining quota and reset time in headers.

6. Undocumented 96 character limit on function names

nvcf-reqid: 1085140a-4db0-4561-a46c-47870221cf73
HTTP 400 {"message":"Validation: Function name at index 0 exceeds 96 character limit, got 99 characters","type":"Bad Request","code":400}

The whole request is rejected, not just the offending tool. Tool names produced by MCP servers follow a mcp__server__tool pattern and exceed 96 characters routinely, so one verbose server breaks every request in a session. I now shorten such names in my proxy and map them back on the way out, but a client cannot know the limit exists until it hits it.

Request: document the limit, and report the offending name rather than only its index. Raising it to 128 characters would remove the problem entirely for MCP style names.

7. Reasoning controls differ per model while validation is global

reasoning_effort is validated against one shared enum for all models (none, minimal, low, medium, high, xhigh, max), and an unknown value is a hard 400. Yet models do not share that scale or even that control:

  • z-ai/glm-5.2 only thinks when chat_template_kwargs.enable_thinking is set
  • minimaxai/minimax-m3 is controlled by chat_template_kwargs.thinking_mode, and sending reasoning_effort at all, even none, enables thinking
  • only the Nemotron family accepts reasoning_budget, other models reject it with a 400 that kills the whole request
  • nvidia/nemotron-3-super-120b-a12b has three effective modes, so levels above its scale only make generation longer without adding any reasoning

Every one of these had to be found by probing the live API.

Request: expose the supported reasoning controls and the effective levels for each model in /v1/models, and ignore rather than reject reasoning parameters a model does not support.

Priority

Item 1 is the one that matters. Items 2 and 3 are small changes that would make every long running request survivable for clients you do not control. Items 4 to 7 are documentation and metadata, but they cost integrators a lot of time.

I am happy to run any additional probe you want and report the results with request IDs.

Thank you.

1 Like