Architecture for a Live Multi-Agent Desktop App Using NVIDIA NIM

AI & Data Science NVIDIA NIM Models
,
1

Hello everyone,

I am working on a local desktop application in which several AI agents can collaborate inside a shared chat and operate within an isolated sandbox or project folder, somewhat similar to the agent and skill-based workflows available in tools such as OpenCode.

However, I want to build a more complex multi-agent interaction system.

The main idea is to allow three to five AI agents to:

  • Analyze the same task or project from different perspectives

  • See messages generated by the other agents

  • React to arguments, findings, and proposed solutions

  • Adjust their own behavior during the discussion

  • Work inside a shared project folder or restricted sandbox

  • Produce a final consolidated answer through a lead or moderator agent

Initially, the agents could communicate sequentially. Later, I would like to experiment with live or streaming interaction, where several agents generate responses while observing the discussion as it develops.

Why I think this could be useful

When working with AI coding agents, I often notice that different models discover different problems in the same project. One agent may identify architectural issues, while another notices implementation errors, missing edge cases, or security risks.

A structured discussion between several specialized agents could therefore produce a better result than simply asking one model to solve the entire task.

What I have already built

I created an experimental “Fusion” mode with the following workflow:

  1. A single task is sent to several agents.

  2. Each agent independently analyzes the task.

  3. The agents review and debate the other responses.

  4. They vote on the strongest conclusions or solutions.

  5. A moderator model receives the complete interaction history and produces the final answer.

This system works, but during development I noticed several weaknesses. It behaves more like a multi-stage pipeline than an active group discussion.

What I am trying to build now

I want to create a persistent multi-agent chat in which two to five agents communicate with each other more naturally.

The system should still have strict limits so that the discussion does not become an endless loop. A moderator or lead agent would control the process by:

  • Selecting which agent should respond next

  • Detecting repeated arguments

  • Limiting the number of rounds, tokens, or tool calls

  • Requesting clarification from specific agents

  • Resolving conflicts between agent conclusions

  • Stopping the discussion when sufficient agreement or useful disagreement has been reached

  • Producing a concise final summary or implementation plan

Eventually, I would also like agents to use tools and skills inside a shared sandbox, while preventing conflicting file edits and unsafe operations.

Current problems

I have created several prototypes, but I still encounter problems with:

  • Coordinating multiple streaming model responses

  • Allowing agents to observe each other without creating race conditions

  • Preventing repetitive or endless conversations

  • Managing shared context without sending the entire history on every request

  • Separating private agent reasoning from messages visible to other agents

  • Handling simultaneous tool calls and file modifications

  • Determining when the moderator should stop the discussion

  • Managing API rate limits and costs

  • Recovering the session after an interrupted request or application restart

During testing, I used NVIDIA NIM APIs, OpenCode Go API, OpenCode Zen API, and the DeepSeek API.

Questions

Has anyone here worked on a similar multi-agent architecture using NVIDIA NIM?

I would especially appreciate advice about:

  1. Which NVIDIA NIM models would be suitable for different agent roles, such as coding, planning, reviewing, moderation, and vision?

  2. Is there a recommended architecture for coordinating multiple NIM inference streams?

  3. Would it be better to implement the system as a deterministic state machine, an event-driven message bus, or a graph-based workflow?

  4. What is the best way to control discussion length without reducing the usefulness of the debate?

  5. Are there NVIDIA examples, SDKs, reference projects, or documentation related to multi-agent orchestration?

  6. Are there recommended approaches for shared tool execution and sandboxed project access?

  7. How should context caching, summarization, and agent memory be organized in such a system?

  8. Is simultaneous streaming from several agents practical, or should the first version remain turn-based?

The application is currently an experimental personal project rather than a commercial product. My goal is to better understand multi-agent systems and build a useful local environment for coding, research, project analysis, and collaborative problem-solving.

Any architectural suggestions, examples, documentation, or criticism of the concept would be useful.

1 Like