Local Qwen-Controlled 3D Agent with CUDA Action Loop on Jetson Orin Nano

Robotics & Edge Computing Jetson Systems Jetson Orin Nano
, ,
1

Local Qwen-Controlled 3D Agent with CUDA Action Loop on Jetson Orin Nano

Hello NVIDIA Developer Community,

I would like to share a real-time 3D agent experiment that I have been developing on the NVIDIA Jetson Orin Nano Super.

This project combines a local language-model command layer, a GLB-based 3D character, a predefined action system, a dedicated CUDA action loop, and real-time OpenGL rendering.

The main goal is to explore how natural-language instructions can be converted into deterministic 3D character actions while keeping the rendering and action-update pipeline responsive on Jetson.

Video Demonstration

The demonstration shows English text commands such as:

  • jump
  • turn left
  • look left
  • look down
  • cheer to me

The command is interpreted by the agent layer and mapped to an existing action supported by the 3D character.

The character then executes the corresponding motion in real time.

High-Level Architecture

The current pipeline can be simplified as:

User Text Input
→ Local Qwen / Agent Interpretation
→ Action Validation
→ Action ID / Action State
→ CUDA Action Loop
→ Character Transform Matrix Update
→ OpenGL GLB Rendering

One important design requirement is that the language model does not directly generate arbitrary animation data.

Instead, the model selects or resolves an action that already exists in the application.

This keeps the final movement deterministic and allows the application to validate the requested action before it reaches the rendering system.

Natural-Language-to-Action Mapping

The agent receives a natural-language instruction and resolves it into a supported action.

For example:

jump
→ JUMP action

turn left
→ TURN_LEFT action

look down
→ LOOK_DOWN action

The system can also normalize different phrases into the same internal action.

For example:

turn to the left
rotate left
face left

can all be mapped to the same predefined movement state.

This makes the language layer flexible while keeping the underlying animation system constrained.

Why I Use Predefined Actions

I intentionally separate LLM reasoning from low-level character movement.

The LLM is responsible for deciding what the character should do.

The action system is responsible for deciding how that action is executed.

This avoids allowing the language model to directly produce arbitrary transformation matrices or unsupported animation states.

The resulting architecture is approximately:

LLM decision
→ validated action
→ deterministic motion system

rather than:

LLM
→ arbitrary transform values

This also makes the system easier to debug and reproduce.

CUDA Action Loop

A dedicated CUDA-oriented action stage handles the dynamic movement state.

The action system updates parameters such as:

  • Character position
  • Orientation
  • Rotation
  • Look direction
  • Action progress
  • Transition timing
  • Motion phase
  • Return-to-idle state

The objective is to keep frequently updated action calculations separated from the language-model request path.

This allows the LLM to operate at a much lower frequency than the rendering loop.

For example:

LLM / command decision
→ occasional event

Action update
→ continuous

OpenGL rendering
→ continuous frame loop

This separation is useful because an interactive 3D agent should not need to wait for a new LLM response every rendered frame.

GLB Character Rendering

The character is loaded as a GLB asset and rendered through OpenGL.

The rendering stage maintains the current character transformation generated by the action system.

The character system includes:

  • Idle state
  • Direction changes
  • Look-direction changes
  • Jump movement
  • Gesture-style actions
  • Transition back to idle

The rendering thread remains independent from the language-model interaction as much as possible.

Agent State Validation

Another important part of the architecture is action validation.

Before an LLM-selected action is executed, the application checks whether the action belongs to the predefined action set.

Conceptually:

LLM output
→ Parse action
→ Validate action
→ Execute supported action

If the model produces an unsupported command, the system can reject it or fall back to a safe default instead of attempting to generate an undefined character state.

I think this becomes increasingly important when an LLM is connected to a real-time interactive system.

Different Timing Domains

One technical challenge is that several subsystems run at very different rates.

For example:

Text / LLM request
→ relatively infrequent

Action state update
→ frequent

OpenGL rendering
→ every frame

This creates several synchronization questions.

The rendering loop should continue smoothly while the language-model layer is processing a command.

Similarly, the action loop should not unnecessarily block either rendering or future command processing.

I am currently investigating how best to organize these stages using separate threads, queues and GPU synchronization.

Current Development Focus

My current areas of investigation include:

  • Lower command-to-action latency
  • More reliable natural-language action parsing
  • Strict action validation
  • CUDA action-loop synchronization
  • Smooth transition between actions
  • Avoiding rendering stalls during LLM processing
  • Thread-safe action-state exchange
  • GLB rendering performance
  • Local inference latency
  • Expanding the number of supported actions

Toward a More Autonomous 3D Agent

The current demonstration primarily uses explicit user commands.

A future direction would be to give the agent additional state such as:

  • Character position
  • Environment information
  • Nearby objects
  • Previous actions
  • Action cooldowns
  • Current animation state

The language model could then select an appropriate predefined action from the current context instead of responding only to direct commands.

The important design principle would remain the same:

LLM reasoning
→ constrained action selection
→ validated execution

rather than allowing unrestricted low-level control.

Questions for the Community

I would appreciate feedback on several technical areas:

  1. For a local LLM controlling a real-time 3D application on Jetson, is separating the LLM request loop from the high-frequency action/render loop the recommended architecture?
  2. What synchronization strategy would you recommend for transferring a new action state from an LLM thread into a CUDA/OpenGL rendering pipeline without introducing frame stalls?
  3. Would a small lock-free or ring-buffer-based action queue be preferable to sharing a single synchronized action state between the LLM and rendering threads?
  4. For character actions that mainly consist of transformation-matrix updates, is CUDA useful when the number of controlled characters becomes large, while CPU-side updates remain more appropriate for a single character?
  5. When running both a local language model and real-time graphics on the same Jetson GPU, what would be the best way to measure and balance the GPU workload between inference and rendering?
  6. Would separate CUDA streams or priority streams be useful when interactive rendering latency should take priority over background agent processing?
  7. Which Nsight Systems measurements would be most useful for analyzing command-to-action latency across the LLM, action-update and rendering stages?

If useful, I can also prepare a minimal reproducible sample that removes the full GLB asset and isolates only:

Text command
→ action parsing
→ validated action state
→ CUDA matrix update
→ simple OpenGL visualization.

Thank you for reading.

Any feedback regarding Jetson local LLM integration, CUDA action processing, real-time 3D agents or CPU/GPU synchronization would be greatly appreciated.

3

Hi,

Thanks for sharing your experiment.

Sorry that we don’t have much experience on this usecase so might not capture your issue correctly.

What kind of the workflow in your scheme?

LLM -> rendering -> LLM -> rendering?

or

rendering - rendering
    |           |
   LLM    -    LLM

For the second version, you can deploy the CUDA tasks so they can run in parallel.
Please check MPC or NvSCI for multi-process use case.

Nsight System can provide more detailed profiling data but you can check the GPU loading with the following command first.

$ sudo tegrastats

Thanks.

4

Hi AastaLLL,

Thank you for the explanation.

My current design is much closer to the second architecture.

The LLM is not called for every rendered frame, and the rendering loop does not wait for the LLM inference to finish.

The current workflow is approximately:

User text input
|
v
Local Qwen inference
|
v
Parse / validate predefined Action ID
|
v
Action state / command queue
|
±---------------------+
|
v
CUDA Action Loop
|
v
Transform / motion state
|
v
OpenGL Rendering Loop
(continuous)

The OpenGL rendering loop and the high-frequency action-update loop continue running independently.

The Qwen model is invoked only when a new command or agent decision is required.

For example:

Rendering: continuous frame loop
Action update: continuous high-frequency loop
LLM inference: event-driven / relatively low frequency

Therefore, the LLM decides WHAT action should be executed, while the predefined action system determines HOW the motion is executed.

The LLM only outputs a validated action such as:

JUMP
TURN_LEFT
LOOK_DOWN
CHEER

It does not directly generate transformation matrices every frame.

I am currently keeping these components decoupled so that LLM inference does not block the real-time rendering path.

At the moment, I am also investigating the best synchronization method between the LLM command state, CUDA action update, and OpenGL rendering.

I will run sudo tegrastats specifically with this Qwen + GLB application and check the GPU utilization while:

  1. Rendering only
  2. Running Qwen inference only
  3. Running Qwen inference and 3D rendering concurrently

I will share the results after testing.

For the current architecture, would you recommend keeping the LLM and rendering stages as separate threads in one process, or would CUDA MPS / NvSci become useful if I later separate them into multiple processes?

Thank you for the guidance.

5

Hi,

It sounds like you don’t need to share the CUDA memory between two tasks.

If so, you can exchange the buffer with a standard CPU buffer directly.
Then submit the kernel or rendering when the content is updated.

Thanks.

6

Hi AastaLLL,

Thank you for the clarification.

Yes, that matches my current use case. The LLM and rendering/action tasks do not need to share a large CUDA memory buffer directly.

The data exchanged between them is relatively small, mainly an action ID and a few state parameters such as:

action type
target orientation
action progress/state
timing information

So I can keep this information in a normal CPU-side shared buffer or command queue.

When a new validated action is produced by the LLM, the CPU-side state is updated, and then the CUDA action kernel / OpenGL rendering path consumes the new state on the next update.

Conceptually, the workflow is:

LLM → CPU action buffer → CUDA action update → OpenGL rendering

while the rendering loop continues independently.

I agree that this is simpler than sharing CUDA device memory directly between the two tasks.

I will continue testing this architecture and profile the synchronization overhead with Nsight Systems.

Thank you again for the guidance.