EdgeAI Forge: My Journey Toward a Local Agentic AI Platform for Industrial Automation

Robotics & Edge Computing Robotics - Isaac Isaac ROS
1

I am building an early architecture called **EdgeAI Forge** to explore local-first AI-assisted engineering for machine vision, ROS 2, robotics, and industrial edge deployment.

The target design separates:

- a control plane for orchestration, memory, artifacts, APIs, approvals, and observability; and

- a compute plane for local LLM/VLM inference, vision workloads, TensorRT optimization, and robotics models.

One planned workflow begins with a request such as:

> Build a ROS 2 object-detection pipeline, generate the nodes and launch files, test it, benchmark FPS/latency, optimize it, and prepare a reviewed Jetson deployment.

The repository currently contains a FastAPI/Next.js platform scaffold and a small Ollama-backed Planner/Vision/ROS proof of concept. Jetson, Isaac ROS, TensorRT, and production deployment features are roadmap work rather than completed capabilities.

I would appreciate technical guidance from the NVIDIA developer community on:

1. A sensible simulation-to-Jetson workflow for generated ROS 2 vision packages.

2. Where Isaac ROS components provide the most leverage for an initial MVP.

3. Reliable ONNX/TensorRT export and validation patterns across workstation and Jetson targets.

4. Benchmark methodology for end-to-end latency, FPS, power, memory, and thermal behavior.

5. Practical local VLM/LLM serving options when the control and compute planes are separate.

6. Safety and rollback boundaries for agent-assisted deployment.

Project: GitHub - afridali123/EdgeAI_Forge · GitHub

Architecture background: EdgeAI Forge: My Journey Toward a Local Agentic AI Platform for Industrial Automation

This is a request for architecture feedback, not a claim of production readiness. Concrete references, sample projects, and lessons learned would be very helpful.

2

Hello @afridali123,

Welcome to the Isaac ROS forum and thanks for sharing the project.
For an initial MVP, I would recommend narrowing the scope to one deterministic reference workflow before adding broader agentic deployment features:

  1. Simulation / recorded-data validation

  2. Isaac ROS perception pipeline

    • For object detection, start from an existing Isaac ROS object detection package such as YOLOv8 / RT-DETR — Isaac ROS / DetectNet.
    • If you need a custom model, use the Isaac ROS DNN inference path:
      image encoder → TensorRT or Triton inference → decoder/post-processing.
    • This keeps the generated-agent portion focused on launch files, config, tests, and integration rather than generating the whole accelerated perception stack from scratch.

  3. TensorRT export and validation

    • Keep ONNX as the exchange artifact.
    • Build TensorRT engines with trtexec or the Isaac ROS TensorRT node on the target platform, because serialized TensorRT engines are sensitive to TensorRT version, platform, and GPU target.
    • Validate both functional correctness and latency before treating the engine as deployable.

  4. Benchmarking

    • Use Isaac ROS Benchmark / ros2_benchmark for ROS graph latency, throughput, and utilization.
    • On Jetson, also collect tegrastats logs for power, memory, thermals, and throttling behavior.

  5. Local LLM/VLM serving

    • Keep the control plane and compute plane separate as you described.
    • For local model serving, Jetson AI Lab examples using Ollama/vLLM-style OpenAI-compatible endpoints are a good starting point.
    • The LLM/VLM agent should produce plans, configs, code patches, and test requests, but should not directly deploy to hardware without approval gates.

  6. Safety / rollback

    • Treat generated ROS packages, TensorRT engines, Docker images, and deployment manifests as versioned artifacts.
    • Require human review before hardware deployment.
    • Use staged deployment: simulation → rosbag replay → Jetson HIL → real hardware.
    • Keep rollback simple: previous container image tag, previous model/engine artifact, and previous launch/config bundle.

Good luck with your work!