Designing a decentralized GPU inference network — looking for architectural feedback

Accelerated Computing NGC GPU Cloud
, , , ,
1

Hi everyone,

I’m designing a distributed inference system that uses idle GPUs from edge machines (gamers/workstations) to run AI inference jobs.

The goal is:

- stateless jobs

- containerized execution

- central orchestrator

- agents connected via WebSocket

- direct object storage for outputs (to avoid routing binaries through the control plane)

Architecture layers:

1. API Gateway

2. Orchestrator/Scheduler

3. State store (Redis)

4. Edge Agent (Docker runner)

5. Object storage (data plane)

Before implementing everything, I’d really appreciate feedback from people experienced with CUDA, GPU scheduling, or distributed systems.

Main questions:

- Best practices for managing long-lived GPU workers?

- Handling unreliable or intermittent edge nodes?

- Model caching strategies to reduce cold starts?

- Pitfalls you’ve seen in similar distributed inference systems?

- Security: how would you safely execute potentially untrusted models/containers (e.g., dynamically downloaded models) on provider machines without risking the host system?

Thanks in advance for any insights!

2

Reproducible FP16 vs FP32 performance delta on Tesla T4 (CUDA 12.5)

Environment

GPU: Tesla T4 (16 GB)

Driver: 550.54.15

CUDA Toolkit: 12.5 (nvcc 12.5.82)

Python: 3.12.12

Benchmark Method

CUDA-event timing with torch.cuda.synchronize()

Warmup iterations included

Average per-iteration latency reported

Results

FP32: 354.92 ms

FP16: 40.68 ms

Speedup: ~8.7×

Question I suspect this workload is engaging Tensor Cores on T4 under FP16. Aside from Nsight Compute, is there a recommended way to definitively confirm Tensor Core utilization or kernel selection on Turing GPUs?