Tauergon agent harness optimized for local llm

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10 Projects
, , ,
1

Hi,

I’ve been experimenting creating my own agent harness purely to better understand - there are a lot of good (open source) harnesses out there, but for me doing myself is often the way to truely understand.

Eventually it turned into something I’ve started using daily, so I figured I could share as well (and named it tauergon in short).

I’m truely just releasing this as concept - I think there are a couple of nice ideas - but not a polished nice UI system.

To ensure its working I’ve been running SWE Verified, looks like I have 67% with a Qwen 3.6 27b NVFP4 (although I will not be able to officially submit/get this verified based on new submission rules - not academia). (Update: running official validator locally shows almost identical pass rate)

Key ideas:

  • tau agent fully capable of modifying itself - you want it changed, use it to change (and test) itself
  • full agentic loop with tools
  • tried to heavily optimiye for local AI:
    • maximizing prompt/prefix cache re-use to get fast loop times
    • multiple levels of llm response postprocessing, fixing, validating, rejecting (i.e. extracting toolcalls from reason/content, etc.)
    • fork tool to spawn (synchronous) subagent utilizing parents kv cache
  • dream loop for continous self improvement
  • pure python, zero dependancies (0!) - all you need is python (I tested 3.12.3 and 3.13.5)
    • will run very easy on almost anything recent
    • you can “see” (and change) the full end to end in code without diving into complex modules

Warning:

  • This is dangerous and can execute arbitrary code (i.e. bash) - if you give it passwordless sudo even that - you should only use this experimentally in a sandboxed environment

Requirements:

  • OpenAI compatible endpoint
  • Python (3.12.3 and 3.13.5 tested)

I’m currently using this as a daily driver for a lot of my stuff:

  • coding (some other proejcts I’m working on like a worldbuild, some local training, etc. …)
  • configuring my systems, testing new models (and ways to serve them)
  • playing with ways todo tools different (different file edit approaches, web search,

Notes:

  • I am using a local searxng and crawl4ai to improve results - but it will work without them
  • As you see in my hardware spec below I’m typically going for larger contexts (200k) - its not (yet) super context optimizes - but maybe something you’d like to ask it todo to itself ;-)
  • I am currently running SWE Verified with it (and qwen 3.6 27b for now) - time consuming, but will publish as I have it
  • I’ve been running this with different model classes (nemotron, qwen, gemma) mainly with llama.cpp and vllm (briefly tensorrt and sglang), this is currently my main driver (but that changes weekly):

Have fun! Experiment! What its really nice for is to ask it to modify itself…

PC (to get interactive work done quickly - mostly single session):

  • Debian 13

  • Nvidia RTX 5090

  • cuda_13.3.0_610.43.02_linux.run

  • model is sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP, llama-benchy rates it around 10k/sec pp and 150+/sec decode

  • docker-compose.yaml (depending on you system you might need to increase health check timeout for startup):

    version: '3.8'

volumes:
  # Shared volatile storage for the downloaded chat template
  template-share:

services:
  # --- INIT CONTAINER: Downloads the template ---
  template-downloader:
    image: alpine:latest
    container_name: qwen-template-downloader
    volumes:
      - template-share:/tmp/chat-templates
    command: >
      sh -c "apk add --no-cache curl &&
             echo 'Downloading froggeric chat template...' &&
              curl -sL 'https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates/resolve/main/chat_template.jinja' -o /tmp/chat-templates/chat_template.jinja"

  # --- MAIN CONTAINER: vLLM Inference Engine ---
  qwen-vllm:
    image: vllm/vllm-openai:v0.22.1
    container_name: qwen36-27b-nvfp4-mtp
    ports:
      - "8000:8000"
    shm_size: '32gb'
    ipc: host
    restart: always
    depends_on:
      template-downloader:
        condition: service_completed_successfully # Only starts after download finishes
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    volumes:
      - /home/alangeb/.cache/huggingface:/root/.cache/huggingface
      - template-share:/tmp/chat-templates # Mounts the shared volume containing the downloaded file
    environment:
      - VLLM_NVFP4_GEMM_BACKEND=flashinfer-cutlass
      - VLLM_USE_FLASHINFER_SAMPLER=1
      - VLLM_USE_DEEP_GEMM=1
      - PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
    command:
      - sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP
      - --host
      - 0.0.0.0
      - --port
      - "8000"
      - --served-model-name
      - model
      - --quantization
      - modelopt
      - --trust-remote-code
      - --language-model-only
      - --max-model-len
      - "200000"
      - --gpu-memory-utilization
      - "0.95"
      - --max-num-seqs
      - "2"
      - --max-num-batched-tokens
      - "2048"
      - --enable-chunked-prefill
      - --enable-prefix-caching
      - --kv-cache-dtype
      - fp8
      - --calculate-kv-scales
      - --reasoning-parser
      - qwen3
      - --tool-call-parser
      - qwen3_coder
      - --chat-template
      - /tmp/chat-templates/chat_template.jinja
      - --enable-auto-tool-choice
      - --enable-prompt-tokens-details
      - --speculative-config
      - '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'
      - -tp
      - "1"
      - -pp
      - "1"
    
    # Watchdog implementation for hangs/engine death
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://localhost:8000/health || exit 1"]
      interval: 15s     
      timeout: 10s      
      retries: 3        
      start_period: 300s

DGX Spark (to drive multiple parallel batch work sessions):

  • sparkrun with @official/qwen3.6-27b-fp8-mtp-vllm

Alternative for NVFP4:

  • cuda 13.0
  • model is sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP - I am using this here to stay close to my other systems config
  • startup script (I realize I should probably not use v0.23.0… but it seems to work):
#!/bin/bash
# Start Qwen3.6-27B NVFP4-MTP via vLLM Docker (froggeric chat template)
# - NVFP4 quantization with MTP speculative decoding
# - Uses froggeric/Qwen-Fixed-Chat-Templates (v19) for improved agentic loop & KV cache

set -e

CONTAINER_NAME="qwen36-27b-nvfp4-mtp"
MODEL="sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP"
IMAGE="vllm/vllm-openai:v0.23.0"
PORT=8000
HUGGINGFACE_CACHE="/home/alangeb/.cache/huggingface"
CHAT_TEMPLATE_URL="https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates/resolve/main/chat_template.jinja"
CHAT_TEMPLATE_LOCAL="/tmp/chat_template.jinja"

# Download chat template if not present or outdated
if [ ! -f "${CHAT_TEMPLATE_LOCAL}" ]; then
  echo "Downloading froggeric chat template..."
  curl -sL "${CHAT_TEMPLATE_URL}" -o "${CHAT_TEMPLATE_LOCAL}"
fi

# Stop existing container if running
if docker ps -a --format '{{.Names}}' | grep -q "^${CONTAINER_NAME}$"; then
  echo "Stopping existing container ${CONTAINER_NAME}..."
  docker stop "${CONTAINER_NAME}" 2>/dev/null || true
  docker rm "${CONTAINER_NAME}" 2>/dev/null || true
fi

echo "Starting ${MODEL} on port ${PORT}..."

docker run \
  --rm \
  --name "${CONTAINER_NAME}" \
  --gpus all \
  -p ${PORT}:${PORT} \
  -v ${HUGGINGFACE_CACHE}:/root/.cache/huggingface \
  -v ${CHAT_TEMPLATE_LOCAL}:${CHAT_TEMPLATE_LOCAL} \
  -e VLLM_USE_FLASHINFER_SAMPLER=1 \
  -e VLLM_USE_DEEP_GEMM=1 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  ${IMAGE} \
  ${MODEL} \
  --host 0.0.0.0 \
  --port ${PORT} \
  --served-model-name model \
  --quantization modelopt \
  --linear-backend auto \
  --trust-remote-code \
  --language-model-only \
  --max-model-len 200000 \
  --gpu-memory-utilization 0.75 \
  --max-num-seqs 2 \
  --max-num-batched-tokens 2048 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --kv-cache-dtype fp8 \
  --calculate-kv-scales \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --chat-template ${CHAT_TEMPLATE_LOCAL} \
  --enable-auto-tool-choice \
  --enable-prompt-tokens-details \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  -tp 1 \
  -pp 1
1 Like