TZeroAlgorithm: AST-Driven Context Pruning Engine & MCP Server for LLM VRAM Optimization

AI & Data Science NVIDIA AI Workbench
, , , , ,
1

Hi everyone,

I’ve been developing an open-source project called TZeroAlgorithm aimed at solving the context amnesia, KV-cache inflation, and VRAM memory bandwidth pressure caused by large-scale agentic tool calls and codebase-level payload passing.

The Core Problem in Agentic Inference

When LLM agents operate across large repositories or complex tools, passing full code dumps or unoptimized AST payloads creates massive KV-cache bloat. This spikes Time-to-First-Token (TTFT) latency, degrades memory throughput, and exhausts GPU VRAM during multi-turn orchestration.

How TZeroAlgorithm Solves It

Rather than relying on lossy text summarization, TZeroAlgorithm uses deterministic, structural pruning before payloads hit the inference API:

  • Hierarchical AST Pruning: Extracts smart skeletons, call chains (trace_symbol_call_chain), and repository dependency trees (get_project_context_tree, query_module_dependencies) while stripping non-essential bodies.

  • Architecture Boundaries & Quality Audits: Includes rule engines for enforcing structural boundaries (enforce_architecture_boundaries) and detecting dead code / duplication (find_dead_code, find_code_duplicity).

  • Zero-Leak & Hallucination Guardrails: Validates payloads through built-in verification boundaries (verify_zero_leak_security, verify_hallucination_guard) prior to context injection.

  • Model Context Protocol (MCP) Architecture: Runs as a standard MCP server natively exposing 20+ specialized pruning and analysis tools across Claude, Cursor, Windsurf, and custom agent runtimes.

Exploration with NVIDIA Inference Pipelines

I am currently exploring how this pre-inference AST pruning can be paired with NVIDIA TensorRT-LLM and NVIDIA NIM deployment microservices. The objective is to act as a hardware-aware context middleware that minimizes KV-cache memory footprints and token processing overhead on local RTX workstations and datacenter GPUs.

I would love to get feedback from NVIDIA runtime and inference engineers:

  1. Where in the NIM / TensorRT-LLM pre-processing stack could AST-aware context pruning deliver the highest throughput gains?

  2. Has anyone benchmarked structural context pruning against native attention window compression methods on long-context models?

Looking forward to your insights!
Feature Request