Setup
-
Hardware: NVIDIA DGX Spark / Lenovo ThinkStation PGX (GB10 Blackwell Superchip, 128GB unified memory)
-
Base Docker image: custom vLLM build from eugr/spark-vllm-docker, built with the
--tf5flag (vllm-node-tf5:latest) -
Model:
nvidia/LocateAnything-3B
We used the eugr/spark-vllm-docker build pipeline because it targets the GB10 Blackwell (SM121) architecture, which the official vLLM container doesn’t fully support out of the box. The --tf5 flag enables CUDA 13 compatibility needed on DGX Spark. It’s unclear whether a plain PyTorch container would also work — we didn’t test that — but if you’re on a DGX Spark or ThinkStation PGX this is a known-good base image.
LocateAnything is not a vLLM-served model. We run it as a standalone FastAPI server inside the container using the official LocateAnythingWorker from the NVlabs/Eagle repo, exposing a task-oriented REST API.
Issues encountered and how we solved them
1. decord not available on ARM64
The model’s custom code imports decord (a video loading library). transformers’ check_imports scans for it statically before any code runs, so a sys.modules stub doesn’t work. The PyPI decord package also has no ARM64 wheel.
Fix: build and install a minimal stub package locally:
bash
mkdir -p /tmp/decord-stub/decord
cat > /tmp/decord-stub/decord/__init__.py << 'EOF'
class VideoReader:
def __init__(self, *a, **kw): pass
def __len__(self): return 0
def __getitem__(self, idx): return None
def get_avg_fps(self): return 30.0
EOF
cat > /tmp/decord-stub/setup.py << 'EOF'
from setuptools import setup, find_packages
setup(name='decord', version='0.6.0', packages=find_packages())
EOF
pip install /tmp/decord-stub
2. Several deps in pyproject.toml don’t build on ARM64
deepspeed, bitsandbytes, and liger_kernel either have no ARM64 wheels or require compilation that fails in this environment. Since we’re only doing inference, none of them are needed. We install deps manually and use pip install --no-deps -e . for the package itself.
3. MoonViT vision encoder downloads separately
The model config references moonshotai/MoonViT-SO-400M as a sub-model fetched from HF Hub at load time. Without authentication this hangs silently on rate limiting. Pass your HF token via both env vars:
bash
-e HF_TOKEN=your_token_here
-e HUGGING_FACE_HUB_TOKEN=your_token_here
Once downloaded it’s cached in the mounted ~/.cache/huggingface volume permanently.
4. Don’t use device_map='auto'
On the GB10 with 128GB unified memory, device_map='auto' runs a slow metadata analysis pass that can appear frozen for many minutes. The LocateAnythingWorker uses .to(device) directly which loads from cache in under a second.
5. Use LocateAnythingWorker, not raw model.generate()
The model uses Parallel Box Decoding (PBD) with a custom generation_mode parameter (fast/slow/hybrid). Raw model.generate() doesn’t invoke this correctly. Always use the official worker from the repo.
Full working docker run command
bash
docker rm -f locate-anything-vllm 2>/dev/null
docker run -d --gpus all \
--name locate-anything-vllm \
--shm-size=16g \
--ipc=host \
-p 8889:8889 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-e HF_TOKEN=your_token_here \
-e HUGGING_FACE_HUB_TOKEN=your_token_here \
--entrypoint /bin/bash \
vllm-node-tf5 \
-c "
git clone https://github.com/NVlabs/Eagle.git /eagle && \
cd /eagle/Embodied && \
pip install --no-cache-dir \
'transformers==4.57.1' \
'tokenizers==0.22.0' \
'sentencepiece==0.2.0' \
'accelerate==1.5.2' \
'peft==0.12.0' \
'numpy>=1.25,<2' \
'timm>=1.0.11' \
'einops' \
'einops-exts' \
'scipy>=1.10.0' \
'scikit-image' \
'opencv-python-headless' \
'pillow' \
'requests' \
'uvicorn' \
'fastapi' \
'lmdb' \
'filetype' \
'bitstring' \
'shortuuid' && \
mkdir -p /tmp/decord-stub/decord && \
cat > /tmp/decord-stub/decord/__init__.py << 'EOF'
class VideoReader:
def __init__(self, *a, **kw): pass
def __len__(self): return 0
def __getitem__(self, idx): return None
def get_avg_fps(self): return 30.0
EOF
cat > /tmp/decord-stub/setup.py << 'EOF'
from setuptools import setup, find_packages
setup(name='decord', version='0.6.0', packages=find_packages())
EOF
pip install /tmp/decord-stub && \
pip install --no-deps -e /eagle/Embodied && \
cat > /app.py << 'PYEOF'
import torch
import requests
import base64
from io import BytesIO
from PIL import Image
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse
import uvicorn
import sys
sys.path.insert(0, '/eagle/Embodied')
from locateanything_worker import LocateAnythingWorker
app = FastAPI()
model_id = 'nvidia/LocateAnything-3B'
print('Loading LocateAnythingWorker...', flush=True)
worker = LocateAnythingWorker(model_id)
print('Model ready.', flush=True)
@app.post('/v1/locate')
async def locate(request: Request):
body = await request.json()
image_url = body.get('image_url')
image_b64 = body.get('image_b64')
if not image_url and not image_b64:
return JSONResponse({'error': 'image_url or image_b64 required'}, status_code=400)
try:
if image_url:
resp = requests.get(image_url, timeout=10, headers={'User-Agent': 'Mozilla/5.0'})
image = Image.open(BytesIO(resp.content)).convert('RGB')
else:
image = Image.open(BytesIO(base64.b64decode(image_b64))).convert('RGB')
except Exception as e:
return JSONResponse({'error': f'Failed to load image: {e}'}, status_code=400)
task = body.get('task', 'detect')
query = body.get('query', '')
mode = body.get('mode', 'hybrid')
w, h = image.size
if task == 'detect':
categories = body.get('categories', query.split(',') if query else [])
result = worker.detect(image, categories, generation_mode=mode)
elif task == 'ground':
result = worker.ground_multi(image, query, generation_mode=mode)
elif task == 'ground_single':
result = worker.ground_single(image, query, generation_mode=mode)
elif task == 'detect_text':
result = worker.detect_text(image, generation_mode=mode)
elif task == 'ground_gui':
output_type = body.get('output_type', 'box')
result = worker.ground_gui(image, query, output_type=output_type, generation_mode=mode)
elif task == 'point':
result = worker.point(image, query, generation_mode=mode)
else:
return JSONResponse({'error': f'Unknown task: {task}'}, status_code=400)
answer = result['answer']
boxes = LocateAnythingWorker.parse_boxes(answer, w, h)
points = LocateAnythingWorker.parse_points(answer, w, h)
return {
'answer': answer,
'boxes': boxes,
'points': points,
'image_size': {'width': w, 'height': h},
'stats': result.get('stats'),
}
@app.get('/health')
async def health():
return {'status': 'ok'}
if __name__ == '__main__':
uvicorn.run(app, host='0.0.0.0', port=8889)
PYEOF
python3 /app.py
"
Testing — Python client
python
import requests
import base64
import json
with open("your_image.jpg", "rb") as f:
img_b64 = base64.b64encode(f.read()).decode()
response = requests.post(
"http://<your-ip>:8889/v1/locate",
json={
"task": "detect",
"categories": ["dice"],
"image_b64": img_b64
},
timeout=120
)
print(json.dumps(response.json(), indent=2))
Example output detecting dice in a scene:
json
{
"answer": "<ref>dice</ref><box><311><596><339><634></box>...<|im_end|>",
"boxes": [
{"x1": 279.9, "y1": 357.6, "x2": 305.1, "y2": 380.4},
{"x1": 291.6, "y1": 237.0, "x2": 320.4, "y2": 261.0},
{"x1": 370.8, "y1": 240.0, "x2": 397.8, "y2": 264.0},
{"x1": 405.0, "y1": 349.2, "x2": 432.9, "y2": 375.0},
{"x1": 825.3, "y1": 579.0, "x2": 899.1, "y2": 598.8}
],
"image_size": {"width": 900, "height": 600},
"stats": "num_tokens=34; generate_time=1.35s; tps=25.2; num_boxes=5; bps=3.71; switch_to_ar=0"
}
5 dice detected in 1.35 seconds, 3.7 BPS, zero autoregressive fallbacks — Parallel Box Decoding working correctly on GB10 Blackwell.
Supported tasks
| Task | Field | Description |
|---|---|---|
detect |
categories: ["cat", "car"] |
Multi-category object detection |
ground |
query: "people in red shirts" |
Phrase grounding, multiple instances |
ground_single |
query: "the dog on the left" |
Phrase grounding, single instance |
detect_text |
-– | Scene text / OCR detection |
ground_gui |
query: "search button" |
GUI element grounding |
point |
query: "the traffic light" |
Point-based localization |
Inference mode can be set per-request via "mode": "fast", "slow", or "hybrid" (default).