Pulled the full weights (BF16) and ran it under an up-to-date (as of about 4pm this afternoon) spark-vllm-docker image. Recipe was as follows:
# Recipe: ornith-ai/Ornith-1.5-35B-A3B (BF16)
# Conservative single-Spark / GX10 profile for ~70 GB BF16 weights
# on 128 GB unified memory. Start here, then compare vs NVFP4.
recipe_version: "1"
name: Ornith-1.5-35B-A3B-BF16
description: |
Ornith-1.5-35B-A3B BF16 on single DGX Spark / Asus GX10.
Conservative memory settings for unified memory stability.
Use --solo. Prefer NVFP4 recipe for higher context/concurrency later.
model: ornith-ai/Ornith-1.5-35B-A3B
container: vllm-node # or whatever your current spark-vllm image tag is
# solo_only: true # uncomment if you want the runner to enforce solo
defaults:
port: 8000
host: 0.0.0.0
tensor_parallel: 1
gpu_memory_utilization: 0.7 # start conservative; raise carefully (0.50–0.65)
max_model_len: 262144 # start at 128k; try 262144 only after it is stable
max_num_seqs: 4
max_num_batched_tokens: 2048
env:
VLLM_MARLIN_USE_ATOMIC_ADD: "1"
# Optional GB10 helpers (uncomment if your image benefits):
# VLLM_USE_V1: "1"
# CUTE_DSL_ARCH: "sm_121a"
# FLASHINFER_DISABLE_VERSION_CHECK: "1"
command: |
vllm serve ornith-ai/Ornith-1.5-35B-A3B \
--served-model-name Ornith-1.5-35B-A3B \
--host {host} \
--port {port} \
--tensor-parallel-size {tensor_parallel} \
--max-model-len {max_model_len} \
--gpu-memory-utilization {gpu_memory_utilization} \
--max-num-seqs {max_num_seqs} \
--max-num-batched-tokens {max_num_batched_tokens} \
--kv-cache-dtype fp8_e4m3 \
--enable-prefix-caching \
--enable-chunked-prefill \
--enable-auto-tool-choice \
--tool-call-parser qwen3_xml \
--reasoning-parser qwen3 \
--trust-remote-code \
--override-generation-config '{{"temperature":1.0,"top_p":0.95,"top_k":20}}'
Per the Orinth model card they recommend temperature:0.6 for general tasks and 1.0 for benchmarking. I ignored the AI suggestions to start gpu_memory_utilization at 0.55 and pushed it to 0.7. There was nothing else running on the node, so why not? I also opted to max out max_model_len instead of starting conservatively. It should be noted that Orinth’s model card states that context window can be pushed to 1MM via Rope. I did not enable that, but might test it.
A word about my position on benchmarks before I post the results. I find them useful only insofar as they give me a rough idea of where to set my expectations for a new model. I don’t chase benchmark numbers, nor try to tune to max them out. My primary concern/focus is on getting actual work done. Many models benchmark well, but once I start pushing a handful of concurrent Hermes profiles doing complex coding through them, they fall apart in interesting ways. I have not tried to push any complex coding/concurrency through Orinth-1.5-35B-A3B…yet.
All benchmarks are based on the latest versions of the tools as of late this afternoon. As benchmark suites are updated, current results may not be directly comparable to results from older versions. Just something to be aware of. That being said:
llama-benchy results:
| model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:-------------------|----------------:|-----------------:|-------------:|-----------------:|-----------------:|-----------------:|
| Ornith-1.5-35B-A3B | pp2048 | 3261.57 ± 276.86 | | 635.89 ± 55.90 | 632.86 ± 55.90 | 635.89 ± 55.90 |
| Ornith-1.5-35B-A3B | tg32 | 31.07 ± 0.02 | 32.07 ± 0.02 | | | |
| Ornith-1.5-35B-A3B | pp2048 @ d4096 | 3134.63 ± 36.15 | | 1963.43 ± 22.37 | 1960.40 ± 22.37 | 1963.43 ± 22.37 |
| Ornith-1.5-35B-A3B | tg32 @ d4096 | 30.84 ± 0.01 | 31.00 ± 0.00 | | | |
| Ornith-1.5-35B-A3B | pp2048 @ d8192 | 3026.43 ± 10.88 | | 3386.82 ± 12.22 | 3383.79 ± 12.22 | 3386.82 ± 12.22 |
| Ornith-1.5-35B-A3B | tg32 @ d8192 | 30.67 ± 0.03 | 31.00 ± 0.00 | | | |
| Ornith-1.5-35B-A3B | pp2048 @ d16384 | 2876.20 ± 15.17 | | 6412.13 ± 33.65 | 6409.10 ± 33.65 | 6412.70 ± 33.68 |
| Ornith-1.5-35B-A3B | tg32 @ d16384 | 30.33 ± 0.07 | 31.00 ± 0.00 | | | |
| Ornith-1.5-35B-A3B | pp2048 @ d32768 | 2649.70 ± 13.33 | | 13143.35 ± 66.23 | 13140.32 ± 66.23 | 13144.78 ± 66.91 |
| Ornith-1.5-35B-A3B | tg32 @ d32768 | 29.70 ± 0.07 | 30.00 ± 0.00 | | | |
spark-bench results:
## TrueScore 91.4/100 — ⭐⭐⭐⭐⭐ Excellent (grade A)
| headline score | value | meaning |
|----------------|------:|---------|
| Capability Score | 88.5 | quality/correctness without speed penalty |
| Operational Score | 84.8 | efficiency + latency/responsiveness |
| **TrueScore** | **91.4** | combined deployment score |
| component | score | TrueScore weight |
|-----------|------:|-----------------:|
| quality | 88.5 | 55% |
| calibration | 96.0 | 25% |
| reliability | 96.4 | 15% |
| efficiency | 100.0 | 2% |
| responsiveness | 78.3 | 4% |
Median turn latency 5.53s · 76 scenarios · thinking off
## Trial Statistics
| metric | value | meaning |
|--------|------:|---------|
| Methodology | v6.7.1-full | scenario and grader contract |
| Run Valid | yes | transport error rate 0.0% |
| Pass@1 | 97.4% | scenarios passing (≥50%) on at least 1 repeat |
| Pass@K | 94.7% | scenarios passing on ALL repeats |
| Reliability Gap | 2.6% | Pass@1 − Pass@K (flakiness cost) |
| Score StdDev | 0.19 | cross-scenario score spread |
| Scenario StdDev | 0.019 | mean per-scenario repeat variance |
## Domain breakdown
| domain | group | n | quality | reliability |
|--------|-------|--:|--------:|------------:|
| robustness | calibration | 4 | 100.0 | 100.0 |
| safety | calibration | 11 | 96.0 | 91.9 |
| agentic | capability | 12 | 97.7 | 98.6 |
| classification | capability | 1 | 83.3 | 66.7 |
| code | capability | 14 | 85.5 | 95.7 |
| composition | capability | 2 | 100.0 | 100.0 |
| instruction | capability | 9 | 89.8 | 96.7 |
| long_context | capability | 2 | 55.9 | 100.0 |
| planning | capability | 5 | 80.7 | 100.0 |
| structured | capability | 5 | 96.5 | 100.0 |
| tool_use | capability | 6 | 81.0 | 93.3 |
| visual | capability | 5 | 77.0 | 96.6 |
tool-eval-bench results:
Tool-Call Benchmark — Ornith-1.5-35B-A3B
- **Run ID**: `2026-08-19T23-41-00.601055Z_739f7600`
- **Date**: `2026-08-20T00:04:18.460537+00:00`
- **tool-eval-bench**: `v2.5.1.dev35+g84757dd4f`
- **Final Score**: **91** / 100
- **Total Points**: 125 / 138
- **Rating**: ★★★★★ Excellent
- **Tool Definition Overhead**: ~4,742 tokens (52 tools, 18,970 chars)
- **Deployability**: **72** / 100 (α=0.7)
- **Quality**: 91 / 100
- **Responsiveness**: 29 / 100 (median turn: 5.5s)
> [!WARNING]
> **1 safety-critical failure(s) detected:**
> - TC-35 (Contradictory Parameters): Answered in the wrong unit instead of recognizing the Kelvin identity.
## Run Context
| Parameter | Value |
|---|---|
| Backend | vllm |
| Server | `http://***:8000` |
| Model (API) | `Ornith-1.5-35B-A3B` |
| Model (Root) | `ornith-ai/Ornith-1.5-35B-A3B` |
| Temperature | 0.0 |
| Seed | 42 |
| Max Turns | 8 |
| Timeout | 120.0s |
| Scenarios | all (69) |
| Parallel | 1 (sequential) |
| Error Rate | 0.0 |
| Thinking | enabled |
## Inference Engine
| Property | Value |
|---|---|
| Engine | vLLM 0.27.2rc1.dev256+geac636a7f.d20260819 |
| Max Model Length | 262,144 |
| Host | `palapa` |
| Platform | `Linux-6.17.0-1029-nvidia-aarch64-with-glibc2.39` |
| Python | 3.12.3 |
## Category Scores
| Category | Earned | Max | Percent |
|---|---|---|---|
| Tool Selection | 6 | 6 | 100% |
| Parameter Precision | 6 | 6 | 100% |
| Multi-Step Chains | 8 | 8 | 100% |
| Restraint & Refusal | 6 | 6 | 100% |
| Error Recovery | 5 | 6 | 83% |
| Localization | 6 | 6 | 100% |
| Structured Reasoning | 6 | 6 | 100% |
| Instruction Following | 8 | 10 | 80% |
| Context & State | 16 | 20 | 80% |
| Code Patterns | 6 | 6 | 100% |
| Safety & Boundaries | 22 | 26 | 85% |
| Toolset Scale | 8 | 8 | 100% |
| Autonomous Planning | 4 | 6 | 67% |
| Creative Composition | 6 | 6 | 100% |
| Structured Output | 12 | 12 | 100% |
I have the official NVFP4 weights downloading at the moment. Once those complete, I’ll switch over to that and re-run the benchmarks for comparison.