CohereLabs/command-a-plus-05-2026-w4a4 · Hugging Face
I created and ran a Docker image based on spark-vllm-docker by adding uv pip install cohere_melody>=0.9.0 to the Dockerfile.
The recipe I used is as follows:
# Recipe: CohereLabs/command-a-plus-05-2026-w4a4
recipe_version: "1"
name: CohereLabs/command-a-plus-05-2026-w4a4
description: vLLM serving CohereLabs/command-a-plus-05-2026-w4a4
# HuggingFace model to download (optional, for --download-model)
model: CohereLabs/command-a-plus-05-2026-w4a4
# Container image to use
container: vllm-node-cohere
# Can only be run in a cluster
cluster_only: true
# No mods required
mods: []
# Default settings (can be overridden via CLI)
defaults:
port: 8000
host: 0.0.0.0
tensor_parallel: 2
gpu_memory_utilization: 0.8
max_model_len: 131072
# Environment variables
env: {}
# The vLLM serve command template
command: |
vllm serve CohereLabs/command-a-plus-05-2026-w4a4 \
--trust-remote-code \
--port {port} \
--host {host} \
--gpu-memory-utilization {gpu_memory_utilization} \
-tp {tensor_parallel} \
--distributed-executor-backend ray \
--max-model-len {max_model_len} \
--load-format fastsafetensors \
--enable-auto-tool-choice \
--tool-call-parser cohere_command4 \
--reasoning-parser cohere_command4
llama-benchy
| model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:---------------------------------------|-----------------:|-----------------:|-------------:|------------------:|------------------:|------------------:|
| CohereLabs/command-a-plus-05-2026-w4a4 | pp2048 | 3638.57 ± 95.89 | | 706.60 ± 14.89 | 563.25 ± 14.89 | 706.60 ± 14.89 |
| CohereLabs/command-a-plus-05-2026-w4a4 | tg32 | 18.22 ± 0.02 | 19.00 ± 0.00 | | | |
| CohereLabs/command-a-plus-05-2026-w4a4 | ctx_pp @ d4096 | 3057.09 ± 143.71 | | 1486.25 ± 65.26 | 1342.90 ± 65.26 | 1486.25 ± 65.26 |
| CohereLabs/command-a-plus-05-2026-w4a4 | ctx_tg @ d4096 | 17.46 ± 0.05 | 18.00 ± 0.00 | | | |
| CohereLabs/command-a-plus-05-2026-w4a4 | pp2048 @ d4096 | 2498.73 ± 11.81 | | 962.98 ± 3.89 | 819.63 ± 3.89 | 962.98 ± 3.89 |
| CohereLabs/command-a-plus-05-2026-w4a4 | tg32 @ d4096 | 17.46 ± 0.04 | 18.00 ± 0.00 | | | |
| CohereLabs/command-a-plus-05-2026-w4a4 | ctx_pp @ d8192 | 2828.04 ± 14.49 | | 3040.13 ± 14.86 | 2896.78 ± 14.86 | 3040.13 ± 14.86 |
| CohereLabs/command-a-plus-05-2026-w4a4 | ctx_tg @ d8192 | 17.40 ± 0.01 | 18.00 ± 0.00 | | | |
| CohereLabs/command-a-plus-05-2026-w4a4 | pp2048 @ d8192 | 2349.46 ± 15.36 | | 1015.08 ± 5.69 | 871.73 ± 5.69 | 1015.08 ± 5.69 |
| CohereLabs/command-a-plus-05-2026-w4a4 | tg32 @ d8192 | 17.44 ± 0.04 | 18.00 ± 0.00 | | | |
| CohereLabs/command-a-plus-05-2026-w4a4 | ctx_pp @ d16384 | 2589.35 ± 8.91 | | 6470.87 ± 21.78 | 6327.52 ± 21.78 | 6472.62 ± 21.69 |
| CohereLabs/command-a-plus-05-2026-w4a4 | ctx_tg @ d16384 | 17.31 ± 0.01 | 18.00 ± 0.00 | | | |
| CohereLabs/command-a-plus-05-2026-w4a4 | pp2048 @ d16384 | 2091.71 ± 53.75 | | 1123.11 ± 25.58 | 979.76 ± 25.58 | 1124.93 ± 25.58 |
| CohereLabs/command-a-plus-05-2026-w4a4 | tg32 @ d16384 | 16.68 ± 0.85 | 18.00 ± 0.00 | | | |
| CohereLabs/command-a-plus-05-2026-w4a4 | ctx_pp @ d32768 | 2374.20 ± 8.82 | | 13945.24 ± 51.35 | 13801.89 ± 51.35 | 13948.28 ± 51.50 |
| CohereLabs/command-a-plus-05-2026-w4a4 | ctx_tg @ d32768 | 17.13 ± 0.03 | 18.00 ± 0.00 | | | |
| CohereLabs/command-a-plus-05-2026-w4a4 | pp2048 @ d32768 | 1813.92 ± 30.17 | | 1272.71 ± 19.00 | 1129.36 ± 19.00 | 1275.44 ± 19.12 |
| CohereLabs/command-a-plus-05-2026-w4a4 | tg32 @ d32768 | 17.09 ± 0.06 | 18.00 ± 0.00 | | | |
| CohereLabs/command-a-plus-05-2026-w4a4 | ctx_pp @ d65535 | 2038.10 ± 7.74 | | 32298.77 ± 121.83 | 32155.42 ± 121.83 | 32304.09 ± 122.08 |
| CohereLabs/command-a-plus-05-2026-w4a4 | ctx_tg @ d65535 | 16.77 ± 0.06 | 18.00 ± 0.00 | | | |
| CohereLabs/command-a-plus-05-2026-w4a4 | pp2048 @ d65535 | 1364.45 ± 2.37 | | 1644.32 ± 2.60 | 1500.97 ± 2.60 | 1649.53 ± 2.63 |
| CohereLabs/command-a-plus-05-2026-w4a4 | tg32 @ d65535 | 15.78 ± 0.94 | 17.00 ± 0.82 | | | |
| CohereLabs/command-a-plus-05-2026-w4a4 | ctx_pp @ d100000 | 1778.86 ± 10.18 | | 56360.87 ± 320.60 | 56217.52 ± 320.60 | 56369.04 ± 320.48 |
| CohereLabs/command-a-plus-05-2026-w4a4 | ctx_tg @ d100000 | 16.26 ± 0.23 | 17.67 ± 0.47 | | | |
| CohereLabs/command-a-plus-05-2026-w4a4 | pp2048 @ d100000 | 1076.68 ± 5.93 | | 2045.55 ± 10.51 | 1902.20 ± 10.51 | 2053.64 ± 9.31 |
| CohereLabs/command-a-plus-05-2026-w4a4 | tg32 @ d100000 | 15.84 ± 0.71 | 17.00 ± 0.82 | | | |