You managed to convince me :D
So I gave it a try, but I don’t see any improvement—in fact, it’s actually much slower. It almost feels like the Dflash isn’t even loaded; performance is worse in the tests and it feels sluggish too—each chunk comes out character by character. I’m probably missing a parameter, but I can’t figure out which one. Here’s my command:
./gb10_inference --server \
--model-dir ~/.cache/huggingface/hub/models--doth4580--Qwen3.8-27B-NVFP4-FULL/snapshots/1b49c1e2ad0b7621e9a991bc4e2dd10380cfc175 \
--model-name Acantic \
--port 8002 \
--max-seq-len 262144 \
--max-batch 3 \
--max-tokens 65536 \
--prefix-cache on \
--default-presence-penalty 1.5 \
--mtp=auto \
--spec-source dflash2-auto \
--draft-dir ~/.cache/huggingface/hub/models--doth4580--Qwen3.8-27B-DFlash2/snapshots/91a59627d8e504687daf82dd341d1e1dcf33671b
[mem-budget] 1b49c1e2ad0b7621e9a991bc4e2dd10380cfc175 (single node), max-seq-len 262144, batch 3, kv-cache bf16:
weights (per rank) ~0.0 GB
KV cache (~4 slots) ~64.0 GB
calibration transient ~8.0 GB (startup only, freed)
pools/workspaces (est) ~3.0 GB
----------------------------------------------
steady-state estimate ~67.0 GB of 121.6 GB physical
startup peak estimate ~75.0 GB of 121.6 GB physical
available NOW ~115.8 GB (other load on this box already eats the rest)
Loading model from /home/acantic/.cache/huggingface/hub/models--doth4580--Qwen3.8-27B-NVFP4-FULL/snapshots/1b49c1e2ad0b7621e9a991bc4e2dd10380cfc175 (streaming bf16)...
[mem] pre-shards: Some("VmRSS:\t 549384 kB") Some("MemAvailable: 120743956 kB")
Loading MTP head...
QUANTIZED artifact: 506 NVFP4 + 0 FP8 tensors (kept packed; dequantized in-register by the fused GEMV)
[mem] post-draft: Some("VmRSS:\t 549384 kB") Some("MemAvailable: 120743956 kB")
Shard 1/2: model-00001.safetensors
Shard 2/2: model-00002.safetensors
[mem] up#0: Some("VmRSS:\t12434696 kB") Some("MemAvailable: 107374132 kB") dev=0.0GB
[mem] up#8: Some("VmRSS:\t12590292 kB") Some("MemAvailable: 106469004 kB") dev=0.0GB
[mem] up#16: Some("VmRSS:\t12890148 kB") Some("MemAvailable: 105996152 kB") dev=0.0GB
[mem] up#24: Some("VmRSS:\t12976948 kB") Some("MemAvailable: 105437880 kB") dev=0.0GB
[mem] up#32: Some("VmRSS:\t13128384 kB") Some("MemAvailable: 105000536 kB") dev=0.0GB
[mem] up#40: Some("VmRSS:\t11708008 kB") Some("MemAvailable: 105178480 kB") dev=0.0GB
[mem] up#48: Some("VmRSS:\t11351808 kB") Some("MemAvailable: 105528432 kB") dev=0.0GB
[mem] up#56: Some("VmRSS:\t10526216 kB") Some("MemAvailable: 105245728 kB") dev=0.0GB
[mem] up#64: Some("VmRSS:\t10530764 kB") Some("MemAvailable: 104974844 kB") dev=0.0GB
[mem] up#72: Some("VmRSS:\t10348964 kB") Some("MemAvailable: 104642084 kB") dev=0.0GB
[mem] up#80: Some("VmRSS:\t 9331308 kB") Some("MemAvailable: 105237488 kB") dev=0.0GB
[mem] up#88: Some("VmRSS:\t 9672948 kB") Some("MemAvailable: 104926088 kB") dev=0.0GB
[mem] up#96: Some("VmRSS:\t 9420088 kB") Some("MemAvailable: 104632552 kB") dev=0.0GB
[mem] up#104: Some("VmRSS:\t 9229084 kB") Some("MemAvailable: 104474428 kB") dev=0.0GB
[mem] up#112: Some("VmRSS:\t 9116432 kB") Some("MemAvailable: 104276860 kB") dev=0.0GB
[mem] up#120: Some("VmRSS:\t 8072468 kB") Some("MemAvailable: 104923788 kB") dev=0.0GB
[mem] up#128: Some("VmRSS:\t 7874704 kB") Some("MemAvailable: 104742080 kB") dev=0.0GB
[mem] up#136: Some("VmRSS:\t 7976344 kB") Some("MemAvailable: 104420440 kB") dev=0.0GB
[mem] up#144: Some("VmRSS:\t 7812936 kB") Some("MemAvailable: 104145492 kB") dev=0.0GB
[mem] up#152: Some("VmRSS:\t 6718176 kB") Some("MemAvailable: 104805032 kB") dev=0.0GB
[mem] up#160: Some("VmRSS:\t 6561000 kB") Some("MemAvailable: 104724792 kB") dev=0.0GB
[mem] up#168: Some("VmRSS:\t 6535464 kB") Some("MemAvailable: 104482336 kB") dev=0.0GB
[mem] up#176: Some("VmRSS:\t 6455056 kB") Some("MemAvailable: 104228364 kB") dev=0.0GB
[mem] up#184: Some("VmRSS:\t 6505344 kB") Some("MemAvailable: 103938864 kB") dev=0.0GB
[mem] up#192: Some("VmRSS:\t 5312620 kB") Some("MemAvailable: 104637784 kB") dev=0.0GB
[mem] up#200: Some("VmRSS:\t 5327968 kB") Some("MemAvailable: 104429808 kB") dev=0.0GB
[mem] up#208: Some("VmRSS:\t 5183004 kB") Some("MemAvailable: 104220760 kB") dev=0.0GB
[mem] up#216: Some("VmRSS:\t 5293280 kB") Some("MemAvailable: 104011552 kB") dev=0.0GB
[mem] up#224: Some("VmRSS:\t 3964700 kB") Some("MemAvailable: 104689548 kB") dev=0.0GB
[mem] up#232: Some("VmRSS:\t 4105584 kB") Some("MemAvailable: 104445376 kB") dev=0.0GB
[mem] up#240: Some("VmRSS:\t 3899524 kB") Some("MemAvailable: 104234428 kB") dev=0.0GB
[mem] up#248: Some("VmRSS:\t 3913780 kB") Some("MemAvailable: 103902276 kB") dev=0.0GB
[mem] up#256: Some("VmRSS:\t 2763032 kB") Some("MemAvailable: 104533192 kB") dev=0.0GB
[mem] up#264: Some("VmRSS:\t 5511972 kB") Some("MemAvailable: 101276904 kB") dev=0.0GB
[mem] up#272: Some("VmRSS:\t 5693652 kB") Some("MemAvailable: 100958136 kB") dev=0.0GB
[mem] up#280: Some("VmRSS:\t 5991780 kB") Some("MemAvailable: 100595792 kB") dev=0.0GB
[mem] up#288: Some("VmRSS:\t 6061104 kB") Some("MemAvailable: 100035712 kB") dev=0.0GB
[mem] up#296: Some("VmRSS:\t 5555600 kB") Some("MemAvailable: 100163320 kB") dev=0.0GB
[mem] up#304: Some("VmRSS:\t 5793188 kB") Some("MemAvailable: 99732728 kB") dev=0.0GB
[mem] up#312: Some("VmRSS:\t 4429000 kB") Some("MemAvailable: 99850408 kB") dev=0.0GB
[mem] up#320: Some("VmRSS:\t 3052180 kB") Some("MemAvailable: 100219704 kB") dev=0.0GB
Loading MTP head...
[load] total 29.4s | ptx+jit 5.1s rope 0.1s | shards: read 17.0s cpu 3.4s repack-inline 0.0s upload 0.1s | assemble 23.9s (draft 0.0s fuse 20.3s repack 12.4s host-shard 0.0s upload 6.1s gwn-cpu 0.0s omma-repack 0.0s) | other 0.0s
[mem] post-pipeline: Some("VmRSS:\t 1970024 kB") Some("MemAvailable: 100984792 kB")
[mem] post-mxfp4-build: Some("VmRSS:\t 1970280 kB") Some("MemAvailable: 100984792 kB")
Context: --max-seq-len 262144 (model max 262144). KV cache ~51.5 GB at batch 3.
Loading tokenizer from /home/acantic/.cache/huggingface/hub/models--doth4580--Qwen3.8-27B-NVFP4-FULL/snapshots/1b49c1e2ad0b7621e9a991bc4e2dd10380cfc175/tokenizer.json...
[tokenizer] loaded chat template from /home/acantic/.cache/huggingface/hub/models--doth4580--Qwen3.8-27B-NVFP4-FULL/snapshots/1b49c1e2ad0b7621e9a991bc4e2dd10380cfc175/chat_template.jinja
Stop tokens: [248044, 248046] (config.json advertises 248044)
Calibrating MTP cost per depth, per context bucket...
ctx 2048 depth 2: a step costs 1.13x a decode (pays if it emits > 1.13 tok)
ctx 2048 depth 3: a step costs 1.19x a decode (pays if it emits > 1.19 tok)
ctx 2048 depth 4: a step costs 1.27x a decode (pays if it emits > 1.27 tok)
ctx 2048 depth 5: a step costs 1.34x a decode (pays if it emits > 1.34 tok)
ctx 2048 depth 6: a step costs 1.43x a decode (pays if it emits > 1.43 tok)
ctx 2048 depth 8: a step costs 1.58x a decode (pays if it emits > 1.58 tok)
ctx 8192 depth 2: a step costs 1.13x a decode (pays if it emits > 1.13 tok)
ctx 8192 depth 3: a step costs 1.21x a decode (pays if it emits > 1.21 tok)
ctx 8192 depth 4: a step costs 1.31x a decode (pays if it emits > 1.31 tok)
ctx 8192 depth 5: a step costs 1.39x a decode (pays if it emits > 1.39 tok)
ctx 8192 depth 6: a step costs 1.47x a decode (pays if it emits > 1.47 tok)
ctx 8192 depth 8: a step costs 1.64x a decode (pays if it emits > 1.64 tok)
ctx 32768 depth 2: a step costs 1.15x a decode (pays if it emits > 1.15 tok)
ctx 32768 depth 3: a step costs 1.26x a decode (pays if it emits > 1.26 tok)
ctx 32768 depth 4: a step costs 1.38x a decode (pays if it emits > 1.38 tok)
ctx 32768 depth 5: a step costs 1.50x a decode (pays if it emits > 1.50 tok)
ctx 32768 depth 6: a step costs 1.62x a decode (pays if it emits > 1.62 tok)
ctx 32768 depth 8: a step costs 1.87x a decode (pays if it emits > 1.87 tok)
ctx 131072 depth 2: a step costs 1.17x a decode (pays if it emits > 1.17 tok)
ctx 131072 depth 3: a step costs 1.34x a decode (pays if it emits > 1.34 tok)
ctx 131072 depth 4: a step costs 1.56x a decode (pays if it emits > 1.56 tok)
ctx 131072 depth 5: a step costs 1.79x a decode (pays if it emits > 1.79 tok)
ctx 131072 depth 6: a step costs 1.97x a decode (pays if it emits > 1.97 tok)
ctx 131072 depth 8: a step costs 2.41x a decode (pays if it emits > 2.41 tok)
MTP cost/depth: cached to /home/acantic/velogb10/mtp_calib/1b49c1e2ad0b7621e9a991bc4e2dd10380cfc175.json
[df2] DFlash2 round RESIDENT (spec-source=dflash2-auto) — serving via the S4F integrated round (b==1 lanes); MTP remains the fallback (standing directive)
MTP: auto (disables itself if no depth beats plain decode); depth starts at 4, re-picked from live acceptance; greedy requests verify by argmax (bitwise lossless), temp>0 requests by rejection sampling (distribution-exact).
Prefix cache: ON â a conversation's history is reused instead of re-prefilled (~3x faster follow-up turns). Cached turns are NOT bit-identical to cold ones: reuse re-chunks the prefill and cuBLAS picks a kernel per shape.
Attempting CUDA graph capture for batch sizes 1..=3... captured (smem 18 KB).
captured 3 GPU-sample graph(s).
[df2] draft-round CUDA graph captured (eager fallback via GB10_NO_DF2_GRAPH)
OpenAI-compatible server running on http://0.0.0.0:8002
Serving model: Acantic (GET /v1/models)
POST /v1/chat/completions max_batch=3 default max_tokens=65536
[req] params temp=0.7 top_p=0.8 top_k=20 max_tok=Some(1) rep_pen=None presence=None freq=None stream=false
[req] sync prompt_tokens=72 max_tokens=1 stop=[]
[req] ttft=1046.8ms plen=72
[req] done tok=2 (1.7 tok/s wall) finish=length
[req] params temp=0.7 top_p=0.8 top_k=20 max_tok=Some(1) rep_pen=None presence=None freq=None stream=false
[req] sync prompt_tokens=74 max_tokens=1 stop=[]
[req] ttft=760.1ms plen=74
[req] done tok=2 (2.2 tok/s wall) finish=length
[req] params temp=0.7 top_p=0.8 top_k=20 max_tok=Some(100) rep_pen=None presence=None freq=None stream=false
[req] sync prompt_tokens=65 max_tokens=100 stop=[]
[req] ttft=761.1ms plen=65
[req] done tok=37 (23.7 tok/s wall) finish=stop
[req] params temp=0.7 top_p=0.8 top_k=20 max_tok=Some(1) rep_pen=None presence=None freq=None stream=true
[req] stream prompt_tokens=53 max_tokens=1 stop=[]
[req] ttft=914.7ms plen=53
[req] done tok=2 (1.9 tok/s wall) finish=length stop_hit=false
[req] params temp=0.7 top_p=0.8 top_k=20 max_tok=Some(1) rep_pen=None presence=None freq=None stream=true
[req] stream prompt_tokens=53 max_tokens=1 stop=[]
[req] prefix hit (prompt checkpoint): 48/53 tokens cached, prefilling 5 (91% skipped)
[req] ttft=271.4ms plen=53
[req] done tok=2 (5.3 tok/s wall) finish=length stop_hit=false
[req] params temp=0.7 top_p=0.8 top_k=20 max_tok=Some(1) rep_pen=None presence=None freq=None stream=true
[req] stream prompt_tokens=53 max_tokens=1 stop=[]
[req] ttft=897.4ms plen=53
[req] done tok=2 (1.9 tok/s wall) finish=length stop_hit=false
[req] params temp=0.7 top_p=0.8 top_k=20 max_tok=Some(1) rep_pen=None presence=None freq=None stream=true
[req] stream prompt_tokens=53 max_tokens=1 stop=[]
[req] prefix hit (prompt checkpoint): 48/53 tokens cached, prefilling 5 (91% skipped)
[req] ttft=253.0ms plen=53
[req] done tok=2 (5.6 tok/s wall) finish=length stop_hit=false
...
I’ve tested with a real harness :
[req] params temp=0.1 top_p=0.8 top_k=20 max_tok=None rep_pen=None presence=None freq=None stream=true
[req] tools 86 offered: ["create_directory", "create_file", "create_new_jupyter_notebook", "create_new_workspace", "edit_notebook_file", "fetch_webpage", "file_search", "grep_search", "get_errors", "copilot_getNotebookSummary", "get_vscode_api", "github_repo", "github_text_search", "install_extension", "list_dir", "memory", "multi_replace_string_in_file", "read_file", "read_notebook_cell_output", "replace_string_in_file", "resolve_memory_file_uri", "run_notebook_cell", "run_vscode_command", "session_store_sql", "view_image", "vscode_askQuestions", "vscode_listCodeUsages", "vscode_renameSymbol", "vscode_searchExtensions_internal", "click_element", "create_and_run_task", "drag_element", "get_task_output", "get_terminal_output", "handle_dialog", "hover_element", "kill_terminal", "manage_todo_list", "navigate_page", "open_browser_page", "read_page", "run_in_terminal", "run_playwright_code", "runSubagent", "screenshot_page", "send_to_terminal", "terminal_last_command", "terminal_selection", "testFailure", "type_in_page", "activate_notebook_package_management", "configure_notebook", "configure_python_environment", "get_python_environment_details", "get_python_executable_details", "install_python_packages", "container-tools_get-config", "activate_dotnet_sdk_management_tools", "activate_dotnet_version_listing_tools", "find_dotnet_executable_path", "get_settings_info_for_dotnet_installation_management", "uninstall_vscode_owned_dotnet_runtime", "activate_git_commit_management_tools", "activate_branch_and_worktree_management", "activate_remote_repository_interaction", "activate_git_workflow_and_pr_management", "activate_pull_request_management_and_review", "mcp_gitkraken_cli_git_stash", "mcp_gitkraken_cli_issues_add_comment", "mcp_gitkraken_cli_issues_assigned_to_me", "mcp_gitkraken_cli_issues_create", "mcp_gitkraken_cli_issues_get_detail", "mcp_gitkraken_cli_repository_get_file_content", "mcp_pylance_mcp_s_pylanceAnalyze", "mcp_pylance_mcp_s_pylanceCheckSignatureCompatibility", "mcp_pylance_mcp_s_pylanceDocuments", "mcp_pylance_mcp_s_pylanceFileSyntaxErrors", "mcp_pylance_mcp_s_pylanceImports", "mcp_pylance_mcp_s_pylanceInstalledTopLevelModules", "mcp_pylance_mcp_s_pylanceInvokeRefactoring", "activate_fallback_mcp_pylance_mcp_s_pylanceLSP_1", "mcp_pylance_mcp_s_pylancePythonProfileExplorer", "mcp_gitkraken_cli_git_commit_composer", "mcp_gitkraken_cli_git_branch", "mcp_gitkraken_cli_gitkraken_workspace_list", "mcp_pylance_mcp_s_pylanceLSP"] tool_choice=None
[req] ttft=2035.1ms plen=314
[req] stream prompt_tokens=29936 max_tokens=65536 stop=[]
[req] ttft=42649.4ms plen=29936
[req] raw model output (74 chars): "Bonjour ! Je suis GitHub Copilot. Comment puis-je vous aider aujourd'hui ?"
[req] done tok=74 (1.5 tok/s wall) finish=stop stop_hit=false
[req] params temp=0.1 top_p=0.8 top_k=20 max_tok=None rep_pen=None presence=None freq=None stream=true
[req] tools 86 offered: ["create_directory", "create_file", "create_new_jupyter_notebook", "create_new_workspace", "edit_notebook_file", "fetch_webpage", "file_search", "grep_search", "get_errors", "copilot_getNotebookSummary", "get_vscode_api", "github_repo", "github_text_search", "install_extension", "list_dir", "memory", "multi_replace_string_in_file", "read_file", "read_notebook_cell_output", "replace_string_in_file", "resolve_memory_file_uri", "run_notebook_cell", "run_vscode_command", "session_store_sql", "view_image", "vscode_askQuestions", "vscode_listCodeUsages", "vscode_renameSymbol", "vscode_searchExtensions_internal", "click_element", "create_and_run_task", "drag_element", "get_task_output", "get_terminal_output", "handle_dialog", "hover_element", "kill_terminal", "manage_todo_list", "navigate_page", "open_browser_page", "read_page", "run_in_terminal", "run_playwright_code", "runSubagent", "screenshot_page", "send_to_terminal", "terminal_last_command", "terminal_selection", "testFailure", "type_in_page", "activate_notebook_package_management", "configure_notebook", "configure_python_environment", "get_python_environment_details", "get_python_executable_details", "install_python_packages", "container-tools_get-config", "activate_dotnet_sdk_management_tools", "activate_dotnet_version_listing_tools", "find_dotnet_executable_path", "get_settings_info_for_dotnet_installation_management", "uninstall_vscode_owned_dotnet_runtime", "activate_git_commit_management_tools", "activate_branch_and_worktree_management", "activate_remote_repository_interaction", "activate_git_workflow_and_pr_management", "activate_pull_request_management_and_review", "mcp_gitkraken_cli_git_stash", "mcp_gitkraken_cli_issues_add_comment", "mcp_gitkraken_cli_issues_assigned_to_me", "mcp_gitkraken_cli_issues_create", "mcp_gitkraken_cli_issues_get_detail", "mcp_gitkraken_cli_repository_get_file_content", "mcp_pylance_mcp_s_pylanceAnalyze", "mcp_pylance_mcp_s_pylanceCheckSignatureCompatibility", "mcp_pylance_mcp_s_pylanceDocuments", "mcp_pylance_mcp_s_pylanceFileSyntaxErrors", "mcp_pylance_mcp_s_pylanceImports", "mcp_pylance_mcp_s_pylanceInstalledTopLevelModules", "mcp_pylance_mcp_s_pylanceInvokeRefactoring", "activate_fallback_mcp_pylance_mcp_s_pylanceLSP_1", "mcp_pylance_mcp_s_pylancePythonProfileExplorer", "mcp_gitkraken_cli_git_commit_composer", "mcp_gitkraken_cli_git_branch", "mcp_gitkraken_cli_gitkraken_workspace_list", "mcp_pylance_mcp_s_pylanceLSP"] tool_choice=None
[req] stream prompt_tokens=30198 max_tokens=65536 stop=[]
[pool] trimmed 4.6 GB -> 3.0 GB of idle buffers
[req] prefix hit (prompt checkpoint): 29931/30198 tokens cached, prefilling 267 (99% skipped)
[req] ttft=1668.9ms plen=30198
[req] tool_call call_0 run_in_terminal({"command":"date","explanation":"Affiche la date et l'heure courantes.","goal":"Connaître l'heure actuelle","mode":"sync"})
[req] done tok=135 (9.8 tok/s wall) finish=tool_calls stop_hit=false
[req] params temp=0.1 top_p=0.8 top_k=20 max_tok=None rep_pen=None presence=None freq=None stream=true
[req] tools 86 offered: ["create_directory", "create_file", "create_new_jupyter_notebook", "create_new_workspace", "edit_notebook_file", "fetch_webpage", "file_search", "grep_search", "get_errors", "copilot_getNotebookSummary", "get_vscode_api", "github_repo", "github_text_search", "install_extension", "list_dir", "memory", "multi_replace_string_in_file", "read_file", "read_notebook_cell_output", "replace_string_in_file", "resolve_memory_file_uri", "run_notebook_cell", "run_vscode_command", "session_store_sql", "view_image", "vscode_askQuestions", "vscode_listCodeUsages", "vscode_renameSymbol", "vscode_searchExtensions_internal", "click_element", "create_and_run_task", "drag_element", "get_task_output", "get_terminal_output", "handle_dialog", "hover_element", "kill_terminal", "manage_todo_list", "navigate_page", "open_browser_page", "read_page", "run_in_terminal", "run_playwright_code", "runSubagent", "screenshot_page", "send_to_terminal", "terminal_last_command", "terminal_selection", "testFailure", "type_in_page", "activate_notebook_package_management", "configure_notebook", "configure_python_environment", "get_python_environment_details", "get_python_executable_details", "install_python_packages", "container-tools_get-config", "activate_dotnet_sdk_management_tools", "activate_dotnet_version_listing_tools", "find_dotnet_executable_path", "get_settings_info_for_dotnet_installation_management", "uninstall_vscode_owned_dotnet_runtime", "activate_git_commit_management_tools", "activate_branch_and_worktree_management", "activate_remote_repository_interaction", "activate_git_workflow_and_pr_management", "activate_pull_request_management_and_review", "mcp_gitkraken_cli_git_stash", "mcp_gitkraken_cli_issues_add_comment", "mcp_gitkraken_cli_issues_assigned_to_me", "mcp_gitkraken_cli_issues_create", "mcp_gitkraken_cli_issues_get_detail", "mcp_gitkraken_cli_repository_get_file_content", "mcp_pylance_mcp_s_pylanceAnalyze", "mcp_pylance_mcp_s_pylanceCheckSignatureCompatibility", "mcp_pylance_mcp_s_pylanceDocuments", "mcp_pylance_mcp_s_pylanceFileSyntaxErrors", "mcp_pylance_mcp_s_pylanceImports", "mcp_pylance_mcp_s_pylanceInstalledTopLevelModules", "mcp_pylance_mcp_s_pylanceInvokeRefactoring", "activate_fallback_mcp_pylance_mcp_s_pylanceLSP_1", "mcp_pylance_mcp_s_pylancePythonProfileExplorer", "mcp_gitkraken_cli_git_commit_composer", "mcp_gitkraken_cli_git_branch", "mcp_gitkraken_cli_gitkraken_workspace_list", "mcp_pylance_mcp_s_pylanceLSP"] tool_choice=None
[req] stream prompt_tokens=30340 max_tokens=65536 stop=[]
[req] prefix hit (prompt checkpoint): 30193/30340 tokens cached, prefilling 147 (100% skipped)
[req] ttft=1574.1ms plen=30340
[req] raw model output (80 chars): "Il est **21h08** (heure d'été centrale européenne, CEST), mercredi 26 août 2026."
[req] done tok=79 (9.1 tok/s wall) finish=stop stop_hit=false
[req] done tok=726 (6.6 tok/s wall) finish=stop stop_hit=false
[req] done tok=880 (7.2 tok/s wall) finish=stop stop_hit=false
Here is the llama-benchy test :
| model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:--------|---------------:|---------------------:|-------------:|--------------:|---------------:|-----------------:|
| Acantic | pp2048 | 97576.46 ± 29790.27 | | 189.61 ± 8.52 | 22.07 ± 8.52 | 2854.62 ± 30.20 |
| Acantic | tg32 | 33.48 ± 7.83 | 34.33 ± 8.39 | | | |
| Acantic | pp2048 | 210146.88 ± 58732.71 | | 177.75 ± 3.31 | 10.21 ± 3.31 | 2863.68 ± 5.27 |
| Acantic | tg128 | 29.08 ± 3.20 | 29.33 ± 3.40 | | | |
| Acantic | pp2048 @ d4096 | 302042.62 ± 56542.04 | | 186.88 ± 4.11 | 19.33 ± 4.11 | 6857.66 ± 119.69 |
| Acantic | tg32 @ d4096 | 29.37 ± 7.50 | 29.96 ± 7.99 | | | |
| Acantic | pp2048 @ d4096 | 302316.57 ± 68002.24 | | 186.66 ± 4.80 | 19.12 ± 4.80 | 6687.26 ± 69.92 |
| Acantic | tg128 @ d4096 | 23.54 ± 1.12 | 24.00 ± 0.82 | | | |
llama-benchy (0.4.1.dev1+ge9be34457)
date: 2026-08-26 21:00:27 | latency mode: generation
I’ve tested with an other benchmark :
Workload matrix v1 — endpoint http://127.0.0.1:8002 (model id: Acantic)
greedy, fresh single-turn prompts, decode net of prefill (two-call delta)
code (EN) : 23.5 tok/s
code (DE) : 28.9 tok/s
technical explain (FR) : 23.3 tok/s
reasoning (FR) : 32.0 tok/s
free prose (EN) : 18.6 tok/s
free prose (FR) : 20.1 tok/s
free prose (DE) : 19.8 tok/s
But the main issue bothering me is that it makes the case run really hot; as soon as I start a benchmark, the temperature shoots up to 85°C—I’ve never experienced that before :
nvidia-smi --query-gpu=power.draw,temperature.gpu,utilization.gpu,clocks.sm --format=csv,noheader
89.67 W, 84, 96 %, 2392 MHz
That said, bravo for the simplicity, no need to clone an entire repository, a tiny binary and a folder, simple settings, it’s good :)
