Trouble with Llama 70b 3.3 Instruct FP8 Model at 3 tokens per second

[jperez10]

Hi, can anyone help me with tuning my model? I see so many people here posting about larger models at higher tokens per second. I am currently getting 2 - 3 tokens per second with my Spark setup. Here is my setup:

DGX Spark (GB10) – Llama 3.3 70B FP8 Performance + Memory Behavior

Hardware

• System: NVIDIA DGX Spark

• GPU: NVIDIA GB10 (128GB unified memory)

• Driver: 580.126.09

• CUDA: 13.0

• OS: Ubuntu (GNU/Linux 6.14.x-nvidia aarch64)

---

Stack Architecture

• Model Server: vLLM (Docker)

• Model: RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic

• Web UI: Open WebUI (Docker)

• API Gateway: Custom LLM gateway (Docker)

• Embeddings: sentence-transformers/all-MiniLM-L6-v2 (vLLM embed)

• Deployment: Docker Compose + systemd

• Networking: Internal Docker network, Cloudflare Tunnel for public access

---

Current vLLM Runtime Configuration

docker inspect vllm --format '{{range .Config.Cmd}}{{.}} {{end}}'

Relevant flags:

--model RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic
--served-model-name Llama-3.3-70B-Instruct
--dtype auto
--gpu-memory-utilization 0.92
--max-model-len 16384
--max-num-seqs 4
--max-num-batched-tokens 16384
--swap-space 16
--enforce-eager
--tensor-parallel-size 1

---

Observed Performance

Generation throughput

• ~2–3 tokens/sec

• GPU utilization ~95–96% during decode

nvidia-smi (during generation)

GPU Util: 96%
Power: ~46W
Memory-Usage: Not supported (unified memory)

---

System Memory Behavior

During generation:

free -h

Mem: 119Gi total
Used: 119Gi
Free: ~1Gi
Available: ~350–500Mi

swapon --show

/swapfile 16G
Used: ~3–4.7G

Observations:

• RAM fully pinned at 119Gi

• 3–5Gi swap in use

• Swap does not always increase during short generation

• Running swapoff -a && swapon -a frequently results in a Killed message (likely OOM killer terminating a large process)

---

Tuning Attempts So Far

1️⃣ Reduced context length

• From 32768 → 16384

• Result: Slight improvement, but memory still fully pinned

2️⃣ Reduced max batched tokens

• From 32768 → 16384

• Result: Some improvement, still near memory cliff

3️⃣ Increased GPU memory utilization

• From 0.85 → 0.92

• Result: Improved GPU occupancy but worsened system memory pressure

4️⃣ Swap reset test

• swapoff -a && swapon -a

• Often results in process being killed under pressure

---

Current Stability Status

• vLLM runs successfully

• No immediate OOM at startup

• System operates extremely close to memory limits

• Unified memory heavily utilized

• Swap consistently in use

• Throughput remains ~2–3 tok/sec

---

Questions for the Community

1. Is 2–3 tok/sec expected for FP8 70B on single GB10?

2. Is unified memory + swap usage normal at this level with FP8?

3. Would switching to NF4 4-bit (bitsandbytes) materially improve:

   • KV cache headroom

   • Unified memory pressure

   • Decode throughput?

4. Is --enforce-eager required for stability at this memory level, or safe to remove?

5. Are there GB10-specific vLLM tuning flags recommended beyond:

   • --max-model-len

   • --max-num-seqs

   • --max-num-batched-tokens

   • --gpu-memory-utilization

[coder543]

You’re getting low speed because Llama 70b has 70b active parameters. The other large models that you see people running are MoE models that have more total parameters, but only a small number of parameters activate for each token. GPT-OSS 120B has 120B total parameters, but it only activates about 5B parameters per token since it is a MoE, not a dense model.

If you are willing to use llama-server, it is possible to get much better performance out of Llama 70B by using what is called a draft model, where it uses a small model to predict tokens, and then the big model verifies those. Draft models do not affect output quality, because the big model is ultimately in charge of every output token, but draft models can speed things up.

Here is an example command:

llama-server \
    -ngl 999 \
    -fit off \
    --parallel 1 \
    --flash-attn on \
    --draft-min 2 \
    --draft-max 8 \
    --no-warmup \
    --no-mmap \
    --jinja \
    --model llama-3.3-70b-instruct-ud-q8_k_xl.gguf \
    --model-draft llama-3.2-1b-instruct-ud-q8_k_xl.gguf \
    --temp 0.0 \
    -c 80000

I am using the unsloth gguf files here.

In general chat, I’m getting about 7 to 8 tokens per second, and for generating code, it is much faster at upwards of 18 tokens per second.

But, I do not recommend using such an old, slow model for anything. There are much, much better models out there.

[jperez10]

Thank you @coder543 . I will have a look at this. I am using a production api on this machine so i was hoping to not have to change the whole setup. Will the Builds from the NVIDIA playbook work for the Meta 70b model? The one here: GPU-optimized AI, Machine Learning, & HPC Software | NVIDIA NGC | NVIDIA NGC

According to AI, the above setup would improve speed but rearchitect my entire gateway and i already have multiple ai agents using it.

[coder543]

I don’t work for Nvidia, so I have no clue what their playbooks do. Either way, I strongly discourage using Llama 3.3 70B. It is not a useful model in 2026.

[jperez10]

@coder543

You’re getting low speed because Llama 70b has 70b active parameters. The other large models that you see people running are MoE models that have more total parameters, but only a small number of parameters activate for each token. GPT-OSS 120B has 120B total parameters, but it only activates about 5B parameters per token since it is a MoE, not a dense model.

If you are willing to use llama-server, it is possible to get much better performance out of Llama 70B by using what is called a draft model, where it uses a small model to predict tokens, and then the big model verifies those. Draft models do not affect output quality, because the big model is ultimately in charge of every output token, but draft models can speed things up.

Here is an example command:

llama-server \
    -ngl 999 \
    -fit off \
    --parallel 1 \
    --flash-attn on \
    --draft-min 2 \
    --draft-max 8 \
    --no-warmup \
    --no-mmap \
    --jinja \
    --model llama-3.3-70b-instruct-ud-q8_k_xl.gguf \
    --model-draft llama-3.2-1b-instruct-ud-q8_k_xl.gguf \
    --temp 0.0 \
    -c 80000

I am using the unsloth gguf files here.

In general chat, I’m getting about 7 to 8 tokens per second, and for generating code, it is much faster at upwards of 18 tokens per second.

But, I do not recommend using such an old, slow model for anything. There are much, much better models out there.

What do you recommend?

[coder543]

GPT-OSS-120B is a safe starting place, as it is a much better, much more well-rounded model, and it is very well supported. You could also look into Nemotron-3-Nano, Qwen3-Next, Qwen3-Coder-Next, GLM-4.7-Flash, Step-3.5-Flash, and MiniMax-M2.5, depending on exactly what you need.

[jperez10]

Will 120b run on a Spark Mini Desktop? If my 70b models are doing 2 to 3 tokens per second, wont OSS 120b run worse? Willing to try this and maybe do a lower Quant of it.

[coder543]

As I said, it only activates about 5B parameters for every token, which is a lot less than 70B. So, yes, it will run, and it will run much, much faster. The native quant is 4-bit, so it is already a compact model.

[norman.2]

Also have a look at GitHub - eugr/spark-vllm-docker: Docker configuration for running VLLM on dual DGX Sparks for easier setup times :)

[jperez10]

Thank you!

[notmy.reward438]

These all work 100% well with llama.cpp? I tried with with vLLM and only two of them are really usable. Other go crazy, calls tools wrong, etc.

[jperez10]

Can I run an API server with llama.cpp? My use case is to run a dedicated server for API calls. I am summarizing meetings, doing lead enrichment, etc.

[Balaxxe]

Highly recommend for you to just us LM Studio as it uses lama.cpp as it’s backend and has a nice gui for you to interface with.

Especially if you only have one spark.

Easy to host an api server as well - check youtube as there are a plethora of quick setup guides.

It is very straight forward.

[notmy.reward438]

llama.cpp itself is pretty much an API.

Of course, you can attach something like Flask/FastAPI/etc. to it if you want some other processing.

[notmy.reward438]

Personally would not recommend GPT OSS. I would recommend Qwen3 or GLM.

[SkyWanderer]

It may also be useful to compare your local results against latency and throughput numbers from hosted Llama 3.3 70B deployments. eg

Llama 3.3 70B Benchmark Results Across Providers | Benchscope

That should give you a practical reference point for what different serving stacks and provider environments are achieving

[tudor5]

1. defo run stuff from the qwen family

2. ~3 tok/s on dense 70B FP8 at batch 1 is the expected ceiling on a GB10, no misconfig there. Decode reads all weights once per token, so max tok/s ≈ bandwidth ÷ bytes per token → 273 GB/s ÷ ~70 GB ≈ 3.9 (~2.7–3 real). Prefill is fine because it’s compute-bound, which the Spark has plenty of.

The MoE suggestions above are right: decode cost is active params, not total. Dense 70B reads ~70 GB/token; gpt-oss-120b activates ~5B, so it reads a few GB/token and flies. Other levers, best first: harder quant (Q4 70B ~40 GB → ~6–7 tok/s), speculative decoding (NVIDIA Spark playbook), and batching (one weight read serves the whole batch).

It really comes down to workload shape:

• Long input → short output (RAG, summarization, extraction, long-context Q&A) — prefill-bound, so bandwidth barely matters. Spark is good here

• Short input → long output, single user (interactive chat, long-form generation), decode-bound, this is what you’re hitting. Spark’s worst case.

• High concurrency / batched (serving many users, batch jobs) — one weight read serves the whole batch, so aggregate throughput scales. Spark == good.

Fast single-user, long-output chat on a dense 70B is the Spark’s worst case, go MoE/quant/spec-decoding to work around it.