You need the specific Marlin atomic add environment variable to reach optimal performance on Marlin with Int4 variants
VLLM hard quits at the specified max model length, so I have no idea how you’re getting any response at all from a 285k prompt when your max model length is 256K.
Before chasing a config fix, let’s separate the two issues:
your 285K prompt exceeds max-model-len (262k), and that’s is causing your performance variation from 1.8 to 4.9 because of 2 request’s decode steps are nterleaved with “contaminated” request’s prefill due to your max-batched-tokens configuration. I would reduce it to 32k or 64k, but would not expect huge improvements, but that’s one first test.
But I really feel the best scenario is going multi-node, because two Sparks over the ConnectX-7 link give you ~546 GB/s aggregate bandwidth and double the KV capacity, which is the only change that meaningfully scales both the context and the decode rate at that length. Dense models scale really well on Spark clusters.
I use it for coding with open code (I try to use it…) this is the reason why I have 262k context… But I heard often I should put it lower around 100k… ClaudeCode has the context 200k…
Yea the same one from Spark Arena I saw it couple minutes ago and try to run it right now.
I tried hardest to get anything useful out of 27b. In theory it’s great but I feel it is incredibly benchmaxxed by Qwen first, and way too slow and has verbose thinking second. There is a reason we are getting almost all new models as MoE (except perhaps Mistral 3.5 128b which is so slow it’s totally irrelevant). For coding I would say Nemotron Cascade 2 is amazing as a build model. 60 tps tg, very fast and 1m tokens window, similar to Super speed does not degrade with context. And you can have 5m tokens cache on one spark to run fleet of subagents. Only like 12gb per 1m q8 tokens. I would use it this way: setup a cloud model like deepseek V4 flash or pro as your planning and review model, you need big brains, and setup you cascade as your builder and subagents model. Tool calling is tge most of the expense and totally can be done by local model. If you want to stay purely local I would so for Nemotron 3 Super. Again 1m tokens window, hybrid mamba attention guarantee stable speed on context large and small, small tokens let you have 2m+ total cache on spark even after 75gb weights, meaning you can hot cache you codebade and docs in karpathi wiki once and all next requests hit prefix cache and are very fast. Hard to do with 256k models. I am using 1m models exclusively. Lotsa code and documentation. Stuff a massive prompt, let is warm the cache, slow start fast ride. Lol.
Speed wise: best I could get out of 27 b on production ready setup (not nightly builds that can produce one benchmark out of 5 it’s crashes vllm) is about 21-23 tg tok/s at small prompt. Super is around 22-23 tg t/s with my maxxed optimization recipe. But once context and cache grows 27 drops to 15, then sub 10. Super stays at 22 and when prefix cache hits above 75% it goes to 25-27 quite often. It does not slow down. At 120b A12b I would argue any day it’s better than 27b dense.
And if you review the argument that not everybody needs a lot of context… Let me ask then why on earth you wasted 4k on spark? Just use cloud. Small context = small expense. No tinkering. Yuo want spark to have a lot of ram, be able to fine tune, stuff vast corpus of data, stay sovereign. If you need it for concurrency and many small requests - wrong choice, it’s too slow. Rent freaking gh200 for 2$ an hour and get 10x speed for your team, private docker, encrypted data, ssl/ssh tunnel, nobody peeks on your precious.
That’s solvable with Yarn, I used to run 27b up to 500k tokens and it does not lose its mind thanks to 27b dense core unlike 35b cousin, but it’s still becomes so slow I just ctrl-c it and switch to cloud out of frustration. Hence I have abandoned it.