I remain really impressed by Step-3.5-Flash and am excited to see 3.7!
Single Spark users can run the capable IQ4_XS quant using llama.cpp - full context definitely works text-only, adding vision may cut down KV cache a little. Sweet spot is definitely 2 Sparks.
Built fresh container images, downloaded the model, tried to launch the FP8 recipe and when I launch the cluster it just fails on both nodes:
Traceback (most recent call last):
File "/usr/local/bin/vllm", line 10, in <module>
sys.exit(main())
^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py", line 18, in main
import vllm.entrypoints.cli.benchmark.main
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/benchmark/main.py", line 10, in <module>
from vllm.entrypoints.utils import VLLM_SUBCMD_PARSER_EPILOG
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/utils.py", line 20, in <module>
from vllm.engine.arg_utils import EngineArgs
File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 35, in <module>
from vllm.config import (
File "/usr/local/lib/python3.12/dist-packages/vllm/config/__init__.py", line 6, in <module>
from vllm.config.compilation import (
File "/usr/local/lib/python3.12/dist-packages/vllm/config/compilation.py", line 22, in <module>
from vllm.platforms import current_platform
File "/usr/local/lib/python3.12/dist-packages/vllm/platforms/__init__.py", line 278, in __getattr__
_current_platform = resolve_obj_by_qualname(platform_cls_qualname)()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/utils/import_utils.py", line 109, in resolve_obj_by_qualname
module = importlib.import_module(module_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/platforms/cuda.py", line 21, in <module>
import vllm._C # noqa
^^^^^^^^^^^^^^
ImportError: libtorch_cuda.so: cannot open shared object file: No such file or directory
Do the published 3 day old wheels have the vLLM patches as well? The vLLM repo only had [Model]Support Step-3.7-Flash (#43859) merge in a few hours ago.
If anyone has this issue, please rebuild with ./build-and-copy.sh --cleanup -c
I’ve changed the way the version checks work to avoid rate limit errors with GitHub APIs, so if you already had flashinfer wheels it may not pick it up automatically.
Also the KV cache in the FP8 recipe doesn’t seem to fit either.
I bumped it up to my max of gpu_memory_utilization_gb: 111.0 and it still wasn’t happy:
(EngineCore pid=136) ValueError: To serve at least one request with the model’s max seq len (262144), (6.18 GiB KV cache is needed, which is larger than the available KV cache memory (5.45 GiB). Based on the available memory, the estimated maximum model length is 230336. Try increasing gpu_memory_utilization or gpu_memory_utilization_gb, or decreasing max_model_len when initializing the engine (gpu_memory_utilization also controls CPU memory on the CPU backend). See Conserving Memory - vLLM for more details.
tg seems a bit low for the model of this size? I wonder if FP8 will be not slower. @eugr_nv Do you happen to know if we should expect any tangible NVFP4 decode performance increase when all sm120 PRs related to nvfp4 support get merged?
