I have this error trying to build the latest eugr/spark-vllm-docker for mxfp4 to use gpt-oss.
I’ve tried docker builder prune -a
But it still doesn’t work.
If @eugr has time or anyone can please have a look at my error message.
Thank you!
Looks like the upstream repo had some breaking changes. I’ve updated the references and trying to build it locally. If it succeeds, I’ll push to the repo. Please, stay tuned.
OK, the build has succeeded, pushed a new version into the repo, please update.
I’ll try again and let you know
I delete the folder and cloned again but still get errors.
Do I have to clear anything else besides running the docker prune?
I also tried to clear the ~/.cache folder.
❯ ./run-recipe.sh openai-gpt-oss-120b --solo
Recipe: OpenAI GPT-OSS 120B
vLLM serving openai/gpt-oss-120b with MXFP4 quantization and FlashInfer
Container image ‘vllm-node-mxfp4’ not found locally.
Options:
Use --setup to build and run
Build manually: ./build-and-copy.sh -t vllm-node-mxfp4
Build now? [y/N] y
Building image ‘vllm-node-mxfp4’…
Build args: --exp-mxfp4
Building with experimental MXFP4 support…
Building image with command: docker build -t vllm-node-mxfp4 -f Dockerfile.mxfp4 --build-arg TRITON_REF=v3.6.0 --build-arg VLLM_REF=main --build-arg BUILD_JOBS=16 --build-arg TORCH_CUDA_ARCH_LIST=12.1a --build-arg FLASHINFER_CUDA_ARCH_LIST=12.1a .
[+] Building 2.9s (19/34) docker:default
=> [internal] load build definition from Dockerfile.mxfp4 0.0s
=> => transferring dockerfile: 10.79kB 0.0s
=> resolve image config for docker-image://docker.io/docker/dockerfile:1.6 0.7s
=> CACHED docker-image://docker.io/docker/dockerfile:1.6@sha256:ac85f380a63b13dfcefa89046420e1781752bab202122f8f50032edf31be0021 0.0s
=> [internal] load metadata for nvcr.io/nvidia/pytorch:26.01-py3 1.3s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 88B 0.0s
=> [base 1/5] FROM nvcr.io/nvidia/pytorch:26.01-py3@sha256:38ed2ecb2c16d10677006d73fb0a150855d6ec81db8fc66e800b5ae92741007e 0.0s
=> CACHED [runner 2/9] RUN apt update && apt install -y --no-install-recommends curl vim git libxcb1 && rm -rf /var/lib/apt/lists/* && pip install uv && pip unins 0.0s
=> CACHED [runner 3/9] WORKDIR /workspace/vllm 0.0s
=> CACHED [runner 4/9] RUN mkdir -p tiktoken_encodings && wget -O tiktoken_encodings/o200k_base.tiktoken "https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken 0.0s
=> CACHED [base 2/5] RUN apt update && apt install -y --no-install-recommends curl vim ninja-build git ccache && rm -rf /var/lib/apt/lists/* && pip install uv && 0.0s
=> CACHED [base 3/5] WORKDIR /workspace/vllm 0.0s
=> CACHED [base 4/5] RUN --mount=type=cache,id=uv-cache,target=/root/.cache/uv uv pip install fastsafetensors 0.0s
=> CACHED [base 5/5] RUN --mount=type=cache,id=uv-cache,target=/root/.cache/uv if [ “0” = “1” ]; then uv pip install -U transformers --pre; fi 0.0s
=> CACHED [builder 1/15] WORKDIR /workspace/vllm 0.0s
=> CACHED [builder 2/15] RUN --mount=type=cache,id=uv-cache,target=/root/.cache/uv uv pip install nvidia-nvshmem-cu13 “apache-tvm-ffi<0.2” 0.0s
=> CACHED [builder 3/15] RUN --mount=type=cache,id=git-flashinfer,target=/git-cache/flashinfer if [ -d /git-cache/flashinfer/.git ]; then echo "=== Using cached FlashInf 0.0s
=> CACHED [builder 4/15] RUN cd /workspace/flashinfer && git submodule update --init 3rdparty/spdlog 0.0s
=> ERROR [builder 5/15] RUN --mount=type=cache,id=git-cutlass,target=/git-cache/cutlass cd /workspace/flashinfer && rm -rf 3rdparty/cutlass && if [ -d /git-cache/cutlass 0.7s
[builder 5/15] RUN --mount=type=cache,id=git-cutlass,target=/git-cache/cutlass cd /workspace/flashinfer && rm -rf 3rdparty/cutlass && if [ -d /git-cache/cutlass/.git ] && [ -d /git-cache/cutlass/.git/objects ]; then echo “=== Using cached CUTLASS repo ===” && cp -a /git-cache/cutlass 3rdparty/cutlass && cd 3rdparty/cutlass && git fetch origin; else echo “=== Cloning CUTLASS (first build) ===” && rm -rf /git-cache/cutlass/* /git-cache/cutlass/.* 2>/dev/null || true && git clone https://github.com/christopherowen/cutlass.git 3rdparty/cutlass && cp -a /workspace/flashinfer/3rdparty/cutlass/. /git-cache/cutlass/; fi && cd /workspace/flashinfer/3rdparty/cutlass && git checkout 7554e4a32b15929289aa80e8bc8a4e5f6f779407:
0.104 === Using cached CUTLASS repo ===
0.666 fatal: reference is not a tree: 7554e4a32b15929289aa80e8bc8a4e5f6f779407
ERROR: failed to build: failed to solve: process “/bin/sh -c cd /workspace/flashinfer && rm -rf 3rdparty/cutlass && if [ -d /git-cache/cutlass/.git ] && [ -d /git-cache/cutlass/.git/objects ]; then echo "=== Using cached CUTLASS repo ===" && cp -a /git-cache/cutlass 3rdparty/cutlass && cd 3rdparty/cutlass && git fetch origin; else echo "=== Cloning CUTLASS (first build) ===" && rm -rf /git-cache/cutlass/* /git-cache/cutlass/.* 2>/dev/null || true && git clone ${CUTLASS_REPO} 3rdparty/cutlass && cp -a /workspace/flashinfer/3rdparty/cutlass/. /git-cache/cutlass/; fi && cd /workspace/flashinfer/3rdparty/cutlass && git checkout ${CUTLASS_SHA}” did not complete successfully: exit code: 128
Error: Failed to build image
Edita Dockerfile.mxfp4 y cambia la línea 101: work for me.
De: ARG CUTLASS_SHA=7554e4a32b15929289aa80e8bc8a4e5f6f779407
A: ARG CUTLASS_SHA=11af7f02ab52c9130e422eeb4b44042fbd60c083
Luego rebuild:
./build-and-copy.sh -t vllm-node-mxfp4 --exp-mxfp4 --rebuild-deps -c
Thank you. Now that step is solved. I’ll let you know how far it’ll build.
Edit: did not work, fails here:
569.7 Remark: The warnings can be suppressed with “-diag-suppress ”
569.7
569.7 [1293/3206] /usr/local/cuda/bin/nvcc --generate-dependencies-with-compile --dependency-output /workspace/flashinfer/build/aot/cached_ops/batch_prefill_with_attention_sink_kv_cache_dtype_q_f16_dtype_kv_f16_dtype_o_f16_dtype_idx_i32_head_dim_qk_64_head_dim_vo_64_posenc_0_use_swa_False_use_logits_cap_False_use_fp16_qk_False/batch_prefill_paged_kernel_mask_3.cuda.o.d -DPy_LIMITED_API=0x03090000 -D_GLIBCXX_USE_CXX11_ABI=1 -isystem /usr/include/python3.12 -isystem /usr/local/cuda/include -isystem /usr/local/cuda/include/cccl -isystem /usr/local/lib/python3.12/dist-packages/tvm_ffi/include -isystem /usr/local/lib/python3.12/dist-packages/tvm_ffi/include -isystem /workspace/flashinfer/include -isystem /workspace/flashinfer/csrc -isystem /workspace/flashinfer/3rdparty/cutlass/include -isystem /workspace/flashinfer/3rdparty/cutlass/tools/util/include -isystem /workspace/flashinfer/3rdparty/spdlog/include --compiler-options=-fPIC --expt-relaxed-constexpr -static-global-template-stub=false -gencode=arch=compute_121a,code=sm_121a -DFLASHINFER_ENABLE_FP8_E8M0 -DFLASHINFER_ENABLE_FP4_E2M1 -std=c++17 --threads=1 -use_fast_math -DFLASHINFER_ENABLE_F16 -DFLASHINFER_ENABLE_BF16 -DFLASHINFER_ENABLE_FP8_E4M3 -DFLASHINFER_ENABLE_FP8_E5M2 -DNDEBUG -O3 -c /workspace/flashinfer/build/aot/generated/batch_prefill_with_attention_sink_kv_cache_dtype_q_f16_dtype_kv_f16_dtype_o_f16_dtype_idx_i32_head_dim_qk_64_head_dim_vo_64_posenc_0_use_swa_False_use_logits_cap_False_use_fp16_qk_False/batch_prefill_paged_kernel_mask_3.cu -o /workspace/flashinfer/build/aot/cached_ops/batch_prefill_with_attention_sink_kv_cache_dtype_q_f16_dtype_kv_f16_dtype_o_f16_dtype_idx_i32_head_dim_qk_64_head_dim_vo_64_posenc_0_use_swa_False_use_logits_cap_False_use_fp16_qk_False/batch_prefill_paged_kernel_mask_3.cuda.o
569.7 /workspace/flashinfer/build/aot/generated/batch_prefill_with_attention_sink_kv_cache_dtype_q_f16_dtype_kv_f16_dtype_o_f16_dtype_idx_i32_head_dim_qk_64_head_dim_vo_64_posenc_0_use_swa_False_use_logits_cap_False_use_fp16_qk_False/batch_prefill_paged_kernel_mask_3.cu(6): warning #177-D: variable “flashinfer::use_custom_mask” was declared but never referenced
569.7 constexpr auto use_custom_mask = MaskMode::kMultiItemScoring == MaskMode::kCustom;
569.7 ^
569.7
569.7 Remark: The warnings can be suppressed with “-diag-suppress ”
569.7
569.7 [1294/3206] /usr/local/cuda/bin/nvcc --generate-dependencies-with-compile --dependency-output /workspace/flashinfer/build/aot/cached_ops/batch_prefill_with_attention_sink_kv_cache_dtype_q_f16_dtype_kv_f16_dtype_o_f16_dtype_idx_i32_head_dim_qk_64_head_dim_vo_64_posenc_0_use_swa_False_use_logits_cap_False_use_fp16_qk_False/batch_prefill_jit_binding.cuda.o.d -DPy_LIMITED_API=0x03090000 -D_GLIBCXX_USE_CXX11_ABI=1 -isystem /usr/include/python3.12 -isystem /usr/local/cuda/include -isystem /usr/local/cuda/include/cccl -isystem /usr/local/lib/python3.12/dist-packages/tvm_ffi/include -isystem /usr/local/lib/python3.12/dist-packages/tvm_ffi/include -isystem /workspace/flashinfer/include -isystem /workspace/flashinfer/csrc -isystem /workspace/flashinfer/3rdparty/cutlass/include -isystem /workspace/flashinfer/3rdparty/cutlass/tools/util/include -isystem /workspace/flashinfer/3rdparty/spdlog/include --compiler-options=-fPIC --expt-relaxed-constexpr -static-global-template-stub=false -gencode=arch=compute_121a,code=sm_121a -DFLASHINFER_ENABLE_FP8_E8M0 -DFLASHINFER_ENABLE_FP4_E2M1 -std=c++17 --threads=1 -use_fast_math -DFLASHINFER_ENABLE_F16 -DFLASHINFER_ENABLE_BF16 -DFLASHINFER_ENABLE_FP8_E4M3 -DFLASHINFER_ENABLE_FP8_E5M2 -DNDEBUG -O3 -c /workspace/flashinfer/build/aot/generated/batch_prefill_with_attention_sink_kv_cache_dtype_q_f16_dtype_kv_f16_dtype_o_f16_dtype_idx_i32_head_dim_qk_64_head_dim_vo_64_posenc_0_use_swa_False_use_logits_cap_False_use_fp16_qk_False/batch_prefill_jit_binding.cu -o /workspace/flashinfer/build/aot/cached_ops/batch_prefill_with_attention_sink_kv_cache_dtype_q_f16_dtype_kv_f16_dtype_o_f16_dtype_idx_i32_head_dim_qk_64_head_dim_vo_64_posenc_0_use_swa_False_use_logits_cap_False_use_fp16_qk_False/batch_prefill_jit_binding.cuda.o
569.7 ninja: build stopped: subcommand failed.
569.7 Traceback (most recent call last):
569.7 File “/workspace/flashinfer/flashinfer/jit/cpp_ext.py”, line 328, in run_ninja
569.7 subprocess.run(
569.7 File “/usr/lib/python3.12/subprocess.py”, line 571, in run
569.7 raise CalledProcessError(retcode, process.args,
569.7 subprocess.CalledProcessError: Command ‘[‘ninja’, ‘-v’, ‘-C’, ‘/workspace/flashinfer/build/aot/cached_ops’, ‘-f’, ‘/workspace/flashinfer/build/aot/cached_ops/tmp/flashinfer_jit.ninja’, ‘-j’, ‘16’]’ returned non-zero exit status 1.
569.7
569.7 The above exception was the direct cause of the following exception:
569.7
569.7 Traceback (most recent call last):
569.7 File “”, line 11, in
569.7 File “/workspace/flashinfer/flashinfer-jit-cache/build_backend.py”, line 182, in build_wheel
569.7 _prepare_build()
569.7 File “/workspace/flashinfer/flashinfer-jit-cache/build_backend.py”, line 124, in _prepare_build
569.7 _build_aot_modules()
569.7 File “/workspace/flashinfer/flashinfer-jit-cache/build_backend.py”, line 108, in _build_aot_modules
569.7 _compile_jit_cache(aot_package_dir)
569.7 File “/workspace/flashinfer/flashinfer-jit-cache/build_backend.py”, line 91, in _compile_jit_cache
569.7 aot.compile_and_package_modules(
569.7 File “/workspace/flashinfer/flashinfer/aot.py”, line 696, in compile_and_package_modules
569.7 build_jit_specs(jit_specs, verbose=verbose, skip_prebuilt=skip_prebuilt)
569.7 File “/workspace/flashinfer/flashinfer/jit/core.py”, line 503, in build_jit_specs
569.7 run_ninja(jit_env.FLASHINFER_JIT_DIR, ninja_path, verbose)
569.7 File “/workspace/flashinfer/flashinfer/jit/cpp_ext.py”, line 340, in run_ninja
569.7 raise RuntimeError(msg) from e
569.7 RuntimeError: Ninja build failed.
569.7 Failed to compile AOT modules: Ninja build failed.
569.7 × Failed to build /workspace/flashinfer/flashinfer-jit-cache
569.7 ├─▶ The build backend returned an error
569.7 ╰─▶ Call to build_backend.build_wheel failed (exit status: 1)
569.7 hint: This usually indicates a problem with the package or the build
569.7 environment.
569.7 DEBUG Released lock at /root/.cache/uv/.lock
ERROR: failed to build: failed to solve: process “/bin/sh -c cd flashinfer-jit-cache && uv build --no-build-isolation --wheel . --out-dir=/workspace/wheels -v” did not complete successfully: exit code: 2
Have you done git pull for the repo?
I’ve been able to rebuild successfully.
Actually, looks like some old bug that I fixed in my main Dockerfile is still there and prevents fetching proper commits. I’ll fix it.
so that the constructed image loads sed -i ‘s/–mxfp4-layers moe,qkv,o,lm_head/–mxfp4-layers moe,qkv,o/’ recipes/openai-gpt-oss-120b.yaml
yup I did this
tatami in 🌐 zgx-1155 in spark-vllm-docker on main via 🐍 v3.12.3 took 2s
❯ git pull
Already up to date.
tatami in 🌐 zgx-1155 in spark-vllm-docker on main via 🐍 v3.12.3
❯ sudo rm -Rf ~/.triton
tatami in 🌐 zgx-1155 in spark-vllm-docker on main via 🐍 v3.12.3
❯ sudo rm -Rf ~/.cache/vllm
tatami in 🌐 zgx-1155 in spark-vllm-docker on main via 🐍 v3.12.3
❯ sudo rm -Rf ~/.cache/flashinfer
tatami in 🌐 zgx-1155 in spark-vllm-docker on main via 🐍 v3.12.3
❯ ./build-and-copy.sh -t vllm-node-mxfp4 --exp-mxfp4
Building with experimental MXFP4 support...
Building image with command: docker build -t vllm-node-mxfp4 --build-arg BUILD_JOBS=16 --build-arg TORCH_CUDA_ARCH_LIST=12.1a --build-arg FLASHINFER_CUDA_ARCH_LIST=12.1a -f Dockerfile.mxfp4 .
[+] Building 35.7s (19/34) docker:default
=> [internal] load build definition from Dockerfile.mxfp4 0.0s
=> => transferring dockerfile: 10.96kB 0.0s
=> resolve image config for docker-image://docker.io/docker/dockerfile:1.6 1.4s
=> docker-image://docker.io/docker/dockerfile:1.6@sha256:ac85f380a63b13dfcefa89046420e1781752bab202122f8f50032edf31be0021 0.8s
=> => resolve docker.io/docker/dockerfile:1.6@sha256:ac85f380a63b13dfcefa89046420e1781752bab202122f8f50032edf31be0021 0.0s
=> => sha256:e929b0d024894103bb3a8577aad825a7df006a8f6767747bffd373d804c3ee67 482B / 482B 0.0s
=> => sha256:19c37871da0fa7da4b2a871455ccea62d3b08eb94a7f6f0cf310fe02f14f5089 1.27kB / 1.27kB 0.0s
=> => sha256:cf80d9b4bd1c75ee551ce5ff6f950a178fbb62e661c02e42d41b8c772d1efc1d 11.02MB / 11.02MB 0.6s
=> => sha256:ac85f380a63b13dfcefa89046420e1781752bab202122f8f50032edf31be0021 8.40kB / 8.40kB 0.0s
=> => extracting sha256:cf80d9b4bd1c75ee551ce5ff6f950a178fbb62e661c02e42d41b8c772d1efc1d 0.2s
=> [internal] load metadata for nvcr.io/nvidia/pytorch:26.01-py3 1.9s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 4.75kB 0.0s
=> [base 1/5] FROM nvcr.io/nvidia/pytorch:26.01-py3@sha256:38ed2ecb2c16d10677006d73fb0a150855d6ec81db8fc66e800b5ae92741007e 0.4s
=> => resolve nvcr.io/nvidia/pytorch:26.01-py3@sha256:38ed2ecb2c16d10677006d73fb0a150855d6ec81db8fc66e800b5ae92741007e 0.0s
=> => sha256:1b6dd8dc60741fb99b9aa343fa03fc897fbb64fbdcb1c8afcbd48c75cba57255 16.43kB / 16.43kB 0.0s
=> => sha256:38ed2ecb2c16d10677006d73fb0a150855d6ec81db8fc66e800b5ae92741007e 745B / 745B 0.0s
=> => sha256:5072d2d33da67f3099d83e89718cec113c89ff3c7b67851a5d435ff585b447ba 74.17kB / 74.17kB 0.0s
=> [base 2/5] RUN apt update && apt install -y --no-install-recommends curl vim ninja-build git ccache && rm -rf /var/lib/apt/lists/* && pip install uv && pip uninstall -y flash-attn 7.1s
=> [runner 2/9] RUN apt update && apt install -y --no-install-recommends curl vim git libxcb1 && rm -rf /var/lib/apt/lists/* && pip install uv && pip uninstall -y flash-attn 8.4s
=> [base 3/5] WORKDIR /workspace/vllm 0.0s
=> [base 4/5] RUN --mount=type=cache,id=uv-cache,target=/root/.cache/uv uv pip install fastsafetensors 7.9s
=> [runner 3/9] WORKDIR /workspace/vllm 0.0s
=> [runner 4/9] RUN mkdir -p tiktoken_encodings && wget -O tiktoken_encodings/o200k_base.tiktoken "https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken" && wget -O tiktoken_encodings/cl100k_base.tiktoken "http 4.6s
=> [base 5/5] RUN --mount=type=cache,id=uv-cache,target=/root/.cache/uv if [ "0" = "1" ]; then uv pip install -U transformers --pre; fi 0.2s
=> [builder 1/15] WORKDIR /workspace/vllm 0.0s
=> [builder 2/15] RUN --mount=type=cache,id=uv-cache,target=/root/.cache/uv uv pip install nvidia-nvshmem-cu13 "apache-tvm-ffi<0.2" 5.0s
=> [builder 3/15] RUN --mount=type=cache,id=git-flashinfer,target=/git-cache/flashinfer if [ -d /git-cache/flashinfer/.git ]; then echo "=== Using cached FlashInfer repo ===" && cp -a /git-cache/flashinfer /workspace 2.0s
=> [builder 4/15] RUN cd /workspace/flashinfer && git submodule update --init 3rdparty/spdlog 3.1s
=> ERROR [builder 5/15] RUN --mount=type=cache,id=git-cutlass,target=/git-cache/cutlass cd /workspace/flashinfer && rm -rf 3rdparty/cutlass && if [ -d /git-cache/cutlass/.git ] && [ -d /git-cache/cutlass/.git/objects ]; then 5.8s
------
> [builder 5/15] RUN --mount=type=cache,id=git-cutlass,target=/git-cache/cutlass cd /workspace/flashinfer && rm -rf 3rdparty/cutlass && if [ -d /git-cache/cutlass/.git ] && [ -d /git-cache/cutlass/.git/objects ]; then echo "=== Using cached CUTLASS repo ===" && cp -a /git-cache/cutlass 3rdparty/cutlass && cd 3rdparty/cutlass && git fetch origin && git fetch origin --tags; else echo "=== Cloning CUTLASS (first build) ===" && rm -rf /git-cache/cutlass/* /git-cache/cutlass/.* 2>/dev/null || true && git clone https://github.com/christopherowen/cutlass.git 3rdparty/cutlass && cp -a /workspace/flashinfer/3rdparty/cutlass/. /git-cache/cutlass/; fi && cd /workspace/flashinfer/3rdparty/cutlass && (git checkout --detach origin/7554e4a32b15929289aa80e8bc8a4e5f6f779407 2>/dev/null || git checkout 7554e4a32b15929289aa80e8bc8a4e5f6f779407):
0.141 === Cloning CUTLASS (first build) ===
0.143 Cloning into '3rdparty/cutlass'...
5.658 fatal: reference is not a tree: 7554e4a32b15929289aa80e8bc8a4e5f6f779407
------
ERROR: failed to build: failed to solve: process "/bin/sh -c cd /workspace/flashinfer && rm -rf 3rdparty/cutlass && if [ -d /git-cache/cutlass/.git ] && [ -d /git-cache/cutlass/.git/objects ]; then echo \"=== Using cached CUTLASS repo ===\" && cp -a /git-cache/cutlass 3rdparty/cutlass && cd 3rdparty/cutlass && git fetch origin && git fetch origin --tags; else echo \"=== Cloning CUTLASS (first build) ===\" && rm -rf /git-cache/cutlass/* /git-cache/cutlass/.* 2>/dev/null || true && git clone ${CUTLASS_REPO} 3rdparty/cutlass && cp -a /workspace/flashinfer/3rdparty/cutlass/. /git-cache/cutlass/; fi && cd /workspace/flashinfer/3rdparty/cutlass && (git checkout --detach origin/${CUTLASS_SHA} 2>/dev/null || git checkout ${CUTLASS_SHA})" did not complete successfully: exit code: 128
Can you pull the latest changes and try again please? I can’t reproduce locally because this particular error requires a specific cache state, but I think I fixed that.
EDIT: looks like there is an issue with upstream repository
with @vedcsolution suggestion to use
ARG CUTLASS_SHA=11af7f02ab52c9130e422eeb4b44042fbd60c083
I get to this point:
some cubin download error for non US like last time you told me?
Downloading cubins: 100%|██████████| 5299/5299 [06:46<00:00, 13.03it/s]
445.1 Traceback (most recent call last):
445.1 File "<string>", line 11, in <module>
445.1 File "/workspace/flashinfer/flashinfer-cubin/build_backend.py", line 100, in build_wheel
445.1 _download_cubins()
445.1 File "/workspace/flashinfer/flashinfer-cubin/build_backend.py", line 33, in _download_cubins
445.1 download_artifacts()
445.1 File "/workspace/flashinfer/flashinfer/artifacts.py", line 219, in download_artifacts
445.1 raise RuntimeError("Failed to download cubins")
445.1 RuntimeError: Failed to download cubins
445.1 × Failed to build `/workspace/flashinfer/flashinfer-cubin`
445.1 ├─▶ The build backend returned an error
445.1 ╰─▶ Call to `build_backend.build_wheel` failed (exit status: 1)
445.1 hint: This usually indicates a problem with the package or the build
445.1 environment.
445.1 DEBUG Released lock at `/root/.cache/uv/.lock`
------
ERROR: failed to build: failed to solve: process "/bin/sh -c cd flashinfer-cubin && uv build --no-build-isolation --wheel . --out-dir=/workspace/wheels -v" did not complete successfully: exit code: 2
tatami in 🌐 zgx-1155 in spark-vllm-docker on main [!] via 🐍 v3.12.3 took 7m27s
❯ git pull
Already up to date.
docker builder prune -f
arg
build
para cargar una vez construida sed -i ‘s/–mxfp4-layers moe,qkv,o,lm_head/–mxfp4-layers moe,qkv,o/’ recipes/openai-gpt-oss-120b.yaml
Yes, this is the same cubin issue - I didn’t implement caching or prebuilt wheels for mxfp4 builds yet, only for main builds.
I suggest using a different CUTLASS_SHA though: c7516ad20f3d022fdbc93e9468643bf3b577e02c
I’ve just incorporated it into the Dockerfile, you can pull the latest.
To get around cubins issue (for mxfp4 build only!) you can apply this patch:
cat <<EOM | git apply
diff --git a/Dockerfile.mxfp4 b/Dockerfile.mxfp4
index 138a903..e5c598d 100644
--- a/Dockerfile.mxfp4
+++ b/Dockerfile.mxfp4
@@ -149,15 +149,15 @@ RUN --mount=type=cache,id=uv-cache,target=/root/.cache/uv \
uv build --no-build-isolation --wheel . --out-dir=/workspace/wheels -v
# flashinfer-cubin
-RUN --mount=type=cache,id=uv-cache,target=/root/.cache/uv \
- --mount=type=cache,id=ccache,target=/root/.ccache \
- cd flashinfer-cubin && uv build --no-build-isolation --wheel . --out-dir=/workspace/wheels -v
-
-# flashinfer-jit-cache
-RUN --mount=type=cache,id=uv-cache,target=/root/.cache/uv \
- --mount=type=cache,id=ccache,target=/root/.ccache \
- cd flashinfer-jit-cache && \
- uv build --no-build-isolation --wheel . --out-dir=/workspace/wheels -v
+# RUN --mount=type=cache,id=uv-cache,target=/root/.cache/uv \
+# --mount=type=cache,id=ccache,target=/root/.ccache \
+# cd flashinfer-cubin && uv build --no-build-isolation --wheel . --out-dir=/workspace/wheels -v
+
+# # flashinfer-jit-cache
+# RUN --mount=type=cache,id=uv-cache,target=/root/.cache/uv \
+# --mount=type=cache,id=ccache,target=/root/.ccache \
+# cd flashinfer-jit-cache && \
+# uv build --no-build-isolation --wheel . --out-dir=/workspace/wheels -v
# --- VLLM SOURCE CACHE BUSTER ---
# Change THIS argument to force a fresh git clone and rebuild of vLLM
EOM
This will skip generating cubins and jit-cache. The build will still work, but initial startup time will take a bit longer, because flashinfer will have to compile some stuff on the fly.
Build successful with the other CUTLASS_SHA.
Will clear cache and pull the changes and try your version now.
Ive rebuilt the image, but seem to be getting this now when launching:
(EngineCore_DP0 pid=1335) (RayWorkerWrapper pid=384, ip=192.168.177.11) INFO 02-19 09:58:11 [decorators.py:432] Directly load AOT compilation from path /root/.cache/vllm/torch_aot_compile/f6ac3aae49e9bcbb7edc2f268eed5df1956062c6f5a28e80ea688bcc734cf27c/rank_1_0/model
(EngineCore_DP0 pid=1335) (RayWorkerWrapper pid=384, ip=192.168.177.11) INFO 02-19 09:58:12 [backends.py:644] Using cache directory: /root/.cache/vllm/torch_compile_cache/1cb2a24d41/rank_1_0/backbone for vLLM’s torch.compile
(EngineCore_DP0 pid=1335) (RayWorkerWrapper pid=384, ip=192.168.177.11) INFO 02-19 09:58:12 [backends.py:704] Dynamo bytecode transform time: 1.19 s
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] EngineCore failed to start.
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] Traceback (most recent call last):
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py”, line 891, in run_engine_core
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py”, line 656, in init
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] super().init(
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py”, line 112, in init
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] num_gpu_blocks, num_cpu_blocks, kv_cache_config = self._initialize_kv_caches(
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py”, line 243, in _initialize_kv_caches
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py”, line 126, in determine_available_memory
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] return self.collective_rpc(“determine_available_memory”)
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/ray_executor.py”, line 490, in collective_rpc
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] return ray.get(ray_worker_outputs, timeout=timeout)
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] File “/usr/local/lib/python3.12/dist-packages/ray/_private/auto_init_hook.py”, line 22, in auto_init_wrapper
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] return fn(*args, **kwargs)
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] ^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] File “/usr/local/lib/python3.12/dist-packages/ray/_private/client_mode_hook.py”, line 104, in wrapper
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] return func(*args, **kwargs)
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] File “/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py”, line 2981, in get
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] values, debugger_breakpoint = worker.get_objects(
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] ^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] File “/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py”, line 1012, in get_objects
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] raise value.as_instanceof_cause()
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] ray.exceptions.RayTaskError(RuntimeError): ray::RayWorkerWrapper.execute_method() (pid=1451, ip=192.168.177.12, actor_id=dabbcfe2dcc84ebf935d3bbe01000000, repr=<vllm.v1.executor.ray_utils.RayWorkerWrapper object at 0xeef2508c11c0>)
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/worker_base.py”, line 339, in execute_method
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] raise e
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/worker_base.py”, line 328, in execute_method
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] return run_method(self, method, args, kwargs)
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/serial_utils.py”, line 461, in run_method
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] return func(*args, **kwargs)
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] File “/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py”, line 124, in decorate_context
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] return func(*args, **kwargs)
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py”, line 322, in determine_available_memory
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] self.model_runner.profile_run()
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py”, line 4750, in profile_run
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] output = self._dummy_sampler_run(last_hidden_states)
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] File “/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py”, line 124, in decorate_context
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] return func(*args, **kwargs)
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py”, line 4545, in _dummy_sampler_run
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] logits = self.model.compute_logits(hidden_states)
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/gpt_oss.py”, line 726, in compute_logits
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] logits = self.logits_processor(self.lm_head, hidden_states)
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] File “/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py”, line 1776, in _wrapped_call_impl
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] File “/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py”, line 1787, in _call_impl
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/logits_processor.py”, line 60, in forward
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] logits = self._get_logits(hidden_states, lm_head, embedding_bias)
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/logits_processor.py”, line 92, in _get_logits
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] logits = lm_head.quant_method.apply(lm_head, hidden_states, bias=embedding_bias)
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/mxfp4.py”, line 828, in apply
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] return apply_fp4_marlin_linear(
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] ^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/marlin_utils_fp4.py”, line 123, in apply_fp4_marlin_linear
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] output = ops.gptq_marlin_gemm(
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] File “/usr/local/lib/python3.12/dist-packages/vllm/_custom_ops.py”, line 1348, in gptq_marlin_gemm
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] return torch.ops._C.gptq_marlin_gemm(
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] File “/usr/local/lib/python3.12/dist-packages/torch/_ops.py”, line 1205, in call
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] return self._op(*args, **kwargs)
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1335) ERROR 02-19 09:58:20 [core.py:900] RuntimeError: Invalid thread config: thread_m_blocks = 1, thread_k = -1, thread_n = -1, num_threads = -1 for MKN = [256, 2880, 100544] and num_bits = 4, prob_m_split = 256, group_size = 32, has_act_order = 0, is_k_full = 1, has_zp = 0, is_zp_float = 0, stages = 4, max_shared_mem_new = 101376
(EngineCore_DP0 pid=1335) (RayWorkerWrapper pid=384, ip=192.168.177.11) INFO 02-19 09:58:20 [backends.py:226] Directly load the compiled graph(s) for compile range (1, 8192) from the cache, took 6.789 s
Does it launch on a single Spark (without Ray)?
yep, it does work when using solo mode.
Hmm… Something got broken in that SHA though - I just used the latest one that Christopher is trying to merge upstream.
@christopher.owen - what are the proper SHAs to use for mxfp4 builds? The ones in your Docker seem to be incorrect, at least for Cutlass repo.