Spark: cudaErrorNotPermitted in comfyui - but only after Docker sits idle for hours

[fierywaters13]

I’m experiencing an issue when running docker containers on the DGX Spark with cuda, that the container will work fine, but after a day or several days of sitting idle, I get cuda errors which are resolved by rebooting the container.

The issue has happened multiple times, on two different applications, and fixed by simply rebooting the container.

This is an LLM generated summary created by Claude Code Opus 4.5:

Summary: CUDA Error on NVIDIA GB10 (Grace Blackwell)

Error Details

• Error Type: torch.AcceleratorError / RuntimeError
• Error Code: cudaErrorNotPermitted (CUDA error: operation not permitted)
• Failing Operation: argmin(dim=0) tensor operation in comfy/model_sampling.py:166
• Context: KSamplerAdvanced node during first sampling step (0/20)

System Information

• GPU: NVIDIA GB10 (Grace Blackwell / Project Digits)
• Driver: 580.95.05
• CUDA Version: 13.0
• PyTorch: 2.10.0a0+b558c986e8.nv25.11 (NVIDIA pre-release build)
• Python: 3.12.3
• OS: Linux 6.14.0-1013-nvidia (ARM64)
• Memory: 122506 MB unified CPU/GPU memory
• ComfyUI: 0.3.76
• Docker Base Image: nvcr.io/nvidia/pytorch:25.11-py3

Observed Behavior

1. ComfyUI starts normally and loads models to GPU successfully
2. CLIP and SDXL models load without issue (~4.9GB to VRAM)
3. Error occurs when sampling begins - specifically on argmin() tensor operation
4. After error, new Python processes in the container report “No CUDA GPUs are available”
5. Container restart resolves the issue - CUDA works normally after docker compose down/up

Key Observations

• The GPU remained visible to the host (nvidia-smi showed the Python process using 7GB)
• But CUDA context inside the container became corrupted/inaccessible
• The error suggests permission/access issues rather than out-of-memory

Recurring Issue

This problem has occurred multiple times across different Docker images, suggesting:

• Likely related to the base image or GPU driver interaction
• Not specific to ComfyUI - appears to be a container/CUDA context issue
• May be a bug or incompatibility with the GB10 hardware and current driver stack

Potential Root Causes

1. GB10/Grace Blackwell immaturity - New ARM-based unified memory architecture
2. Driver bugs - NVIDIA 580.x drivers may have issues with container GPU passthrough on GB10
3. PyTorch pre-release issues - The nv25.11 build is not a stable release
4. Container runtime interaction - NVIDIA container toolkit may have edge cases on this platform

Workaround

docker compose down && docker compose up -d
Restarting the container restores CUDA functionality.

Full dockerfile:

jacob@spark:~/git-repos/comfyui-minimal$ cat Dockerfile 
FROM nvcr.io/nvidia/pytorch:25.11-py3

WORKDIR /workspace

# Install git and uv
RUN apt-get update && apt-get install -y --no-install-recommends git \
    && rm -rf /var/lib/apt/lists/* \
    && pip install uv

# Clone ComfyUI
RUN git clone https://github.com/comfyanonymous/ComfyUI.git

# Install ComfyUI requirements
WORKDIR /workspace/ComfyUI
# Install requirements but skip torch packages (already in NGC image with CUDA)
RUN grep -vE '^(torch|torchvision|torchaudio)$' requirements.txt > requirements-notorch.txt \
    && uv pip install --system --break-system-packages -r requirements-notorch.txt

EXPOSE 8188

CMD ["python", "main.py", "--listen", "0.0.0.0"]

Full Errorcode from ComfyUI:

# ComfyUI Error Report
## Error Details
- **Node ID:** 10
- **Node Type:** KSamplerAdvanced
- **Exception Type:** torch.AcceleratorError
- **Exception Message:** CUDA error: operation not permitted
Search for `cudaErrorNotPermitted' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.


## Stack Trace
```
  File "/workspace/ComfyUI/execution.py", line 515, in execute
    output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
                                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/workspace/ComfyUI/execution.py", line 329, in get_output_data
    return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/workspace/ComfyUI/execution.py", line 303, in _async_map_node_over_list
    await process_inputs(input_dict, i)

  File "/workspace/ComfyUI/execution.py", line 291, in process_inputs
    result = f(**inputs)
             ^^^^^^^^^^^

  File "/workspace/ComfyUI/nodes.py", line 1572, in sample
    return common_ksampler(model, noise_seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise, disable_noise=disable_noise, start_step=start_at_step, last_step=end_at_step, force_full_denoise=force_full_denoise)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/workspace/ComfyUI/nodes.py", line 1505, in common_ksampler
    samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/workspace/ComfyUI/comfy/sample.py", line 60, in sample
    samples = sampler.sample(noise, positive, negative, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/workspace/ComfyUI/comfy/samplers.py", line 1163, in sample
    return sample(self.model, noise, positive, negative, cfg, self.device, sampler, sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/workspace/ComfyUI/comfy/samplers.py", line 1053, in sample
    return cfg_guider.sample(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/workspace/ComfyUI/comfy/samplers.py", line 1035, in sample
    output = executor.execute(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed, latent_shapes=latent_shapes)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/workspace/ComfyUI/comfy/patcher_extension.py", line 112, in execute
    return self.original(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/workspace/ComfyUI/comfy/samplers.py", line 997, in outer_sample
    output = self.inner_sample(noise, latent_image, device, sampler, sigmas, denoise_mask, callback, disable_pbar, seed, latent_shapes=latent_shapes)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/workspace/ComfyUI/comfy/samplers.py", line 980, in inner_sample
    samples = executor.execute(self, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/workspace/ComfyUI/comfy/patcher_extension.py", line 112, in execute
    return self.original(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/workspace/ComfyUI/comfy/samplers.py", line 752, in sample
    samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **self.extra_options)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 121, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^

  File "/workspace/ComfyUI/comfy/k_diffusion/sampling.py", line 199, in sample_euler
    denoised = model(x, sigma_hat * s_in, **extra_args)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/workspace/ComfyUI/comfy/samplers.py", line 401, in __call__
    out = self.inner_model(x, sigma, model_options=model_options, seed=seed)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/workspace/ComfyUI/comfy/samplers.py", line 953, in __call__
    return self.outer_predict_noise(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/workspace/ComfyUI/comfy/samplers.py", line 960, in outer_predict_noise
    ).execute(x, timestep, model_options, seed)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/workspace/ComfyUI/comfy/patcher_extension.py", line 112, in execute
    return self.original(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/workspace/ComfyUI/comfy/samplers.py", line 963, in predict_noise
    return sampling_function(self.inner_model, x, timestep, self.conds.get("negative", None), self.conds.get("positive", None), self.cfg, model_options=model_options, seed=seed)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/workspace/ComfyUI/comfy/samplers.py", line 381, in sampling_function
    out = calc_cond_batch(model, conds, x, timestep, model_options)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/workspace/ComfyUI/comfy/samplers.py", line 206, in calc_cond_batch
    return _calc_cond_batch_outer(model, conds, x_in, timestep, model_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/workspace/ComfyUI/comfy/samplers.py", line 214, in _calc_cond_batch_outer
    return executor.execute(model, conds, x_in, timestep, model_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/workspace/ComfyUI/comfy/patcher_extension.py", line 112, in execute
    return self.original(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/workspace/ComfyUI/comfy/samplers.py", line 326, in _calc_cond_batch
    output = model.apply_model(input_x, timestep_, **c).chunk(batch_chunks)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/workspace/ComfyUI/comfy/model_base.py", line 162, in apply_model
    return comfy.patcher_extension.WrapperExecutor.new_class_executor(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/workspace/ComfyUI/comfy/patcher_extension.py", line 112, in execute
    return self.original(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/workspace/ComfyUI/comfy/model_base.py", line 183, in _apply_model
    t = self.model_sampling.timestep(t).float()
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/workspace/ComfyUI/comfy/model_sampling.py", line 166, in timestep
    return dists.abs().argmin(dim=0).view(sigma.shape).to(sigma.device)
           ^^^^^^^^^^^^^^^^^^^^^^^^^

```
## System Information
- **ComfyUI Version:** 0.3.76
- **Arguments:** main.py --listen 0.0.0.0
- **OS:** linux
- **Python Version:** 3.12.3 (main, Aug 14 2025, 17:47:21) [GCC 13.3.0]
- **Embedded Python:** false
- **PyTorch Version:** 2.10.0a0+b558c986e8.nv25.11
## Devices

- **Name:** cuda:0 NVIDIA GB10 : native
  - **Type:** cuda
  - **VRAM Total:** 128457015296
  - **VRAM Free:** 54328942592
  - **Torch VRAM Total:** 0
  - **Torch VRAM Free:** 0

## Logs
```
2025-12-22T03:51:42.258027 - [START] Security scan2025-12-22T03:51:42.258040 - 
2025-12-22T03:51:42.637620 - [DONE] Security scan2025-12-22T03:51:42.637632 - 
2025-12-22T03:51:42.655375 - ## ComfyUI-Manager: installing dependencies done.2025-12-22T03:51:42.655400 - 
2025-12-22T03:51:42.655411 - ** ComfyUI startup time:2025-12-22T03:51:42.655418 -  2025-12-22T03:51:42.655426 - 2025-12-22 03:51:42.6552025-12-22T03:51:42.655432 - 
2025-12-22T03:51:42.655440 - ** Platform:2025-12-22T03:51:42.655446 -  2025-12-22T03:51:42.655452 - Linux2025-12-22T03:51:42.655457 - 
2025-12-22T03:51:42.655463 - ** Python version:2025-12-22T03:51:42.655469 -  2025-12-22T03:51:42.655474 - 3.12.3 (main, Aug 14 2025, 17:47:21) [GCC 13.3.0]2025-12-22T03:51:42.655480 - 
2025-12-22T03:51:42.655486 - ** Python executable:2025-12-22T03:51:42.655491 -  2025-12-22T03:51:42.655496 - /usr/bin/python2025-12-22T03:51:42.655501 - 
2025-12-22T03:51:42.655507 - ** ComfyUI Path:2025-12-22T03:51:42.655512 -  2025-12-22T03:51:42.655517 - /workspace/ComfyUI2025-12-22T03:51:42.655522 - 
2025-12-22T03:51:42.655528 - ** ComfyUI Base Folder Path:2025-12-22T03:51:42.655533 -  2025-12-22T03:51:42.655538 - /workspace/ComfyUI2025-12-22T03:51:42.655543 - 
2025-12-22T03:51:42.655549 - ** User directory:2025-12-22T03:51:42.655554 -  2025-12-22T03:51:42.655560 - /workspace/ComfyUI/user2025-12-22T03:51:42.655565 - 
2025-12-22T03:51:42.655571 - ** ComfyUI-Manager config path:2025-12-22T03:51:42.655576 -  2025-12-22T03:51:42.655582 - /workspace/ComfyUI/user/__manager/config.ini2025-12-22T03:51:42.655587 - 
2025-12-22T03:51:42.655594 - ** Log path:2025-12-22T03:51:42.655599 -  2025-12-22T03:51:42.655604 - /workspace/ComfyUI/user/comfyui.log2025-12-22T03:51:42.655609 - 
2025-12-22T03:51:43.026025 - [ComfyUI-Manager] PyTorch is not installed
2025-12-22T03:51:43.026303 - 
Prestartup times for custom nodes:
2025-12-22T03:51:43.026365 -    0.9 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-Manager
2025-12-22T03:51:43.026401 - 
2025-12-22T03:51:43.630583 - Checkpoint files will always be loaded safely.
2025-12-22T03:51:43.941933 - Total VRAM 122506 MB, total RAM 122506 MB
2025-12-22T03:51:43.942020 - pytorch version: 2.10.0a0+b558c986e8.nv25.11
2025-12-22T03:51:43.942304 - Set vram state to: NORMAL_VRAM
2025-12-22T03:51:43.942463 - Device: cuda:0 NVIDIA GB10 : native
2025-12-22T03:51:43.942523 - Using async weight offloading with 2 streams
2025-12-22T03:51:43.942588 - Enabled pinned memory 116380.0
2025-12-22T03:51:44.371504 - Using pytorch attention
2025-12-22T03:51:44.375602 - torchaudio missing, ACE model will be broken
2025-12-22T03:51:44.375875 - torchaudio missing, ACE model will be broken
2025-12-22T03:51:44.376967 - torchaudio missing, MMAudio VAE model will be broken
2025-12-22T03:51:45.125325 - Python version: 3.12.3 (main, Aug 14 2025, 17:47:21) [GCC 13.3.0]
2025-12-22T03:51:45.125399 - ComfyUI version: 0.3.76
2025-12-22T03:51:45.127238 - ComfyUI frontend version: 1.33.10
2025-12-22T03:51:45.127563 - [Prompt Server] web root: /usr/local/lib/python3.12/dist-packages/comfyui_frontend_package/static
2025-12-22T03:51:45.240569 - Traceback (most recent call last):
  File "/workspace/ComfyUI/nodes.py", line 2149, in load_custom_node
    module_spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/workspace/ComfyUI/comfy_extras/nodes_audio.py", line 4, in <module>
    import torchaudio
ModuleNotFoundError: No module named 'torchaudio'

2025-12-22T03:51:45.240717 - Cannot import /workspace/ComfyUI/comfy_extras/nodes_audio.py module for custom nodes: No module named 'torchaudio'
2025-12-22T03:51:45.242377 - Total VRAM 122506 MB, total RAM 122506 MB
2025-12-22T03:51:45.242414 - pytorch version: 2.10.0a0+b558c986e8.nv25.11
2025-12-22T03:51:45.242508 - Set vram state to: NORMAL_VRAM
2025-12-22T03:51:45.242546 - Device: cuda:0 NVIDIA GB10 : native
2025-12-22T03:51:45.242641 - Using async weight offloading with 2 streams
2025-12-22T03:51:45.242696 - Enabled pinned memory 116380.0
2025-12-22T03:51:45.249917 - Traceback (most recent call last):
  File "/workspace/ComfyUI/nodes.py", line 2149, in load_custom_node
    module_spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/workspace/ComfyUI/comfy_extras/nodes_audio_encoder.py", line 2, in <module>
    import comfy.audio_encoders.audio_encoders
  File "/workspace/ComfyUI/comfy/audio_encoders/audio_encoders.py", line 2, in <module>
    from .whisper import WhisperLargeV3
  File "/workspace/ComfyUI/comfy/audio_encoders/whisper.py", line 4, in <module>
    import torchaudio
ModuleNotFoundError: No module named 'torchaudio'

2025-12-22T03:51:45.249971 - Cannot import /workspace/ComfyUI/comfy_extras/nodes_audio_encoder.py module for custom nodes: No module named 'torchaudio'
2025-12-22T03:51:45.506088 - ### Loading: ComfyUI-Manager (V3.38.3)
2025-12-22T03:51:45.506292 - [ComfyUI-Manager] network_mode: public
2025-12-22T03:51:45.545857 - ### ComfyUI Version: v0.3.76-36-gd7a0aef6 | Released on '2025-12-06'
2025-12-22T03:51:45.558055 - 
Import times for custom nodes:
2025-12-22T03:51:45.558141 -    0.0 seconds: /workspace/ComfyUI/custom_nodes/comfyui-model-downloader
2025-12-22T03:51:45.558163 -    0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI_UltimateSDUpscale
2025-12-22T03:51:45.558179 -    0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-Manager
2025-12-22T03:51:45.558194 - 
2025-12-22T03:51:45.558217 - WARNING: some comfy_extras/ nodes did not import correctly. This may be because they are missing some dependencies.

2025-12-22T03:51:45.558232 - IMPORT FAILED: nodes_audio.py
2025-12-22T03:51:45.558246 - IMPORT FAILED: nodes_audio_encoder.py
2025-12-22T03:51:45.558260 - 
This issue might be caused by new missing dependencies added the last time you updated ComfyUI.
2025-12-22T03:51:45.558275 - Please do a: pip install -r requirements.txt
2025-12-22T03:51:45.558290 - 
2025-12-22T03:51:45.651709 - [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/model-list.json
2025-12-22T03:51:45.691069 - [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/github-stats.json
2025-12-22T03:51:45.702730 - [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/alter-list.json
2025-12-22T03:51:45.729623 - Context impl SQLiteImpl.
2025-12-22T03:51:45.729938 - Will assume non-transactional DDL.
2025-12-22T03:51:45.730797 - No target revision found.
2025-12-22T03:51:45.746942 - [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/extension-node-map.json
2025-12-22T03:51:45.798098 - [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/custom-node-list.json
2025-12-22T03:51:45.802668 - Starting server

2025-12-22T03:51:45.802822 - To see the GUI go to: http://0.0.0.0:8188
2025-12-22T03:51:48.410557 - FETCH ComfyRegistry Data: 5/1142025-12-22T03:51:48.410583 - 
2025-12-22T03:51:51.484817 - FETCH ComfyRegistry Data: 10/1142025-12-22T03:51:51.484858 - 
2025-12-22T03:51:54.748441 - FETCH ComfyRegistry Data: 15/1142025-12-22T03:51:54.748465 - 
2025-12-22T03:51:57.845630 - FETCH ComfyRegistry Data: 20/1142025-12-22T03:51:57.845674 - 
2025-12-22T03:52:00.903828 - FETCH ComfyRegistry Data: 25/1142025-12-22T03:52:00.903865 - 
2025-12-22T03:52:03.998738 - FETCH ComfyRegistry Data: 30/1142025-12-22T03:52:03.998782 - 
2025-12-22T03:52:07.056273 - FETCH ComfyRegistry Data: 35/1142025-12-22T03:52:07.056310 - 
2025-12-22T03:52:10.678253 - FETCH ComfyRegistry Data: 40/1142025-12-22T03:52:10.678277 - 
2025-12-22T03:52:13.724392 - FETCH ComfyRegistry Data: 45/1142025-12-22T03:52:13.724436 - 
2025-12-22T03:52:16.807864 - FETCH ComfyRegistry Data: 50/1142025-12-22T03:52:16.807904 - 
2025-12-22T03:52:19.861281 - FETCH ComfyRegistry Data: 55/1142025-12-22T03:52:19.861324 - 
2025-12-22T03:52:22.997704 - FETCH ComfyRegistry Data: 60/1142025-12-22T03:52:22.997727 - 
2025-12-22T03:52:26.054098 - FETCH ComfyRegistry Data: 65/1142025-12-22T03:52:26.054137 - 
2025-12-22T03:52:29.201137 - FETCH ComfyRegistry Data: 70/1142025-12-22T03:52:29.201177 - 
2025-12-22T03:52:32.252122 - FETCH ComfyRegistry Data: 75/1142025-12-22T03:52:32.252145 - 
2025-12-22T03:52:35.309211 - FETCH ComfyRegistry Data: 80/1142025-12-22T03:52:35.309233 - 
2025-12-22T03:52:38.365254 - FETCH ComfyRegistry Data: 85/1142025-12-22T03:52:38.365276 - 
2025-12-22T03:52:41.488468 - FETCH ComfyRegistry Data: 90/1142025-12-22T03:52:41.488489 - 
2025-12-22T03:52:44.572844 - FETCH ComfyRegistry Data: 95/1142025-12-22T03:52:44.572874 - 
2025-12-22T03:52:47.655597 - FETCH ComfyRegistry Data: 100/1142025-12-22T03:52:47.655620 - 
2025-12-22T03:52:50.776188 - FETCH ComfyRegistry Data: 105/1142025-12-22T03:52:50.776210 - 
2025-12-22T03:52:53.857130 - FETCH ComfyRegistry Data: 110/1142025-12-22T03:52:53.857151 - 
2025-12-22T03:52:56.944174 - FETCH ComfyRegistry Data [DONE]2025-12-22T03:52:56.944215 - 
2025-12-22T03:52:57.009509 - [ComfyUI-Manager] default cache updated: https://api.comfy.org/nodes
2025-12-22T03:52:57.016731 - FETCH DATA from: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/custom-node-list.json2025-12-22T03:52:57.016750 - 2025-12-22T03:52:57.100899 -  [DONE]2025-12-22T03:52:57.100926 - 
2025-12-22T03:52:57.127486 - [ComfyUI-Manager] All startup tasks have been completed.
2025-12-24T08:31:01.032187 - [DEPRECATION WARNING] Detected import of deprecated legacy API: /scripts/ui.js. This is likely caused by a custom node extension using outdated APIs. Please update your extensions or contact the extension author for an updated version.
2025-12-24T08:31:01.035600 - [DEPRECATION WARNING] Detected import of deprecated legacy API: /extensions/core/groupNode.js. This is likely caused by a custom node extension using outdated APIs. Please update your extensions or contact the extension author for an updated version.
2025-12-24T08:31:01.844541 - [DEPRECATION WARNING] Detected import of deprecated legacy API: /scripts/ui/components/buttonGroup.js. This is likely caused by a custom node extension using outdated APIs. Please update your extensions or contact the extension author for an updated version.
2025-12-24T08:31:01.902804 - [DEPRECATION WARNING] Detected import of deprecated legacy API: /scripts/ui/components/button.js. This is likely caused by a custom node extension using outdated APIs. Please update your extensions or contact the extension author for an updated version.
2025-12-24T08:31:24.816503 - got prompt
2025-12-24T08:31:26.634216 - model weight dtype torch.float16, manual cast: None
2025-12-24T08:31:26.650884 - model_type EPS
2025-12-24T08:31:27.824086 - Using pytorch attention in VAE
2025-12-24T08:31:27.825600 - Using pytorch attention in VAE
2025-12-24T08:31:27.926965 - VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16
2025-12-24T08:31:28.395849 - CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
2025-12-24T08:31:28.878641 - model weight dtype torch.float16, manual cast: None
2025-12-24T08:31:28.878937 - model_type EPS
2025-12-24T08:31:30.348826 - Using pytorch attention in VAE
2025-12-24T08:31:30.350320 - Using pytorch attention in VAE
2025-12-24T08:31:30.433194 - VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16
2025-12-24T08:31:30.986742 - CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
2025-12-24T08:31:31.097957 - Requested to load SDXLClipModel
2025-12-24T08:31:31.315761 - loaded completely; 25272.46 MB usable, 1560.80 MB loaded, full load: True
2025-12-24T08:31:31.737275 - Requested to load SDXL
2025-12-24T08:31:32.374956 - loaded completely; 23828.91 MB usable, 4897.05 MB loaded, full load: True
2025-12-24T08:31:32.434762 - 
  0%|          | 0/20 [00:00<?, ?it/s]2025-12-24T08:31:32.466989 - 
  0%|          | 0/20 [00:00<?, ?it/s]2025-12-24T08:31:32.467012 - 
2025-12-24T08:31:32.476163 - !!! Exception during processing !!! CUDA error: operation not permitted
Search for `cudaErrorNotPermitted' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

2025-12-24T08:31:32.479698 - Traceback (most recent call last):
  File "/workspace/ComfyUI/execution.py", line 515, in execute
    output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
                                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/ComfyUI/execution.py", line 329, in get_output_data
    return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/ComfyUI/execution.py", line 303, in _async_map_node_over_list
    await process_inputs(input_dict, i)
  File "/workspace/ComfyUI/execution.py", line 291, in process_inputs
    result = f(**inputs)
             ^^^^^^^^^^^
  File "/workspace/ComfyUI/nodes.py", line 1572, in sample
    return common_ksampler(model, noise_seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise, disable_noise=disable_noise, start_step=start_at_step, last_step=end_at_step, force_full_denoise=force_full_denoise)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/ComfyUI/nodes.py", line 1505, in common_ksampler
    samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/ComfyUI/comfy/sample.py", line 60, in sample
    samples = sampler.sample(noise, positive, negative, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/ComfyUI/comfy/samplers.py", line 1163, in sample
    return sample(self.model, noise, positive, negative, cfg, self.device, sampler, sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/ComfyUI/comfy/samplers.py", line 1053, in sample
    return cfg_guider.sample(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/ComfyUI/comfy/samplers.py", line 1035, in sample
    output = executor.execute(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed, latent_shapes=latent_shapes)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/ComfyUI/comfy/patcher_extension.py", line 112, in execute
    return self.original(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/ComfyUI/comfy/samplers.py", line 997, in outer_sample
    output = self.inner_sample(noise, latent_image, device, sampler, sigmas, denoise_mask, callback, disable_pbar, seed, latent_shapes=latent_shapes)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/ComfyUI/comfy/samplers.py", line 980, in inner_sample
    samples = executor.execute(self, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/ComfyUI/comfy/patcher_extension.py", line 112, in execute
    return self.original(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/ComfyUI/comfy/samplers.py", line 752, in sample
    samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **self.extra_options)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 121, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/ComfyUI/comfy/k_diffusion/sampling.py", line 199, in sample_euler
    denoised = model(x, sigma_hat * s_in, **extra_args)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/ComfyUI/comfy/samplers.py", line 401, in __call__
    out = self.inner_model(x, sigma, model_options=model_options, seed=seed)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/ComfyUI/comfy/samplers.py", line 953, in __call__
    return self.outer_predict_noise(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/ComfyUI/comfy/samplers.py", line 960, in outer_predict_noise
    ).execute(x, timestep, model_options, seed)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/ComfyUI/comfy/patcher_extension.py", line 112, in execute
    return self.original(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/ComfyUI/comfy/samplers.py", line 963, in predict_noise
    return sampling_function(self.inner_model, x, timestep, self.conds.get("negative", None), self.conds.get("positive", None), self.cfg, model_options=model_options, seed=seed)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/ComfyUI/comfy/samplers.py", line 381, in sampling_function
    out = calc_cond_batch(model, conds, x, timestep, model_options)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/ComfyUI/comfy/samplers.py", line 206, in calc_cond_batch
    return _calc_cond_batch_outer(model, conds, x_in, timestep, model_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/ComfyUI/comfy/samplers.py", line 214, in _calc_cond_batch_outer
    return executor.execute(model, conds, x_in, timestep, model_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/ComfyUI/comfy/patcher_extension.py", line 112, in execute
    return self.original(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/ComfyUI/comfy/samplers.py", line 326, in _calc_cond_batch
    output = model.apply_model(input_x, timestep_, **c).chunk(batch_chunks)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/ComfyUI/comfy/model_base.py", line 162, in apply_model
    return comfy.patcher_extension.WrapperExecutor.new_class_executor(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/ComfyUI/comfy/patcher_extension.py", line 112, in execute
    return self.original(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/ComfyUI/comfy/model_base.py", line 183, in _apply_model
    t = self.model_sampling.timestep(t).float()
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/ComfyUI/comfy/model_sampling.py", line 166, in timestep
    return dists.abs().argmin(dim=0).view(sigma.shape).to(sigma.device)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
torch.AcceleratorError: CUDA error: operation not permitted
Search for `cudaErrorNotPermitted' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.


2025-12-24T08:31:32.480770 - Prompt executed in 7.65 seconds

```
## Attached Workflow
Please make sure that workflow does not contain any sensitive information such as API keys or passwords.
```
Workflow too large. Please manually upload the workflow from local file system.
```

## Additional Context
(Please add any additional context or steps to reproduce the error here)

[DannyTup]

This sounds a lot like the issue I reported here:

"Failed to initialize NVML: Unknown Error" running nvidia-smi in a docker container only after some hours/days

I created my own dashboard which shows some metrics and docker containers on my Spark. The full source is at GitHub - DanTup/dgx_dashboard: A simple dashboard for the DGX Spark. but it basically runs: nvidia-smi --query-gpu=utilization.gpu,temperature.gpu,power.draw --format=csv,noheader,nounits -l=5 To get metrics streamed every 5s. It starts the process only when there is a browser connected to the backend, and then terminates the process after a timeout when there are no connections. I ran…

It appears that whenever something triggers systemctl daemon-reload the GPUs disappear from the containers.

This is documented along with a workaround here:

--gpus - Troubleshooting — NVIDIA Container Toolkit

That page suggests that the issue has been resolved, however it definitely hadn’t for me. The workaround (use cgroupfs as the cgroup driver for containers) has solved the issue though.