xiaohong@spark-b3e0:~/ComfyUI$ sudo dmidecode -t memory | grep -E “(Size:|Volatile)” [sudo] password for xiaohong: Size: 128 GB Memory Operating Mode Capability: Volatile memory Non-Volatile Size: None Volatile Size: 128 GB Cache Size: None Logical Size: None xiaohong@spark-b3e0:~/ComfyUI$ sudo dmesg | grep -i memory [ 0.000000] Early memory node ranges [ 0.000000] ACPI GTDT: found 1 memory-mapped timer block(s). [ 0.014276] CPU features: detected: Memory Partitioning And Monitoring [ 0.014279] CPU features: detected: Memory Partitioning And Monitoring Virtualisation [ 0.016797] Memory: 125262500K/133828484K available (24704K kernel code, 5464K rwdata, 17712K rodata, 13568K init, 1196K bss, 8410324K reserved, 131072K cma-reserved) [ 0.037616] DMI: Memory slots populated: 1/1 [ 1.382007] Freeing initrd memory: 80016K [ 1.464234] Freeing unused kernel memory: 13568K [ 319.855990] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 319.932579] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 320.167758] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 320.233788] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 320.566140] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 355.981316] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 356.504534] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 356.623781] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 356.699472] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 356.793011] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 356.870197] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 356.925098] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 361.870956] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 361.881143] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 362.068721] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 362.068728] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from rmStatus @ system_mem.c:343 [ 414.076968] out_of_memory+0xe8/0x398 [ 414.077070] Tasks state (memory values in pages): [ 414.077418] Out of memory: Killed process 5145 (python) total-vm:250566816kB, anon-rss:84859916kB, file-rss:27536kB, shmem-rss:4000kB, UID:1000 pgtables:201688kB oom_score_adj:200 [ 1039.325481] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 1039.331715] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 1048.420732] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 1051.905328] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 1054.983086] audit: type=1400 audit(1764322818.321:195): apparmor=“DENIED” operation=“open” class=“file” profile=“snap.firefox.firefox” name=“/proc/pressure/memory” pid=4007 comm=“MemoryPoller” requested_mask=“r” denied_mask=“r” fsuid=1000 ouid=0 [ 1102.842068] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 1103.703141] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 1104.622854] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 1104.632580] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 1105.700406] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 1106.643235] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 1107.640797] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 1108.918838] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 1109.075527] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 1109.077322] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 1109.319123] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 1109.521835] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 1110.613792] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 1111.623335] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 1117.697053] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 1118.628702] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 1119.624884] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 1151.623550] out_of_memory+0xe8/0x398 [ 1151.623641] Tasks state (memory values in pages): [ 1151.623989] Out of memory: Killed process 7200 (python) total-vm:248744348kB, anon-rss:84124532kB, file-rss:27532kB, shmem-rss:5920kB, UID:1000 pgtables:199960kB oom_score_adj:200 [ 1153.518018] audit: type=1400 audit(1764322916.838:196): apparmor=“DENIED” operation=“open” class=“file” profile=“snap.firefox.firefox” name=“/proc/pressure/memory” pid=4007 comm=“MemoryPoller” requested_mask=“r” denied_mask=“r” fsuid=1000 ouid=0 [ 1153.658571] systemd-journald[627]: Under memory pressure, flushing caches. [ 3289.893671] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 3309.404585] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 3309.419743] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 3309.435603] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 3309.446695] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 3310.012030] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 3310.051737] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 3370.628424] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 3370.666271] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 3370.742462] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 3370.742467] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from rmStatus @ system_mem.c:354 [ 3371.009077] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 3371.448829] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 3445.929849] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 3445.966924] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 3446.569462] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 3456.725374] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 3456.785627] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 3458.064612] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 [ 3458.064618] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from rmStatus @ system_mem.c:343 [ 3474.306485] out_of_memory+0xe8/0x398 [ 3474.306653] Tasks state (memory values in pages): [ 3474.307238] Out of memory: Killed process 10494 (python) total-vm:248910552kB, anon-rss:85237236kB, file-rss:27580kB, shmem-rss:1760kB, UID:1000 pgtables:200820kB oom_score_adj:200
I got it working with fp8 following this workflow: Flux 2 | ComfyUI_examples
But it wouldn’t work with full sized version (just got an error message ‘Killed’!)
Have you guys tried to runsudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'before executing ComfyUI?
Thank you for your reply, I have used
To run the full sized version of FLUX.2 you can either start ComfyUI with the --cache-none argument (which is slower because it does not keep the model in memory after execution) or use the full BF16 GGUF version of FLUX.2 from city96/FLUX.2-dev-gguf · Hugging Face
There is also a thread on the official ComfyUI Github related to this:
I wrote a custom node that loads .safetensors using the fastsafetensors library directly to VRAM from storage, bypassing any copies. This works exactly as expected on DGX Spark, loading the BF16 version of FLUX.2 in only a few seconds:
hedin.e
kralvis
1m
Apologies in advance, i’m still new to Linux in general. but how do you actually install and use this. I did the git pull and the directory is there. so i did the pip install -r requirements.txt and that seemed to work. But I’m still getting the out memory errors so I’m obviously missing a step. any help for a noob would be appreciated. thanks in advance.
Ensure you’re cloning the repo into ComfyUI/custom_nodes. After restarting ComfyUI, add add the new DGXSparkSafetensorsLoader node, which will replace the default “Load Diffusion Model” node. Connect the output of the new DGX loader to the rest of your workflow. That should do it:
I used your node, and it was still the same Killed
Are you using Flux.2? Post your workflow and I’ll have a look.
What happens if you change the text encoder to load on the CPU?
This node works for me, though the only annoying thing is that if you run another workflow after it then it tends to run out of memory because the node hasn’t released the memory for the Flux model. Maybe add a flush DGX model node or something?
Yep, you’re right. Unfortunately, I don’t know enough about ComfyUI to know how to keep a handle to the memory allocated by the fastsafetensors reader. I mention this on the GitHub site, and the workaround is to just quit and relaunch ComfyUI. I agree this is a pain and once I figure it out I’ll update the node to fix this.