An error occurs when loading coredump in cuda-gdb:
root@yy-lite-decode-0:/models# cuda-gdb
NVIDIA (R) cuda-gdb 12.4
Portions Copyright (C) 2007-2023 NVIDIA Corporation
Based on GNU gdb 13.1
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This CUDA-GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://forums.developer.nvidia.com/c/developer-tools/cuda-developer-tools/cuda-gdb>.
Find the CUDA-GDB manual and other documentation resources online at:
<https://docs.nvidia.com/cuda/cuda-gdb/index.html>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
(cuda-gdb) target cudacore cuda_core_dump.yy-lite-decode-0.99967
Opening GPU coredump: cuda_core_dump.yy-lite-decode-0.99967
cuda-gdb/13/gdb/cuda/cuda-state.c:154: internal-error: initialize: Assertion `m_instance.m_num_devices > 0' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
----- Backtrace -----
0x665c37 ???
0xae6144 ???
0xae64b8 ???
0xccc011 ???
0x74fbde ???
0x755e72 ???
0x7283eb ???
0x728487 ???
0x728a93 ???
0xab6852 ???
0x695143 ???
0xac7ca8 ???
0x800fbb ???
0x80220d ???
0x801928 ???
0xb29e30 ???
0x80173d ???
0x80180c ???
0x8009c7 ???
0xccca9c ???
0xcccc70 ???
0x8e46ce ???
0x8e5f64 ???
0x57b9e4 ???
0x7fed9ae13d8f __libc_start_call_main
../sysdeps/nptl/libc_start_call_main.h:58
0x7fed9ae13e3f __libc_start_main_impl
../csu/libc-start.c:392
0x583954 ???
0xffffffffffffffff ???
---------------------
cuda-gdb/13/gdb/cuda/cuda-state.c:154: internal-error: initialize: Assertion `m_instance.m_num_devices > 0' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Quit this debugging session? (y or n) n
This is a bug, please report it. For instructions, see:
<https://forums.developer.nvidia.com/c/developer-tools/cuda-developer-tools/cuda-gdb>.
cuda-gdb/13/gdb/cuda/cuda-state.c:154: internal-error: initialize: Assertion `m_instance.m_num_devices > 0' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Create a core file of GDB? (y or n) n
Command aborted.
env abort CUDA:
root@yy-lite-decode-0:/models# env | grep CUDA
NVIDIA_REQUIRE_CUDA=cuda>=9.0
CUDA_CACHE_DISABLE=1
CUDA_VERSION=12.4.1.003
CUDA_LAUNCH_BLOCKING=0
CUDA_ENABLE_COREDUMP_ON_EXCEPTION=1
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
CUDA_COREDUMP_FILE=/models/cuda_core_dump.%h.%p
CUDA_DRIVER_VERSION=550.54.15
_CUDA_COMPAT_PATH=/usr/local/cuda/compat
CUDA_HOME=/usr/local/cuda
Collecting coredump logs:
[05:26:57.672814] coredump: Starting GPU coredump generation
[05:26:57.672882] coredump: SM 1/108 is not used by any context
[05:26:57.672893] coredump: SM 2/108 is not used by any context
[05:26:57.672902] coredump: SM 3/108 is not used by any context
[05:26:57.672910] coredump: SM 4/108 is not used by any context
[05:26:57.672919] coredump: SM 5/108 is not used by any context
[05:26:57.672927] coredump: SM 6/108 is not used by any context
[05:26:57.672935] coredump: SM 7/108 is not used by any context
[05:26:57.672943] coredump: SM 8/108 is not used by any context
[05:26:57.672952] coredump: SM 9/108 is not used by any context
[05:26:57.672960] coredump: SM 10/108 is not used by any context
[05:26:57.672966] coredump: SM 11/108 is not used by any context
[05:26:57.672974] coredump: SM 12/108 is not used by any context
[05:26:57.672983] coredump: SM 13/108 is not used by any context
[05:26:57.672992] coredump: SM 14/108 is not used by any context
[05:26:57.673000] coredump: SM 15/108 is not used by any context
[05:26:57.673007] coredump: SM 16/108 is not used by any context
[05:26:57.673015] coredump: SM 17/108 is not used by any context
[05:26:57.673023] coredump: SM 18/108 is not used by any context
[05:26:57.673031] coredump: SM 19/108 is not used by any context
[05:26:57.673039] coredump: SM 20/108 is not used by any context
[05:26:57.673047] coredump: SM 21/108 is not used by any context
[05:26:57.673055] coredump: SM 22/108 is not used by any context
[05:26:57.673063] coredump: SM 23/108 is not used by any context
[05:26:57.673071] coredump: SM 24/108 is not used by any context
[05:26:57.673078] coredump: SM 25/108 is not used by any context
[05:26:57.673087] coredump: SM 26/108 is not used by any context
[05:26:57.673095] coredump: SM 27/108 is not used by any context
[05:26:57.673103] coredump: SM 28/108 is not used by any context
[05:26:57.673111] coredump: SM 29/108 is not used by any context
[05:26:57.673119] coredump: SM 30/108 is not used by any context
[05:26:57.673127] coredump: SM 31/108 is not used by any context
[05:26:57.673135] coredump: SM 32/108 is not used by any context
[05:26:57.673143] coredump: SM 33/108 is not used by any context
[05:26:57.673151] coredump: SM 34/108 is not used by any context
[05:26:57.673159] coredump: SM 35/108 is not used by any context
[05:26:57.673167] coredump: SM 36/108 is not used by any context
[05:26:57.673175] coredump: SM 37/108 is not used by any context
[05:26:57.673184] coredump: SM 38/108 is not used by any context
[05:26:57.673191] coredump: SM 39/108 is not used by any context
[05:26:57.673199] coredump: SM 40/108 is not used by any context
[05:26:57.673207] coredump: SM 41/108 is not used by any context
[05:26:57.673213] coredump: SM 42/108 is not used by any context
[05:26:57.673222] coredump: SM 43/108 is not used by any context
[05:26:57.673230] coredump: SM 44/108 is not used by any context
[05:26:57.673238] coredump: SM 45/108 is not used by any context
[05:26:57.673246] coredump: SM 46/108 is not used by any context
[05:26:57.673254] coredump: SM 47/108 is not used by any context
[05:26:57.673262] coredump: SM 48/108 is not used by any context
[05:26:57.673270] coredump: SM 49/108 is not used by any context
[05:26:57.673278] coredump: SM 50/108 is not used by any context
[05:26:57.673286] coredump: SM 51/108 is not used by any context
[05:26:57.673294] coredump: SM 52/108 is not used by any context
[05:26:57.673302] coredump: SM 53/108 is not used by any context
[05:26:57.673310] coredump: SM 54/108 is not used by any context
[05:26:57.673318] coredump: SM 55/108 is not used by any context
[05:26:57.673326] coredump: SM 56/108 is not used by any context
[05:26:57.673334] coredump: SM 57/108 is not used by any context
[05:26:57.673343] coredump: SM 58/108 is not used by any context
[05:26:57.673351] coredump: SM 59/108 is not used by any context
[05:26:57.673359] coredump: SM 60/108 is not used by any context
[05:26:57.673367] coredump: SM 61/108 is not used by any context
[05:26:57.673375] coredump: SM 62/108 is not used by any context
[05:26:57.673382] coredump: SM 63/108 is not used by any context
[05:26:57.673390] coredump: SM 64/108 is not used by any context
[05:26:57.673397] coredump: SM 65/108 is not used by any context
[05:26:57.673405] coredump: SM 66/108 is not used by any context
[05:26:57.673413] coredump: SM 67/108 is not used by any context
[05:26:57.673422] coredump: SM 68/108 is not used by any context
[05:26:57.673429] coredump: SM 69/108 is not used by any context
[05:26:57.673438] coredump: SM 70/108 is not used by any context
[05:26:57.673447] coredump: SM 71/108 is not used by any context
[05:26:57.673456] coredump: SM 72/108 is not used by any context
[05:26:57.673464] coredump: SM 73/108 is not used by any context
[05:26:57.673472] coredump: SM 74/108 is not used by any context
[05:26:57.673480] coredump: SM 75/108 is not used by any context
[05:26:57.673488] coredump: SM 76/108 is not used by any context
[05:26:57.673496] coredump: SM 77/108 is not used by any context
[05:26:57.673504] coredump: SM 78/108 is not used by any context
[05:26:57.673512] coredump: SM 79/108 is not used by any context
[05:26:57.673520] coredump: SM 80/108 is not used by any context
[05:26:57.673528] coredump: SM 81/108 is not used by any context
[05:26:57.673536] coredump: SM 82/108 is not used by any context
[05:26:57.673543] coredump: SM 83/108 is not used by any context
[05:26:57.673551] coredump: SM 84/108 is not used by any context
[05:26:57.673559] coredump: SM 85/108 is not used by any context
[05:26:57.673568] coredump: SM 86/108 is not used by any context
[05:26:57.673575] coredump: SM 87/108 is not used by any context
[05:26:57.673583] coredump: SM 88/108 is not used by any context
[05:26:57.673591] coredump: SM 89/108 is not used by any context
[05:26:57.673599] coredump: SM 90/108 is not used by any context
[05:26:57.673607] coredump: SM 91/108 is not used by any context
[05:26:57.673615] coredump: SM 92/108 is not used by any context
[05:26:57.673623] coredump: SM 93/108 is not used by any context
[05:26:57.673630] coredump: SM 94/108 is not used by any context
[05:26:57.673638] coredump: SM 95/108 is not used by any context
[05:26:57.673646] coredump: SM 96/108 is not used by any context
[05:26:57.673654] coredump: SM 97/108 is not used by any context
[05:26:57.673662] coredump: SM 98/108 is not used by any context
[05:26:57.673670] coredump: SM 99/108 is not used by any context
[05:26:57.673678] coredump: SM 100/108 is not used by any context
[05:26:57.673686] coredump: SM 101/108 is not used by any context
[05:26:57.673694] coredump: SM 102/108 is not used by any context
[05:26:57.673703] coredump: SM 103/108 is not used by any context
[05:26:57.673711] coredump: SM 104/108 is not used by any context
[05:26:57.673718] coredump: SM 105/108 is not used by any context
[05:26:57.673726] coredump: SM 106/108 is not used by any context
[05:26:57.673735] coredump: SM 107/108 is not used by any context
[05:26:57.673743] coredump: SM 108/108 is not used by any context
[05:26:57.673750] coredump: Device 1/8 has finished state collection
[05:26:57.673771] coredump: SM 1/108 is not used by any context
[05:26:57.673779] coredump: SM 2/108 is not used by any context
[05:26:57.673788] coredump: SM 3/108 is not used by any context
[05:26:57.673799] coredump: SM 4/108 is not used by any context
[05:26:57.673807] coredump: SM 5/108 is not used by any context
[05:26:57.673815] coredump: SM 6/108 is not used by any context
[05:26:57.673822] coredump: SM 7/108 is not used by any context
[05:26:57.673830] coredump: SM 8/108 is not used by any context
[05:26:57.673837] coredump: SM 9/108 is not used by any context
[05:26:57.673844] coredump: SM 10/108 is not used by any context
[05:26:57.673852] coredump: SM 11/108 is not used by any context
[05:26:57.673860] coredump: SM 12/108 is not used by any context
[05:26:57.673868] coredump: SM 13/108 is not used by any context
[05:26:57.673876] coredump: SM 14/108 is not used by any context
[05:26:57.673883] coredump: SM 15/108 is not used by any context
[05:26:57.673891] coredump: SM 16/108 is not used by any context
[05:26:57.673899] coredump: SM 17/108 is not used by any context
[05:26:57.673906] coredump: SM 18/108 is not used by any context
[05:26:57.673914] coredump: SM 19/108 is not used by any context
[05:26:57.673922] coredump: SM 20/108 is not used by any context
[05:26:57.673930] coredump: SM 21/108 is not used by any context
[05:26:57.673938] coredump: SM 22/108 is not used by any context
[05:26:57.673946] coredump: SM 23/108 is not used by any context
[05:26:57.673954] coredump: SM 24/108 is not used by any context
[05:26:57.673962] coredump: SM 25/108 is not used by any context
[05:26:57.673970] coredump: SM 26/108 is not used by any context
[05:26:57.673977] coredump: SM 27/108 is not used by any context
[05:26:57.673985] coredump: SM 28/108 is not used by any context
[05:26:57.673993] coredump: SM 29/108 is not used by any context
[05:26:57.674001] coredump: SM 30/108 is not used by any context
[05:26:57.674009] coredump: SM 31/108 is not used by any context
[05:26:57.674016] coredump: SM 32/108 is not used by any context
[05:26:57.674024] coredump: SM 33/108 is not used by any context
[05:26:57.674032] coredump: SM 34/108 is not used by any context
[05:26:57.674040] coredump: SM 35/108 is not used by any context
[05:26:57.674047] coredump: SM 36/108 is not used by any context
[05:26:57.674055] coredump: SM 37/108 is not used by any context
[05:26:57.674063] coredump: SM 38/108 is not used by any context
[05:26:57.674071] coredump: SM 39/108 is not used by any context
[05:26:57.674079] coredump: SM 40/108 is not used by any context
[05:26:57.674087] coredump: SM 41/108 is not used by any context
[05:26:57.674096] coredump: SM 42/108 is not used by any context
[05:26:57.674104] coredump: SM 43/108 is not used by any context
[05:26:57.674112] coredump: SM 44/108 is not used by any context
[05:26:57.674120] coredump: SM 45/108 is not used by any context
[05:26:57.674128] coredump: SM 46/108 is not used by any context
[05:26:57.674136] coredump: SM 47/108 is not used by any context
[05:26:57.674144] coredump: SM 48/108 is not used by any context
[05:26:57.674154] coredump: SM 49/108 is not used by any context
[05:26:57.674162] coredump: SM 50/108 is not used by any context
[05:26:57.674170] coredump: SM 51/108 is not used by any context
[05:26:57.674178] coredump: SM 52/108 is not used by any context
[05:26:57.674184] coredump: SM 53/108 is not used by any context
[05:26:57.674193] coredump: SM 54/108 is not used by any context
[05:26:57.674201] coredump: SM 55/108 is not used by any context
[05:26:57.674209] coredump: SM 56/108 is not used by any context
[05:26:57.674217] coredump: SM 57/108 is not used by any context
[05:26:57.674225] coredump: SM 58/108 is not used by any context
[05:26:57.674233] coredump: SM 59/108 is not used by any context
[05:26:57.674240] coredump: SM 60/108 is not used by any context
[05:26:57.674248] coredump: SM 61/108 is not used by any context
[05:26:57.674255] coredump: SM 62/108 is not used by any context
[05:26:57.674263] coredump: SM 63/108 is not used by any context
[05:26:57.674271] coredump: SM 64/108 is not used by any context
[05:26:57.674279] coredump: SM 65/108 is not used by any context
[05:26:57.674287] coredump: SM 66/108 is not used by any context
[05:26:57.674293] coredump: SM 67/108 is not used by any context
[05:26:57.674302] coredump: SM 68/108 is not used by any context
[05:26:57.674311] coredump: SM 69/108 is not used by any context
[05:26:57.674319] coredump: SM 70/108 is not used by any context
[05:26:57.674327] coredump: SM 71/108 is not used by any context
[05:26:57.674335] coredump: SM 72/108 is not used by any context
[05:26:57.674343] coredump: SM 73/108 is not used by any context
[05:26:57.674350] coredump: SM 74/108 is not used by any context
[05:26:57.674358] coredump: SM 75/108 is not used by any context
[05:26:57.674366] coredump: SM 76/108 is not used by any context
[05:26:57.674374] coredump: SM 77/108 is not used by any context
[05:26:57.674382] coredump: SM 78/108 is not used by any context
[05:26:57.674389] coredump: SM 79/108 is not used by any context
[05:26:57.674397] coredump: SM 80/108 is not used by any context
[05:26:57.674405] coredump: SM 81/108 is not used by any context
[05:26:57.674413] coredump: SM 82/108 is not used by any context
[05:26:57.674421] coredump: SM 83/108 is not used by any context
[05:26:57.674429] coredump: SM 84/108 is not used by any context
[05:26:57.674436] coredump: SM 85/108 is not used by any context
[05:26:57.674444] coredump: SM 86/108 is not used by any context
[05:26:57.674452] coredump: SM 87/108 is not used by any context
[05:26:57.674460] coredump: SM 88/108 is not used by any context
[05:26:57.674468] coredump: SM 89/108 is not used by any context
[05:26:57.674476] coredump: SM 90/108 is not used by any context
[05:26:57.674483] coredump: SM 91/108 is not used by any context
[05:26:57.674491] coredump: SM 92/108 is not used by any context
[05:26:57.674499] coredump: SM 93/108 is not used by any context
[05:26:57.674507] coredump: SM 94/108 is not used by any context
[05:26:57.674515] coredump: SM 95/108 is not used by any context
[05:26:57.674523] coredump: SM 96/108 is not used by any context
[05:26:57.674531] coredump: SM 97/108 is not used by any context
[05:26:57.674539] coredump: SM 98/108 is not used by any context
[05:26:57.674547] coredump: SM 99/108 is not used by any context
[05:26:57.674555] coredump: SM 100/108 is not used by any context
[05:26:57.674563] coredump: SM 101/108 is not used by any context
[05:26:57.674570] coredump: SM 102/108 is not used by any context
[05:26:57.674578] coredump: SM 103/108 is not used by any context
[05:26:57.674586] coredump: SM 104/108 is not used by any context
[05:26:57.674594] coredump: SM 105/108 is not used by any context
[05:26:57.674613] coredump: SM 106/108 is not used by any context
[05:26:57.674621] coredump: SM 107/108 is not used by any context
[05:26:57.674629] coredump: SM 108/108 is not used by any context
[05:26:57.674636] coredump: Device 2/8 has finished state collection
[05:26:57.674658] coredump: SM 1/108 is not used by any context
[05:26:57.674667] coredump: SM 2/108 is not used by any context
[05:26:57.674676] coredump: SM 3/108 is not used by any context
[05:26:57.674684] coredump: SM 4/108 is not used by any context
[05:26:57.674692] coredump: SM 5/108 is not used by any context
[05:26:57.674700] coredump: SM 6/108 is not used by any context
[05:26:57.674706] coredump: SM 7/108 is not used by any context
[05:26:57.674714] coredump: SM 8/108 is not used by any context
[05:26:57.674722] coredump: SM 9/108 is not used by any context
[05:26:57.674730] coredump: SM 10/108 is not used by any context
[05:26:57.674738] coredump: SM 11/108 is not used by any context
[05:26:57.674746] coredump: SM 12/108 is not used by any context
[05:26:57.674754] coredump: SM 13/108 is not used by any context
[05:26:57.674765] coredump: SM 14/108 is not used by any context
[05:26:57.674773] coredump: SM 15/108 is not used by any context
[05:26:57.674781] coredump: SM 16/108 is not used by any context
[05:26:57.674789] coredump: SM 17/108 is not used by any context
[05:26:57.674797] coredump: SM 18/108 is not used by any context
[05:26:57.674806] coredump: SM 19/108 is not used by any context
[05:26:57.674813] coredump: SM 20/108 is not used by any context
[05:26:57.674820] coredump: SM 21/108 is not used by any context
[05:26:57.674828] coredump: SM 22/108 is not used by any context
[05:26:57.674838] coredump: SM 23/108 is not used by any context
[05:26:57.674845] coredump: SM 24/108 is not used by any context
[05:26:57.674855] coredump: SM 25/108 is not used by any context
[05:26:57.674864] coredump: SM 26/108 is not used by any context
[05:26:57.674872] coredump: SM 27/108 is not used by any context
[05:26:57.674880] coredump: SM 28/108 is not used by any context
[05:26:57.674888] coredump: SM 29/108 is not used by any context
[05:26:57.674898] coredump: SM 30/108 is not used by any context
[05:26:57.674906] coredump: SM 31/108 is not used by any context
[05:26:57.674914] coredump: SM 32/108 is not used by any context
[05:26:57.674921] coredump: SM 33/108 is not used by any context
[05:26:57.674929] coredump: SM 34/108 is not used by any context
[05:26:57.674937] coredump: SM 35/108 is not used by any context
[05:26:57.674945] coredump: SM 36/108 is not used by any context
[05:26:57.674953] coredump: SM 37/108 is not used by any context
[05:26:57.674961] coredump: SM 38/108 is not used by any context
[05:26:57.674969] coredump: SM 39/108 is not used by any context
[05:26:57.674977] coredump: SM 40/108 is not used by any context
[05:26:57.674985] coredump: SM 41/108 is not used by any context
[05:26:57.674993] coredump: SM 42/108 is not used by any context
[05:26:57.675001] coredump: SM 43/108 is not used by any context
[05:26:57.675008] coredump: SM 44/108 is not used by any context
[05:26:57.675016] coredump: SM 45/108 is not used by any context
[05:26:57.675023] coredump: SM 46/108 is not used by any context
[05:26:57.675030] coredump: SM 47/108 is not used by any context
[05:26:57.675038] coredump: SM 48/108 is not used by any context
[05:26:57.675046] coredump: SM 49/108 is not used by any context
[05:26:57.675054] coredump: SM 50/108 is not used by any context
[05:26:57.675062] coredump: SM 51/108 is not used by any context
[05:26:57.675070] coredump: SM 52/108 is not used by any context
[05:26:57.675078] coredump: SM 53/108 is not used by any context
[05:26:57.675086] coredump: SM 54/108 is not used by any context
[05:26:57.675094] coredump: SM 55/108 is not used by any context
[05:26:57.675102] coredump: SM 56/108 is not used by any context
[05:26:57.675110] coredump: SM 57/108 is not used by any context
[05:26:57.675121] coredump: SM 58/108 is not used by any context
[05:26:57.675129] coredump: SM 59/108 is not used by any context
[05:26:57.675137] coredump: SM 60/108 is not used by any context
[05:26:57.675143] coredump: SM 61/108 is not used by any context
[05:26:57.675151] coredump: SM 62/108 is not used by any context
[05:26:57.675159] coredump: SM 63/108 is not used by any context
[05:26:57.675171] coredump: SM 64/108 is not used by any context
[05:26:57.675179] coredump: SM 65/108 is not used by any context
[05:26:57.675187] coredump: SM 66/108 is not used by any context
[05:26:57.675195] coredump: SM 67/108 is not used by any context
[05:26:57.675204] coredump: SM 68/108 is not used by any context
[05:26:57.675212] coredump: SM 69/108 is not used by any context
[05:26:57.675219] coredump: SM 70/108 is not used by any context
[05:26:57.675227] coredump: SM 71/108 is not used by any context
[05:26:57.675238] coredump: SM 72/108 is not used by any context
[05:26:57.675246] coredump: SM 73/108 is not used by any context
[05:26:57.675253] coredump: SM 74/108 is not used by any context
[05:26:57.675261] coredump: SM 75/108 is not used by any context
[05:26:57.675267] coredump: SM 76/108 is not used by any context
[05:26:57.675275] coredump: SM 77/108 is not used by any context
[05:26:57.675283] coredump: SM 78/108 is not used by any context
[05:26:57.675291] coredump: SM 79/108 is not used by any context
[05:26:57.675300] coredump: SM 80/108 is not used by any context
[05:26:57.675308] coredump: SM 81/108 is not used by any context
[05:26:57.675316] coredump: SM 82/108 is not used by any context
[05:26:57.675324] coredump: SM 83/108 is not used by any context
[05:26:57.675332] coredump: SM 84/108 is not used by any context
[05:26:57.675340] coredump: SM 85/108 is not used by any context
[05:26:57.675349] coredump: SM 86/108 is not used by any context
[05:26:57.675356] coredump: SM 87/108 is not used by any context
[05:26:57.675364] coredump: SM 88/108 is not used by any context
[05:26:57.675372] coredump: SM 89/108 is not used by any context
[05:26:57.675380] coredump: SM 90/108 is not used by any context
[05:26:57.675387] coredump: SM 91/108 is not used by any context
[05:26:57.675395] coredump: SM 92/108 is not used by any context
[05:26:57.675403] coredump: SM 93/108 is not used by any context
[05:26:57.675411] coredump: SM 94/108 is not used by any context
[05:26:57.675419] coredump: SM 95/108 is not used by any context
[05:26:57.675427] coredump: SM 96/108 is not used by any context
[05:26:57.675435] coredump: SM 97/108 is not used by any context
[05:26:57.675444] coredump: SM 98/108 is not used by any context
[05:26:57.675452] coredump: SM 99/108 is not used by any context
[05:26:57.675460] coredump: SM 100/108 is not used by any context
[05:26:57.675468] coredump: SM 101/108 is not used by any context
[05:26:57.675476] coredump: SM 102/108 is not used by any context
[05:26:57.675484] coredump: SM 103/108 is not used by any context
[05:26:57.675490] coredump: SM 104/108 is not used by any context
[05:26:57.675498] coredump: SM 105/108 is not used by any context
[05:26:57.675506] coredump: SM 106/108 is not used by any context
[05:26:57.675514] coredump: SM 107/108 is not used by any context
[05:26:57.675523] coredump: SM 108/108 is not used by any context
[05:26:57.675531] coredump: Device 3/8 has finished state collection
[05:26:57.675547] coredump: SM 1/108 is not used by any context
[05:26:57.675556] coredump: SM 2/108 is not used by any context
[05:26:57.675564] coredump: SM 3/108 is not used by any context
[05:26:57.675572] coredump: SM 4/108 is not used by any context
[05:26:57.675580] coredump: SM 5/108 is not used by any context
[05:26:57.675588] coredump: SM 6/108 is not used by any context
[05:26:57.675596] coredump: SM 7/108 is not used by any context
[05:26:57.675602] coredump: SM 8/108 is not used by any context
[05:26:57.675610] coredump: SM 9/108 is not used by any context
[05:26:57.675618] coredump: SM 10/108 is not used by any context
[05:26:57.675626] coredump: SM 11/108 is not used by any context
[05:26:57.675633] coredump: SM 12/108 is not used by any context
[05:26:57.675641] coredump: SM 13/108 is not used by any context
[05:26:57.675650] coredump: SM 14/108 is not used by any context
[05:26:57.675657] coredump: SM 15/108 is not used by any context
[05:26:57.675665] coredump: SM 16/108 is not used by any context
[05:26:57.675674] coredump: SM 17/108 is not used by any context
[05:26:57.675682] coredump: SM 18/108 is not used by any context
[05:26:57.675690] coredump: SM 19/108 is not used by any context
[05:26:57.675697] coredump: SM 20/108 is not used by any context
[05:26:57.675707] coredump: SM 21/108 is not used by any context
[05:26:57.675715] coredump: SM 22/108 is not used by any context
[05:26:57.675723] coredump: SM 23/108 is not used by any context
[05:26:57.675731] coredump: SM 24/108 is not used by any context
[05:26:57.675739] coredump: SM 25/108 is not used by any context
[05:26:57.675746] coredump: SM 26/108 is not used by any context
[05:26:57.675753] coredump: SM 27/108 is not used by any context
[05:26:57.675761] coredump: SM 28/108 is not used by any context
[05:26:57.675770] coredump: SM 29/108 is not used by any context
[05:26:57.675778] coredump: SM 30/108 is not used by any context
[05:26:57.675786] coredump: SM 31/108 is not used by any context
[05:26:57.675797] coredump: SM 32/108 is not used by any context
[05:26:57.675805] coredump: SM 33/108 is not used by any context
[05:26:57.675813] coredump: SM 34/108 is not used by any context
[05:26:57.675821] coredump: SM 35/108 is not used by any context
[05:26:57.675829] coredump: SM 36/108 is not used by any context
[05:26:57.675836] coredump: SM 37/108 is not used by any context
[05:26:57.675843] coredump: SM 38/108 is not used by any context
[05:26:57.675851] coredump: SM 39/108 is not used by any context
[05:26:57.675858] coredump: SM 40/108 is not used by any context
[05:26:57.675864] coredump: SM 41/108 is not used by any context
[05:26:57.675872] coredump: SM 42/108 is not used by any context
[05:26:57.675882] coredump: SM 43/108 is not used by any context
[05:26:57.675890] coredump: SM 44/108 is not used by any context
[05:26:57.675898] coredump: SM 45/108 is not used by any context
[05:26:57.675904] coredump: SM 46/108 is not used by any context
[05:26:57.675911] coredump: SM 47/108 is not used by any context
[05:26:57.675920] coredump: SM 48/108 is not used by any context
[05:26:57.675928] coredump: SM 49/108 is not used by any context
[05:26:57.675937] coredump: SM 50/108 is not used by any context
[05:26:57.675945] coredump: SM 51/108 is not used by any context
[05:26:57.675953] coredump: SM 52/108 is not used by any context
[05:26:57.675961] coredump: SM 53/108 is not used by any context
[05:26:57.675969] coredump: SM 54/108 is not used by any context
[05:26:57.675979] coredump: SM 55/108 is not used by any context
[05:26:57.675987] coredump: SM 56/108 is not used by any context
[05:26:57.675995] coredump: SM 57/108 is not used by any context
[05:26:57.676002] coredump: SM 58/108 is not used by any context
[05:26:57.676010] coredump: SM 59/108 is not used by any context
[05:26:57.676018] coredump: SM 60/108 is not used by any context
[05:26:57.676026] coredump: SM 61/108 is not used by any context
[05:26:57.676034] coredump: SM 62/108 is not used by any context
[05:26:57.676042] coredump: SM 63/108 is not used by any context
[05:26:57.676050] coredump: SM 64/108 is not used by any context
[05:26:57.676058] coredump: SM 65/108 is not used by any context
[05:26:57.676066] coredump: SM 66/108 is not used by any context
[05:26:57.676074] coredump: SM 67/108 is not used by any context
[05:26:57.676082] coredump: SM 68/108 is not used by any context
[05:26:57.676090] coredump: SM 69/108 is not used by any context
[05:26:57.676098] coredump: SM 70/108 is not used by any context
[05:26:57.676106] coredump: SM 71/108 is not used by any context
[05:26:57.676114] coredump: SM 72/108 is not used by any context
[05:26:57.676122] coredump: SM 73/108 is not used by any context
[05:26:57.676130] coredump: SM 74/108 is not used by any context
[05:26:57.676138] coredump: SM 75/108 is not used by any context
[05:26:57.676146] coredump: SM 76/108 is not used by any context
[05:26:57.676154] coredump: SM 77/108 is not used by any context
[05:26:57.676160] coredump: SM 78/108 is not used by any context
[05:26:57.676168] coredump: SM 79/108 is not used by any context
[05:26:57.676176] coredump: SM 80/108 is not used by any context
[05:26:57.676184] coredump: SM 81/108 is not used by any context
[05:26:57.676191] coredump: SM 82/108 is not used by any context
[05:26:57.676199] coredump: SM 83/108 is not used by any context
[05:26:57.676207] coredump: SM 84/108 is not used by any context
[05:26:57.676215] coredump: SM 85/108 is not used by any context
[05:26:57.676222] coredump: SM 86/108 is not used by any context
[05:26:57.676230] coredump: SM 87/108 is not used by any context
[05:26:57.676238] coredump: SM 88/108 is not used by any context
[05:26:57.676246] coredump: SM 89/108 is not used by any context
[05:26:57.676253] coredump: SM 90/108 is not used by any context
[05:26:57.676261] coredump: SM 91/108 is not used by any context
[05:26:57.676269] coredump: SM 92/108 is not used by any context
[05:26:57.676277] coredump: SM 93/108 is not used by any context
[05:26:57.676284] coredump: SM 94/108 is not used by any context
[05:26:57.676292] coredump: SM 95/108 is not used by any context
[05:26:57.676300] coredump: SM 96/108 is not used by any context
[05:26:57.676308] coredump: SM 97/108 is not used by any context
[05:26:57.676317] coredump: SM 98/108 is not used by any context
[05:26:57.676324] coredump: SM 99/108 is not used by any context
[05:26:57.676333] coredump: SM 100/108 is not used by any context
[05:26:57.676341] coredump: SM 101/108 is not used by any context
[05:26:57.676349] coredump: SM 102/108 is not used by any context
[05:26:57.676357] coredump: SM 103/108 is not used by any context
[05:26:57.676365] coredump: SM 104/108 is not used by any context
[05:26:57.676374] coredump: SM 105/108 is not used by any context
[05:26:57.676383] coredump: SM 106/108 is not used by any context
[05:26:57.676391] coredump: SM 107/108 is not used by any context
[05:26:57.676399] coredump: SM 108/108 is not used by any context
[05:26:57.676406] coredump: Device 4/8 has finished state collection
[05:26:57.676461] coredump: SM 1/108 has finished state collection
[05:26:57.676471] coredump: SM 2/108 has finished state collection
[05:26:57.676480] coredump: SM 3/108 has finished state collection
[05:26:57.676489] coredump: SM 4/108 has finished state collection
[05:26:57.676497] coredump: SM 5/108 has finished state collection
[05:26:57.676506] coredump: SM 6/108 has finished state collection
[05:26:57.676514] coredump: SM 7/108 has finished state collection
[05:26:57.676523] coredump: SM 8/108 has finished state collection
[05:26:57.676531] coredump: SM 9/108 has finished state collection
[05:26:57.676539] coredump: SM 10/108 has finished state collection
[05:26:57.676547] coredump: SM 11/108 has finished state collection
[05:26:57.676556] coredump: SM 12/108 has finished state collection
[05:26:57.676564] coredump: SM 13/108 has finished state collection
[05:26:57.676575] coredump: SM 14/108 has finished state collection
[05:26:57.676584] coredump: SM 15/108 has finished state collection
[05:26:57.676592] coredump: SM 16/108 has finished state collection
[05:26:57.676601] coredump: SM 17/108 has finished state collection
[05:26:57.676610] coredump: SM 18/108 has finished state collection
[05:26:57.676618] coredump: SM 19/108 has finished state collection
[05:26:57.676626] coredump: SM 20/108 has finished state collection
[05:26:57.676635] coredump: SM 21/108 has finished state collection
[05:26:57.676643] coredump: SM 22/108 has finished state collection
[05:26:57.676651] coredump: SM 23/108 has finished state collection
[05:26:57.676661] coredump: SM 24/108 has finished state collection
[05:26:57.676670] coredump: SM 25/108 has finished state collection
[05:26:57.676678] coredump: SM 26/108 has finished state collection
[05:26:57.676686] coredump: SM 27/108 has finished state collection
[05:26:57.676696] coredump: SM 28/108 has finished state collection
[05:26:57.676704] coredump: SM 29/108 has finished state collection
[05:26:57.676712] coredump: SM 30/108 has finished state collection
[05:26:57.676721] coredump: SM 31/108 has finished state collection
[05:26:57.676728] coredump: SM 32/108 has finished state collection
[05:26:57.676736] coredump: SM 33/108 has finished state collection
[05:26:57.676745] coredump: SM 34/108 has finished state collection
[05:26:57.676753] coredump: SM 35/108 has finished state collection
[05:26:57.676762] coredump: SM 36/108 has finished state collection
[05:26:57.676771] coredump: SM 37/108 has finished state collection
[05:26:57.676779] coredump: SM 38/108 has finished state collection
[05:26:57.676787] coredump: SM 39/108 has finished state collection
[05:26:57.676799] coredump: SM 40/108 has finished state collection
[05:26:57.676808] coredump: SM 41/108 has finished state collection
[05:26:57.676816] coredump: SM 42/108 has finished state collection
[05:26:57.676824] coredump: SM 43/108 has finished state collection
[05:26:57.676832] coredump: SM 44/108 has finished state collection
[05:26:57.676841] coredump: SM 45/108 has finished state collection
[05:26:57.676850] coredump: SM 46/108 has finished state collection
[05:26:57.676858] coredump: SM 47/108 has finished state collection
[05:26:57.676867] coredump: SM 48/108 has finished state collection
[05:26:57.676875] coredump: SM 49/108 has finished state collection
[05:26:57.676883] coredump: SM 50/108 has finished state collection
[05:26:57.676892] coredump: SM 51/108 has finished state collection
[05:26:57.676900] coredump: SM 52/108 has finished state collection
[05:26:57.676908] coredump: SM 53/108 has finished state collection
[05:26:57.676915] coredump: SM 54/108 has finished state collection
[05:26:57.676924] coredump: SM 55/108 has finished state collection
[05:26:57.676932] coredump: SM 56/108 has finished state collection
[05:26:57.676941] coredump: SM 57/108 has finished state collection
[05:26:57.676949] coredump: SM 58/108 has finished state collection
[05:26:57.676958] coredump: SM 59/108 has finished state collection
[05:26:57.676966] coredump: SM 60/108 has finished state collection
[05:26:57.676981] coredump: Detected an exception of type CUDBG_EXCEPTION_WARP_ILLEGAL_ADDRESS (14)
[05:26:57.676991] coredump: - Device: 4
[05:26:57.676999] coredump: - SM: 60
[05:26:57.677007] coredump: - Warp: 1
[05:26:57.677015] coredump: - PC 0x7f7a47b4df00
[05:26:57.677988] coredump: Stack trace (lane masks: active 0xffffffff, valid 0xffffffff):
[05:26:57.678012] coredump: #0 0x7f7a47b4dfb0 memcpy_triton_kernel
[05:26:57.687009] coredump: SM 61/108 has finished state collection
[05:26:57.687023] coredump: SM 62/108 has finished state collection
[05:26:57.687031] coredump: SM 63/108 has finished state collection
[05:26:57.687039] coredump: SM 64/108 has finished state collection
[05:26:57.687046] coredump: SM 65/108 has finished state collection
[05:26:57.687054] coredump: SM 66/108 has finished state collection
[05:26:57.687061] coredump: SM 67/108 has finished state collection
[05:26:57.687068] coredump: SM 68/108 has finished state collection
[05:26:57.687076] coredump: SM 69/108 has finished state collection
[05:26:57.687083] coredump: SM 70/108 has finished state collection
[05:26:57.687090] coredump: SM 71/108 has finished state collection
[05:26:57.687098] coredump: SM 72/108 has finished state collection
[05:26:57.687105] coredump: SM 73/108 has finished state collection
[05:26:57.687113] coredump: SM 74/108 has finished state collection
[05:26:57.687120] coredump: SM 75/108 has finished state collection
[05:26:57.687128] coredump: SM 76/108 has finished state collection
[05:26:57.687136] coredump: SM 77/108 has finished state collection
[05:26:57.687144] coredump: SM 78/108 has finished state collection
[05:26:57.687150] coredump: SM 79/108 has finished state collection
[05:26:57.687157] coredump: SM 80/108 has finished state collection
[05:26:57.687164] coredump: SM 81/108 has finished state collection
[05:26:57.687171] coredump: SM 82/108 has finished state collection
[05:26:57.687179] coredump: SM 83/108 has finished state collection
[05:26:57.687187] coredump: SM 84/108 has finished state collection
[05:26:57.687194] coredump: SM 85/108 has finished state collection
[05:26:57.687200] coredump: SM 86/108 has finished state collection
[05:26:57.687207] coredump: SM 87/108 has finished state collection
[05:26:57.687214] coredump: SM 88/108 has finished state collection
[05:26:57.687221] coredump: SM 89/108 has finished state collection
[05:26:57.687229] coredump: SM 90/108 has finished state collection
[05:26:57.687235] coredump: SM 91/108 has finished state collection
[05:26:57.687243] coredump: SM 92/108 has finished state collection
[05:26:57.687250] coredump: SM 93/108 has finished state collection
[05:26:57.687257] coredump: SM 94/108 has finished state collection
[05:26:57.687265] coredump: SM 95/108 has finished state collection
[05:26:57.687272] coredump: SM 96/108 has finished state collection
[05:26:57.687279] coredump: SM 97/108 has finished state collection
[05:26:57.687287] coredump: SM 98/108 has finished state collection
[05:26:57.687293] coredump: SM 99/108 has finished state collection
[05:26:57.687301] coredump: SM 100/108 has finished state collection
[05:26:57.687308] coredump: SM 101/108 has finished state collection
[05:26:57.687316] coredump: SM 102/108 has finished state collection
[05:26:57.687323] coredump: SM 103/108 has finished state collection
[05:26:57.687333] coredump: SM 104/108 has finished state collection
[05:26:57.687341] coredump: SM 105/108 has finished state collection
[05:26:57.687348] coredump: SM 106/108 has finished state collection
[05:26:57.687355] coredump: SM 107/108 has finished state collection
[05:26:57.687362] coredump: SM 108/108 has finished state collection
[05:26:57.687733] coredump: Device 5/8 has finished state collection
[05:26:57.687753] coredump: SM 1/108 is not used by any context
[05:26:57.687761] coredump: SM 2/108 is not used by any context
[05:26:57.687768] coredump: SM 3/108 is not used by any context
[05:26:57.687775] coredump: SM 4/108 is not used by any context
[05:26:57.687781] coredump: SM 5/108 is not used by any context
[05:26:57.687790] coredump: SM 6/108 is not used by any context
[05:26:57.687797] coredump: SM 7/108 is not used by any context
[05:26:57.687804] coredump: SM 8/108 is not used by any context
[05:26:57.687811] coredump: SM 9/108 is not used by any context
[05:26:57.687816] coredump: SM 10/108 is not used by any context
[05:26:57.687823] coredump: SM 11/108 is not used by any context
[05:26:57.687830] coredump: SM 12/108 is not used by any context
[05:26:57.687836] coredump: SM 13/108 is not used by any context
[05:26:57.687841] coredump: SM 14/108 is not used by any context
[05:26:57.687848] coredump: SM 15/108 is not used by any context
[05:26:57.687854] coredump: SM 16/108 is not used by any context
[05:26:57.687861] coredump: SM 17/108 is not used by any context
[05:26:57.687868] coredump: SM 18/108 is not used by any context
[05:26:57.687874] coredump: SM 19/108 is not used by any context
[05:26:57.687881] coredump: SM 20/108 is not used by any context
[05:26:57.687888] coredump: SM 21/108 is not used by any context
[05:26:57.687894] coredump: SM 22/108 is not used by any context
[05:26:57.687901] coredump: SM 23/108 is not used by any context
[05:26:57.687907] coredump: SM 24/108 is not used by any context
[05:26:57.687914] coredump: SM 25/108 is not used by any context
[05:26:57.687919] coredump: SM 26/108 is not used by any context
[05:26:57.687926] coredump: SM 27/108 is not used by any context
[05:26:57.687933] coredump: SM 28/108 is not used by any context
[05:26:57.687939] coredump: SM 29/108 is not used by any context
[05:26:57.687946] coredump: SM 30/108 is not used by any context
[05:26:57.687951] coredump: SM 31/108 is not used by any context
[05:26:57.687957] coredump: SM 32/108 is not used by any context
[05:26:57.687964] coredump: SM 33/108 is not used by any context
[05:26:57.687972] coredump: SM 34/108 is not used by any context
[05:26:57.687979] coredump: SM 35/108 is not used by any context
[05:26:57.687987] coredump: SM 36/108 is not used by any context
[05:26:57.687995] coredump: SM 37/108 is not used by any context
[05:26:57.688002] coredump: SM 38/108 is not used by any context
[05:26:57.688009] coredump: SM 39/108 is not used by any context
[05:26:57.688015] coredump: SM 40/108 is not used by any context
[05:26:57.688022] coredump: SM 41/108 is not used by any context
[05:26:57.688028] coredump: SM 42/108 is not used by any context
[05:26:57.688035] coredump: SM 43/108 is not used by any context
[05:26:57.688041] coredump: SM 44/108 is not used by any context
[05:26:57.688050] coredump: SM 45/108 is not used by any context
[05:26:57.688057] coredump: SM 46/108 is not used by any context
[05:26:57.688063] coredump: SM 47/108 is not used by any context
[05:26:57.688070] coredump: SM 48/108 is not used by any context
[05:26:57.688076] coredump: SM 49/108 is not used by any context
[05:26:57.688084] coredump: SM 50/108 is not used by any context
[05:26:57.688091] coredump: SM 51/108 is not used by any context
[05:26:57.688096] coredump: SM 52/108 is not used by any context
[05:26:57.688102] coredump: SM 53/108 is not used by any context
[05:26:57.688109] coredump: SM 54/108 is not used by any context
[05:26:57.688116] coredump: SM 55/108 is not used by any context
[05:26:57.688121] coredump: SM 56/108 is not used by any context
[05:26:57.688128] coredump: SM 57/108 is not used by any context
[05:26:57.688134] coredump: SM 58/108 is not used by any context
[05:26:57.688141] coredump: SM 59/108 is not used by any context
[05:26:57.688147] coredump: SM 60/108 is not used by any context
[05:26:57.688154] coredump: SM 61/108 is not used by any context
[05:26:57.688159] coredump: SM 62/108 is not used by any context
[05:26:57.688165] coredump: SM 63/108 is not used by any context
[05:26:57.688172] coredump: SM 64/108 is not used by any context
[05:26:57.688179] coredump: SM 65/108 is not used by any context
[05:26:57.688185] coredump: SM 66/108 is not used by any context
[05:26:57.688191] coredump: SM 67/108 is not used by any context
[05:26:57.688197] coredump: SM 68/108 is not used by any context
[05:26:57.688203] coredump: SM 69/108 is not used by any context
[05:26:57.688210] coredump: SM 70/108 is not used by any context
[05:26:57.688217] coredump: SM 71/108 is not used by any context
[05:26:57.688223] coredump: SM 72/108 is not used by any context
[05:26:57.688231] coredump: SM 73/108 is not used by any context
[05:26:57.688237] coredump: SM 74/108 is not used by any context
[05:26:57.688244] coredump: SM 75/108 is not used by any context
[05:26:57.688249] coredump: SM 76/108 is not used by any context
[05:26:57.688256] coredump: SM 77/108 is not used by any context
[05:26:57.688263] coredump: SM 78/108 is not used by any context
[05:26:57.688270] coredump: SM 79/108 is not used by any context
[05:26:57.688276] coredump: SM 80/108 is not used by any context
[05:26:57.688283] coredump: SM 81/108 is not used by any context
[05:26:57.688289] coredump: SM 82/108 is not used by any context
[05:26:57.688296] coredump: SM 83/108 is not used by any context
[05:26:57.688303] coredump: SM 84/108 is not used by any context
[05:26:57.688309] coredump: SM 85/108 is not used by any context
[05:26:57.688316] coredump: SM 86/108 is not used by any context
[05:26:57.688322] coredump: SM 87/108 is not used by any context
[05:26:57.688328] coredump: SM 88/108 is not used by any context
[05:26:57.688335] coredump: SM 89/108 is not used by any context
[05:26:57.688342] coredump: SM 90/108 is not used by any context
[05:26:57.688350] coredump: SM 91/108 is not used by any context
[05:26:57.688357] coredump: SM 92/108 is not used by any context
[05:26:57.688363] coredump: SM 93/108 is not used by any context
[05:26:57.688370] coredump: SM 94/108 is not used by any context
[05:26:57.688378] coredump: SM 95/108 is not used by any context
[05:26:57.688383] coredump: SM 96/108 is not used by any context
[05:26:57.688390] coredump: SM 97/108 is not used by any context
[05:26:57.688396] coredump: SM 98/108 is not used by any context
[05:26:57.688403] coredump: SM 99/108 is not used by any context
[05:26:57.688408] coredump: SM 100/108 is not used by any context
[05:26:57.688418] coredump: SM 101/108 is not used by any context
[05:26:57.688424] coredump: SM 102/108 is not used by any context
[05:26:57.688431] coredump: SM 103/108 is not used by any context
[05:26:57.688436] coredump: SM 104/108 is not used by any context
[05:26:57.688443] coredump: SM 105/108 is not used by any context
[05:26:57.688450] coredump: SM 106/108 is not used by any context
[05:26:57.688457] coredump: SM 107/108 is not used by any context
[05:26:57.688463] coredump: SM 108/108 is not used by any context
[05:26:57.688469] coredump: Device 6/8 has finished state collection
[05:26:57.688484] coredump: SM 1/108 is not used by any context
[05:26:57.688491] coredump: SM 2/108 is not used by any context
[05:26:57.688498] coredump: SM 3/108 is not used by any context
[05:26:57.688505] coredump: SM 4/108 is not used by any context
[05:26:57.688511] coredump: SM 5/108 is not used by any context
[05:26:57.688518] coredump: SM 6/108 is not used by any context
[05:26:57.688525] coredump: SM 7/108 is not used by any context
[05:26:57.688534] coredump: SM 8/108 is not used by any context
[05:26:57.688541] coredump: SM 9/108 is not used by any context
[05:26:57.688549] coredump: SM 10/108 is not used by any context
[05:26:57.688556] coredump: SM 11/108 is not used by any context
[05:26:57.688562] coredump: SM 12/108 is not used by any context
[05:26:57.688569] coredump: SM 13/108 is not used by any context
[05:26:57.688578] coredump: SM 14/108 is not used by any context
[05:26:57.688585] coredump: SM 15/108 is not used by any context
[05:26:57.688592] coredump: SM 16/108 is not used by any context
[05:26:57.688599] coredump: SM 17/108 is not used by any context
[05:26:57.688605] coredump: SM 18/108 is not used by any context
[05:26:57.688612] coredump: SM 19/108 is not used by any context
[05:26:57.688619] coredump: SM 20/108 is not used by any context
[05:26:57.688626] coredump: SM 21/108 is not used by any context
[05:26:57.688636] coredump: SM 22/108 is not used by any context
[05:26:57.688643] coredump: SM 23/108 is not used by any context
[05:26:57.688649] coredump: SM 24/108 is not used by any context
[05:26:57.688655] coredump: SM 25/108 is not used by any context
[05:26:57.688662] coredump: SM 26/108 is not used by any context
[05:26:57.688669] coredump: SM 27/108 is not used by any context
[05:26:57.688676] coredump: SM 28/108 is not used by any context
[05:26:57.688682] coredump: SM 29/108 is not used by any context
[05:26:57.688689] coredump: SM 30/108 is not used by any context
[05:26:57.688696] coredump: SM 31/108 is not used by any context
[05:26:57.688705] coredump: SM 32/108 is not used by any context
[05:26:57.688712] coredump: SM 33/108 is not used by any context
[05:26:57.688720] coredump: SM 34/108 is not used by any context
[05:26:57.688726] coredump: SM 35/108 is not used by any context
[05:26:57.688733] coredump: SM 36/108 is not used by any context
[05:26:57.688740] coredump: SM 37/108 is not used by any context
[05:26:57.688747] coredump: SM 38/108 is not used by any context
[05:26:57.688753] coredump: SM 39/108 is not used by any context
[05:26:57.688760] coredump: SM 40/108 is not used by any context
[05:26:57.688765] coredump: SM 41/108 is not used by any context
[05:26:57.688772] coredump: SM 42/108 is not used by any context
[05:26:57.688777] coredump: SM 43/108 is not used by any context
[05:26:57.688783] coredump: SM 44/108 is not used by any context
[05:26:57.688790] coredump: SM 45/108 is not used by any context
[05:26:57.688796] coredump: SM 46/108 is not used by any context
[05:26:57.688803] coredump: SM 47/108 is not used by any context
[05:26:57.688808] coredump: SM 48/108 is not used by any context
[05:26:57.688815] coredump: SM 49/108 is not used by any context
[05:26:57.688822] coredump: SM 50/108 is not used by any context
[05:26:57.688829] coredump: SM 51/108 is not used by any context
[05:26:57.688836] coredump: SM 52/108 is not used by any context
[05:26:57.688844] coredump: SM 53/108 is not used by any context
[05:26:57.688851] coredump: SM 54/108 is not used by any context
[05:26:57.688856] coredump: SM 55/108 is not used by any context
[05:26:57.688863] coredump: SM 56/108 is not used by any context
[05:26:57.688870] coredump: SM 57/108 is not used by any context
[05:26:57.688876] coredump: SM 58/108 is not used by any context
[05:26:57.688883] coredump: SM 59/108 is not used by any context
[05:26:57.688890] coredump: SM 60/108 is not used by any context
[05:26:57.688896] coredump: SM 61/108 is not used by any context
[05:26:57.688903] coredump: SM 62/108 is not used by any context
[05:26:57.688910] coredump: SM 63/108 is not used by any context
[05:26:57.688917] coredump: SM 64/108 is not used by any context
[05:26:57.688924] coredump: SM 65/108 is not used by any context
[05:26:57.688930] coredump: SM 66/108 is not used by any context
[05:26:57.688937] coredump: SM 67/108 is not used by any context
[05:26:57.688944] coredump: SM 68/108 is not used by any context
[05:26:57.688950] coredump: SM 69/108 is not used by any context
[05:26:57.688957] coredump: SM 70/108 is not used by any context
[05:26:57.688963] coredump: SM 71/108 is not used by any context
[05:26:57.688970] coredump: SM 72/108 is not used by any context
[05:26:57.688976] coredump: SM 73/108 is not used by any context
[05:26:57.688982] coredump: SM 74/108 is not used by any context
[05:26:57.688989] coredump: SM 75/108 is not used by any context
[05:26:57.688995] coredump: SM 76/108 is not used by any context
[05:26:57.689002] coredump: SM 77/108 is not used by any context
[05:26:57.689009] coredump: SM 78/108 is not used by any context
[05:26:57.689018] coredump: SM 79/108 is not used by any context
[05:26:57.689025] coredump: SM 80/108 is not used by any context
[05:26:57.689032] coredump: SM 81/108 is not used by any context
[05:26:57.689039] coredump: SM 82/108 is not used by any context
[05:26:57.689046] coredump: SM 83/108 is not used by any context
[05:26:57.689053] coredump: SM 84/108 is not used by any context
[05:26:57.689059] coredump: SM 85/108 is not used by any context
[05:26:57.689064] coredump: SM 86/108 is not used by any context
[05:26:57.689071] coredump: SM 87/108 is not used by any context
[05:26:57.689078] coredump: SM 88/108 is not used by any context
[05:26:57.689084] coredump: SM 89/108 is not used by any context
[05:26:57.689091] coredump: SM 90/108 is not used by any context
[05:26:57.689098] coredump: SM 91/108 is not used by any context
[05:26:57.689104] coredump: SM 92/108 is not used by any context
[05:26:57.689110] coredump: SM 93/108 is not used by any context
[05:26:57.689117] coredump: SM 94/108 is not used by any context
[05:26:57.689123] coredump: SM 95/108 is not used by any context
[05:26:57.689130] coredump: SM 96/108 is not used by any context
[05:26:57.689137] coredump: SM 97/108 is not used by any context
[05:26:57.689144] coredump: SM 98/108 is not used by any context
[05:26:57.689150] coredump: SM 99/108 is not used by any context
[05:26:57.689158] coredump: SM 100/108 is not used by any context
[05:26:57.689164] coredump: SM 101/108 is not used by any context
[05:26:57.689171] coredump: SM 102/108 is not used by any context
[05:26:57.689178] coredump: SM 103/108 is not used by any context
[05:26:57.689187] coredump: SM 104/108 is not used by any context
[05:26:57.689194] coredump: SM 105/108 is not used by any context
[05:26:57.689201] coredump: SM 106/108 is not used by any context
[05:26:57.689208] coredump: SM 107/108 is not used by any context
[05:26:57.689215] coredump: SM 108/108 is not used by any context
[05:26:57.689220] coredump: Device 7/8 has finished state collection
[05:26:57.689233] coredump: SM 1/108 is not used by any context
[05:26:57.689240] coredump: SM 2/108 is not used by any context
[05:26:57.689245] coredump: SM 3/108 is not used by any context
[05:26:57.689252] coredump: SM 4/108 is not used by any context
[05:26:57.689259] coredump: SM 5/108 is not used by any context
[05:26:57.689265] coredump: SM 6/108 is not used by any context
[05:26:57.689273] coredump: SM 7/108 is not used by any context
[05:26:57.689280] coredump: SM 8/108 is not used by any context
[05:26:57.689288] coredump: SM 9/108 is not used by any context
[05:26:57.689294] coredump: SM 10/108 is not used by any context
[05:26:57.689300] coredump: SM 11/108 is not used by any context
[05:26:57.689307] coredump: SM 12/108 is not used by any context
[05:26:57.689314] coredump: SM 13/108 is not used by any context
[05:26:57.689320] coredump: SM 14/108 is not used by any context
[05:26:57.689327] coredump: SM 15/108 is not used by any context
[05:26:57.689334] coredump: SM 16/108 is not used by any context
[05:26:57.689339] coredump: SM 17/108 is not used by any context
[05:26:57.689346] coredump: SM 18/108 is not used by any context
[05:26:57.689355] coredump: SM 19/108 is not used by any context
[05:26:57.689362] coredump: SM 20/108 is not used by any context
[05:26:57.689371] coredump: SM 21/108 is not used by any context
[05:26:57.689378] coredump: SM 22/108 is not used by any context
[05:26:57.689385] coredump: SM 23/108 is not used by any context
[05:26:57.689392] coredump: SM 24/108 is not used by any context
[05:26:57.689398] coredump: SM 25/108 is not used by any context
[05:26:57.689406] coredump: SM 26/108 is not used by any context
[05:26:57.689411] coredump: SM 27/108 is not used by any context
[05:26:57.689418] coredump: SM 28/108 is not used by any context
[05:26:57.689425] coredump: SM 29/108 is not used by any context
[05:26:57.689431] coredump: SM 30/108 is not used by any context
[05:26:57.689436] coredump: SM 31/108 is not used by any context
[05:26:57.689443] coredump: SM 32/108 is not used by any context
[05:26:57.689450] coredump: SM 33/108 is not used by any context
[05:26:57.689456] coredump: SM 34/108 is not used by any context
[05:26:57.689463] coredump: SM 35/108 is not used by any context
[05:26:57.689470] coredump: SM 36/108 is not used by any context
[05:26:57.689476] coredump: SM 37/108 is not used by any context
[05:26:57.689483] coredump: SM 38/108 is not used by any context
[05:26:57.689490] coredump: SM 39/108 is not used by any context
[05:26:57.689496] coredump: SM 40/108 is not used by any context
[05:26:57.689503] coredump: SM 41/108 is not used by any context
[05:26:57.689510] coredump: SM 42/108 is not used by any context
[05:26:57.689515] coredump: SM 43/108 is not used by any context
[05:26:57.689522] coredump: SM 44/108 is not used by any context
[05:26:57.689528] coredump: SM 45/108 is not used by any context
[05:26:57.689535] coredump: SM 46/108 is not used by any context
[05:26:57.689542] coredump: SM 47/108 is not used by any context
[05:26:57.689547] coredump: SM 48/108 is not used by any context
[05:26:57.689554] coredump: SM 49/108 is not used by any context
[05:26:57.689560] coredump: SM 50/108 is not used by any context
[05:26:57.689565] coredump: SM 51/108 is not used by any context
[05:26:57.689572] coredump: SM 52/108 is not used by any context
[05:26:57.689579] coredump: SM 53/108 is not used by any context
[05:26:57.689585] coredump: SM 54/108 is not used by any context
[05:26:57.689592] coredump: SM 55/108 is not used by any context
[05:26:57.689598] coredump: SM 56/108 is not used by any context
[05:26:57.689604] coredump: SM 57/108 is not used by any context
[05:26:57.689610] coredump: SM 58/108 is not used by any context
[05:26:57.689617] coredump: SM 59/108 is not used by any context
[05:26:57.689624] coredump: SM 60/108 is not used by any context
[05:26:57.689633] coredump: SM 61/108 is not used by any context
[05:26:57.689640] coredump: SM 62/108 is not used by any context
[05:26:57.689647] coredump: SM 63/108 is not used by any context
[05:26:57.689653] coredump: SM 64/108 is not used by any context
[05:26:57.689660] coredump: SM 65/108 is not used by any context
[05:26:57.689666] coredump: SM 66/108 is not used by any context
[05:26:57.689672] coredump: SM 67/108 is not used by any context
[05:26:57.689679] coredump: SM 68/108 is not used by any context
[05:26:57.689686] coredump: SM 69/108 is not used by any context
[05:26:57.689693] coredump: SM 70/108 is not used by any context
[05:26:57.689700] coredump: SM 71/108 is not used by any context
[05:26:57.689707] coredump: SM 72/108 is not used by any context
[05:26:57.689716] coredump: SM 73/108 is not used by any context
[05:26:57.689723] coredump: SM 74/108 is not used by any context
[05:26:57.689730] coredump: SM 75/108 is not used by any context
[05:26:57.689737] coredump: SM 76/108 is not used by any context
[05:26:57.689743] coredump: SM 77/108 is not used by any context
[05:26:57.689750] coredump: SM 78/108 is not used by any context
[05:26:57.689757] coredump: SM 79/108 is not used by any context
[05:26:57.689764] coredump: SM 80/108 is not used by any context
[05:26:57.689770] coredump: SM 81/108 is not used by any context
[05:26:57.689776] coredump: SM 82/108 is not used by any context
[05:26:57.689783] coredump: SM 83/108 is not used by any context
[05:26:57.689790] coredump: SM 84/108 is not used by any context
[05:26:57.689797] coredump: SM 85/108 is not used by any context
[05:26:57.689803] coredump: SM 86/108 is not used by any context
[05:26:57.689809] coredump: SM 87/108 is not used by any context
[05:26:57.689815] coredump: SM 88/108 is not used by any context
[05:26:57.689823] coredump: SM 89/108 is not used by any context
[05:26:57.689832] coredump: SM 90/108 is not used by any context
[05:26:57.689838] coredump: SM 91/108 is not used by any context
[05:26:57.689845] coredump: SM 92/108 is not used by any context
[05:26:57.689853] coredump: SM 93/108 is not used by any context
[05:26:57.689859] coredump: SM 94/108 is not used by any context
[05:26:57.689866] coredump: SM 95/108 is not used by any context
[05:26:57.689873] coredump: SM 96/108 is not used by any context
[05:26:57.689880] coredump: SM 97/108 is not used by any context
[05:26:57.689885] coredump: SM 98/108 is not used by any context
[05:26:57.689892] coredump: SM 99/108 is not used by any context
[05:26:57.689897] coredump: SM 100/108 is not used by any context
[05:26:57.689904] coredump: SM 101/108 is not used by any context
[05:26:57.689910] coredump: SM 102/108 is not used by any context
[05:26:57.689919] coredump: SM 103/108 is not used by any context
[05:26:57.689925] coredump: SM 104/108 is not used by any context
[05:26:57.689934] coredump: SM 105/108 is not used by any context
[05:26:57.689941] coredump: SM 106/108 is not used by any context
[05:26:57.689948] coredump: SM 107/108 is not used by any context
[05:26:57.689953] coredump: SM 108/108 is not used by any context
[05:26:57.689959] coredump: Device 8/8 has finished state collection
[05:26:57.690330] coredump: Calculating ELF file layout
[05:26:57.691492] coredump: ELF file layout calculated
[05:26:57.691500] coredump: Writing ELF file to /sgl-workspace/coredump.yy-lite-decode-0.107753
[05:26:57.691514] coredump: Current working directory is /sgl-workspace/sglang
[05:26:57.691669] coredump: Writing out global memory (74544383104 bytes)