I’m running a single machine with four A6000s and 1TB of RAM using Ubuntu 20.04.3 LTS. After training with Tensorflow if I try to restart the kernel under Jupyter Notebook, the line “import tensorflow as tf” hangs. I end up having to power the machine off completely to get it working again. This machine is running a M12SWA-TF-B motherboard. Has anyone seen anything like this? Any suggestions as to how to fix this? Thanks!
Here are the last few lines of the strace.
1643302791.282088 stat(“/usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi3/mca_ess_singleton.so”, {st_mode=S_IFREG|0644, st_size=27704, …}) = 0
1643302791.282122 openat(AT_FDCWD, “/usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi3/mca_ess_singleton.so”, O_RDONLY|O_CLOEXEC) = 7
1643302791.282146 read(7, “\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340’\0\0\0\0\0\0”…, 832) = 832
1643302791.282169 fstat(7, {st_mode=S_IFREG|0644, st_size=27704, …}) = 0
1643302791.282194 mmap(NULL, 29544, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 7, 0) = 0x7f3d5f678000
1643302791.282216 mmap(0x7f3d5f67a000, 12288, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 7, 0x2000) = 0x7f3d5f67a000
1643302791.282243 mmap(0x7f3d5f67d000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 7, 0x5000) = 0x7f3d5f67d000
1643302791.282267 mmap(0x7f3d5f67e000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 7, 0x5000) = 0x7f3d5f67e000
1643302791.282295 close(7) = 0
1643302791.282371 mprotect(0x7f3d5f67e000, 4096, PROT_READ) = 0
1643302791.282524 pipe([7, 8]) = 0
1643302791.282551 pipe([9, 10]) = 0
1643302791.282575 stat(“/usr/bin/orted”, {st_mode=S_IFREG|0755, st_size=14648, …}) = 0
1643302791.282611 clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f3dba183a10) = 1251309
1643302791.285529 close(8) = 0
1643302791.285554 close(9) = 0
1643302791.285581 read(7,