Here the Memory Output (Crash at 21:14:33)
±----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130 Driver Version: 384.130 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 980 Off | 00000000:01:00.0 On | N/A |
| 50% 79C P2 133W / 180W | 2911MiB / 4036MiB | 96% Default |
±------------------------------±---------------------±---------------------+
±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1068 G /usr/lib/xorg/Xorg 213MiB |
| 0 1745 G compiz 120MiB |
| 0 6114 C python2 74MiB |
| 0 8186 C /home//caffe/build/tools/caffe 2489MiB |
±----------------------------------------------------------------------------+
Sat Jun 23 21:14:22 2018
±----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130 Driver Version: 384.130 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 980 Off | 00000000:01:00.0 On | N/A |
| 50% 79C P2 133W / 180W | 2911MiB / 4036MiB | 96% Default |
±------------------------------±---------------------±---------------------+
±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1068 G /usr/lib/xorg/Xorg 213MiB |
| 0 1745 G compiz 120MiB |
| 0 6114 C python2 74MiB |
| 0 8186 C /home//caffe/build/tools/caffe 2489MiB |
±----------------------------------------------------------------------------+
Sat Jun 23 21:14:23 2018
±----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130 Driver Version: 384.130 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 980 Off | 00000000:01:00.0 On | N/A |
| 50% 79C P2 136W / 180W | 2911MiB / 4036MiB | 95% Default |
±------------------------------±---------------------±---------------------+
±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1068 G /usr/lib/xorg/Xorg 213MiB |
| 0 1745 G compiz 120MiB |
| 0 6114 C python2 74MiB |
| 0 8186 C /home//caffe/build/tools/caffe 2489MiB |
±----------------------------------------------------------------------------+
Sat Jun 23 21:14:25 2018
±----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130 Driver Version: 384.130 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 980 Off | 00000000:01:00.0 On | N/A |
| 50% 80C P2 138W / 180W | 2911MiB / 4036MiB | 97% Default |
±------------------------------±---------------------±---------------------+
±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1068 G /usr/lib/xorg/Xorg 213MiB |
| 0 1745 G compiz 120MiB |
| 0 6114 C python2 74MiB |
| 0 8186 C /home//caffe/build/tools/caffe 2489MiB |
±----------------------------------------------------------------------------+
Sat Jun 23 21:14:26 2018
±----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130 Driver Version: 384.130 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 980 Off | 00000000:01:00.0 On | N/A |
| 50% 80C P2 138W / 180W | 2911MiB / 4036MiB | 97% Default |
±------------------------------±---------------------±---------------------+
±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1068 G /usr/lib/xorg/Xorg 213MiB |
| 0 1745 G compiz 120MiB |
| 0 6114 C python2 74MiB |
| 0 8186 C /home//caffe/build/tools/caffe 2489MiB |
±----------------------------------------------------------------------------+
Sat Jun 23 21:14:27 2018
±----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130 Driver Version: 384.130 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 980 Off | 00000000:01:00.0 On | N/A |
| 50% 75C P2 69W / 180W | 2911MiB / 4036MiB | 11% Default |
±------------------------------±---------------------±---------------------+
±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1068 G /usr/lib/xorg/Xorg 213MiB |
| 0 1745 G compiz 120MiB |
| 0 6114 C python2 74MiB |
| 0 8186 C /home//caffe/build/tools/caffe 2489MiB |
±----------------------------------------------------------------------------+
Sat Jun 23 21:14:28 2018
±----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130 Driver Version: 384.130 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 980 Off | 00000000:01:00.0 On | N/A |
| 50% 75C P2 102W / 180W | 2911MiB / 4036MiB | 47% Default |
±------------------------------±---------------------±---------------------+
±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1068 G /usr/lib/xorg/Xorg 213MiB |
| 0 1745 G compiz 120MiB |
| 0 6114 C python2 74MiB |
| 0 8186 C /home//caffe/build/tools/caffe 2489MiB |
±----------------------------------------------------------------------------+
Sat Jun 23 21:14:29 2018
±----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130 Driver Version: 384.130 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 980 Off | 00000000:01:00.0 On | N/A |
| 50% 77C P2 109W / 180W | 2911MiB / 4036MiB | 41% Default |
±------------------------------±---------------------±---------------------+
±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1068 G /usr/lib/xorg/Xorg 213MiB |
| 0 1745 G compiz 120MiB |
| 0 6114 C python2 74MiB |
| 0 8186 C /home//caffe/build/tools/caffe 2489MiB |
±----------------------------------------------------------------------------+
Sat Jun 23 21:14:30 2018
±----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130 Driver Version: 384.130 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 980 Off | 00000000:01:00.0 On | N/A |
| 50% 76C P2 95W / 180W | 2911MiB / 4036MiB | 42% Default |
±------------------------------±---------------------±---------------------+
±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1068 G /usr/lib/xorg/Xorg 213MiB |
| 0 1745 G compiz 120MiB |
| 0 6114 C python2 74MiB |
| 0 8186 C /home//caffe/build/tools/caffe 2489MiB |
±----------------------------------------------------------------------------+
Sat Jun 23 21:14:31 2018
±----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130 Driver Version: 384.130 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 980 Off | 00000000:01:00.0 On | N/A |
| 50% 77C P2 101W / 180W | 2911MiB / 4036MiB | 44% Default |
±------------------------------±---------------------±---------------------+
±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1068 G /usr/lib/xorg/Xorg 213MiB |
| 0 1745 G compiz 120MiB |
| 0 6114 C python2 74MiB |
| 0 8186 C /home//caffe/build/tools/caffe 2489MiB |
±----------------------------------------------------------------------------+
Sat Jun 23 21:14:32 2018
±----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130 Driver Version: 384.130 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 980 Off | 00000000:01:00.0 On | N/A |
| 50% 73C P2 62W / 180W | 2911MiB / 4036MiB | 1% Default |
±------------------------------±---------------------±---------------------+
±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1068 G /usr/lib/xorg/Xorg 213MiB |
| 0 1745 G compiz 120MiB |
| 0 6114 C python2 74MiB |
| 0 8186 C /home//caffe/build/tools/caffe 2489MiB |
±----------------------------------------------------------------------------+
Sat Jun 23 21:14:33 2018
±----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130 Driver Version: 384.130 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 980 Off | 00000000:01:00.0 On | N/A |
| 50% 72C P2 60W / 180W | 422MiB / 4036MiB | 32% Default |
±------------------------------±---------------------±---------------------+
±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1068 G /usr/lib/xorg/Xorg 213MiB |
| 0 1745 G compiz 120MiB |
| 0 6114 C python2 74MiB |
±----------------------------------------------------------------------------+
Sat Jun 23 21:14:34 2018
±----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130 Driver Version: 384.130 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 980 Off | 00000000:01:00.0 On | N/A |
| 50% 71C P2 59W / 180W | 422MiB / 4036MiB | 0% Default |
±------------------------------±---------------------±---------------------+
±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1068 G /usr/lib/xorg/Xorg 213MiB |
| 0 1745 G compiz 120MiB |
| 0 6114 C python2 74MiB |
±----------------------------------------------------------------------------+
Sat Jun 23 21:14:35 2018
±----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130 Driver Version: 384.130 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 980 Off | 00000000:01:00.0 On | N/A |
| 49% 71C P2 59W / 180W | 422MiB / 4036MiB | 0% Default |
±------------------------------±---------------------±---------------------+
±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1068 G /usr/lib/xorg/Xorg 213MiB |
| 0 1745 G compiz 120MiB |
| 0 6114 C python2 74MiB |
±----------------------------------------------------------------------------+
Sat Jun 23 21:14:36 2018
±----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130 Driver Version: 384.130 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 980 Off | 00000000:01:00.0 On | N/A |
| 49% 69C P5 23W / 180W | 422MiB / 4036MiB | 1% Default |
±------------------------------±---------------------±---------------------+
±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1068 G /usr/lib/xorg/Xorg 213MiB |
| 0 1745 G compiz 120MiB |
| 0 6114 C python2 74MiB |
±----------------------------------------------------------------------------+
Output in DIGITS:
Train net output #0: loss_bbox = 0.678703 (* 2 = 1.35741 loss)
Train net output #1: loss_coverage = 0.851304 (* 1 = 0.851304 loss)
Iteration 1536, lr = 2.40195e-05
Snapshotting to binary proto file snapshot_iter_1544.caffemodel
Snapshotting solver state to binary proto file snapshot_iter_1544.solverstate
Iteration 1544, Testing net (#0)
Ignoring source layer train_data
Ignoring source layer train_label
Ignoring source layer train_transform
Error Message in Caffe Log (now a different one?!):
*** Aborted at 1529781271 (unix time) try “date -d @1529781271” if you are using GNU date ***
PC: @ 0x7f17c281f39a cv::MatExpr::type()
*** SIGSEGV (@0x7f1500010000) received by PID 8186 (TID 0x7f17e5527740) from PID 65536; stack trace: ***
@ 0x7f17e31354b0 (unknown)
@ 0x7f17c281f39a cv::MatExpr::type()
@ 0x7f17c268abc9 cv::_InputArray::type()
@ 0x7f17c26ca808 cv::Mat::copyTo()
@ 0x7f17bd0be2e6 (unknown)
@ 0x7f17bd153ea8 (unknown)
@ 0x7f17e3d60971 PyEval_EvalFrameEx
@ 0x7f17e3d5f044 PyEval_EvalFrameEx
@ 0x7f17e3d5f044 PyEval_EvalFrameEx
@ 0x7f17e3e9605c PyEval_EvalCodeEx
@ 0x7f17e3dec370 (unknown)
@ 0x7f17e3dbf273 PyObject_Call
@ 0x7f17e3e333ac (unknown)
@ 0x7f17e3dbf273 PyObject_Call
@ 0x7f17e3e95487 PyEval_CallObjectWithKeywords
@ 0x7f17e3df3fa7 PyEval_CallFunction
@ 0x7f1634e11065 caffe::PythonLayer<>::Forward_cpu()
@ 0x7f17e4ca6847 caffe::Net<>::ForwardFromTo()
@ 0x7f17e4ca6bb7 caffe::Net<>::Forward()
@ 0x7f17e4e4a072 caffe::Solver<>::Test()
@ 0x7f17e4e4acee caffe::Solver<>::TestAll()
@ 0x7f17e4e4dad9 caffe::Solver<>::Step()
@ 0x7f17e4e4e829 caffe::Solver<>::Solve()
@ 0x40c6f6 train()
@ 0x408708 main
@ 0x7f17e3120830 __libc_start_main
@ 0x408e79 _start
@ 0x0 (unknown)