DetectNet Tutorial Problem - OpenCV 3?

Hi,

Sorry if this looks like a double post, but I am running into a DetectNet a new issue with DIGITS. (I did everything according to the tutorial - https://github.com/dusty-nv/jetson-inference)

I am training a model and after many hours this happened (just masked it with ):

OpenCV Error: The function/feature is not implemented (Unknown/unsupported array type) in type, file /home/om/test/opencv/opencv/modules/core/src/matrix.cpp, line 2034
OpenCV Error: The function/feature is not implemented (Unknown/unsupported array type) in type, file /home/om/test/opencv/opencv/modules/core/src/matrix.cpp, line 2034
Traceback (most recent call last):
File “/home//caffe/python/caffe/layers/detectnet/clustering.py”, line 133, in forward
bbox = cluster(self, data0, bottom[1].data)
File “/home//caffe/python/caffe/layers/detectnet/clustering.py”, line 227, in cluster
boxes_cur_image = vote_boxes(propose_boxes, propose_cvgs, mask, self)
File “/home//caffe/python/caffe/layers/detectnet/clustering.py”, line 193, in vote_boxes
self.gridbox_rect_eps)
cv2.error: /home/om/test/opencv/opencv/modules/core/src/matrix.cpp:2034: error: (-213) Unknown/unsupported array type in function type

I am using Jetpack 3.2, which is installing Open CV 3.

DIGITS version:
6.1.1
Caffe version:
0.15.14
Caffe flavor:
NVIDIA

Hi J0eBl4ck, this cv.GroupRectangles() function gets called every time, so that is interested that the error only occurs after a while.

Which dataset are you currently training your model on? Does the error keep happening, or is it perhaps related to a malformed/corrupted element of the training dataset?

I am using this dataset:

https://github.com/NVIDIA/DIGITS/tree/master/examples/object-detection

The DetectNet was almost finished (>80%) when the error occurred.

Hi,

Could you also monitor memory status and share information with us?
Thanks.

I am currently away, but I will give you all information as soon as I am back.

Is there any special setting/Operation I need to do for showing the memory status?

Hi,

You can monitor the memory status from calling nvidia-smi periodically.

#!/bin/bash
while [ 1 ]
do
   nvidia-smi
   sleep 1
done

Thanks.

I am back (sorry for the delay) and the problem is very strange.
My computer was only able to finish the object detection job once, but all my tests failed. (Objects couldn’t be detected at all --> so I think something went wrong)

Now I get the error again (sometimes after 9 minutes, sometimes after 3 hours!):

OpenCV Error: The function/feature is not implemented (Unknown/unsupported array type) in create, file /home/om/test/opencv/opencv/modules/core/src/matrix.cpp, line 2821
OpenCV Error: The function/feature is not implemented (Unknown/unsupported array type) in create, file /home/om/test/opencv/opencv/modules/core/src/matrix.cpp, line 2821
Traceback (most recent call last):
File “/home//caffe/python/caffe/layers/detectnet/clustering.py”, line 133, in forward
bbox = cluster(self, data0, bottom[1].data)
File “/home//caffe/python/caffe/layers/detectnet/clustering.py”, line 227, in cluster
boxes_cur_image = vote_boxes(propose_boxes, propose_cvgs, mask, self)
File “/home//caffe/python/caffe/layers/detectnet/clustering.py”, line 193, in vote_boxes
self.gridbox_rect_eps)
cv2.error: /home/om/test/opencv/opencv/modules/core/src/matrix.cpp:2821: error: (-213) Unknown/unsupported array type in function create

I will run another objectdetection training and give you the memory status asap!

Any alternative ObjectDetection tutorial available? I really want to make sure that the error occurs not because of the dataset.

Here the Memory Output (Crash at 21:14:33)

±----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130 Driver Version: 384.130 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 980 Off | 00000000:01:00.0 On | N/A |
| 50% 79C P2 133W / 180W | 2911MiB / 4036MiB | 96% Default |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1068 G /usr/lib/xorg/Xorg 213MiB |
| 0 1745 G compiz 120MiB |
| 0 6114 C python2 74MiB |
| 0 8186 C /home//caffe/build/tools/caffe 2489MiB |
±----------------------------------------------------------------------------+
Sat Jun 23 21:14:22 2018
±----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130 Driver Version: 384.130 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 980 Off | 00000000:01:00.0 On | N/A |
| 50% 79C P2 133W / 180W | 2911MiB / 4036MiB | 96% Default |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1068 G /usr/lib/xorg/Xorg 213MiB |
| 0 1745 G compiz 120MiB |
| 0 6114 C python2 74MiB |
| 0 8186 C /home//caffe/build/tools/caffe 2489MiB |
±----------------------------------------------------------------------------+
Sat Jun 23 21:14:23 2018
±----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130 Driver Version: 384.130 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 980 Off | 00000000:01:00.0 On | N/A |
| 50% 79C P2 136W / 180W | 2911MiB / 4036MiB | 95% Default |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1068 G /usr/lib/xorg/Xorg 213MiB |
| 0 1745 G compiz 120MiB |
| 0 6114 C python2 74MiB |
| 0 8186 C /home//caffe/build/tools/caffe 2489MiB |
±----------------------------------------------------------------------------+
Sat Jun 23 21:14:25 2018
±----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130 Driver Version: 384.130 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 980 Off | 00000000:01:00.0 On | N/A |
| 50% 80C P2 138W / 180W | 2911MiB / 4036MiB | 97% Default |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1068 G /usr/lib/xorg/Xorg 213MiB |
| 0 1745 G compiz 120MiB |
| 0 6114 C python2 74MiB |
| 0 8186 C /home//caffe/build/tools/caffe 2489MiB |
±----------------------------------------------------------------------------+
Sat Jun 23 21:14:26 2018
±----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130 Driver Version: 384.130 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 980 Off | 00000000:01:00.0 On | N/A |
| 50% 80C P2 138W / 180W | 2911MiB / 4036MiB | 97% Default |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1068 G /usr/lib/xorg/Xorg 213MiB |
| 0 1745 G compiz 120MiB |
| 0 6114 C python2 74MiB |
| 0 8186 C /home//caffe/build/tools/caffe 2489MiB |
±----------------------------------------------------------------------------+
Sat Jun 23 21:14:27 2018
±----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130 Driver Version: 384.130 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 980 Off | 00000000:01:00.0 On | N/A |
| 50% 75C P2 69W / 180W | 2911MiB / 4036MiB | 11% Default |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1068 G /usr/lib/xorg/Xorg 213MiB |
| 0 1745 G compiz 120MiB |
| 0 6114 C python2 74MiB |
| 0 8186 C /home//caffe/build/tools/caffe 2489MiB |
±----------------------------------------------------------------------------+
Sat Jun 23 21:14:28 2018
±----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130 Driver Version: 384.130 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 980 Off | 00000000:01:00.0 On | N/A |
| 50% 75C P2 102W / 180W | 2911MiB / 4036MiB | 47% Default |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1068 G /usr/lib/xorg/Xorg 213MiB |
| 0 1745 G compiz 120MiB |
| 0 6114 C python2 74MiB |
| 0 8186 C /home//caffe/build/tools/caffe 2489MiB |
±----------------------------------------------------------------------------+
Sat Jun 23 21:14:29 2018
±----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130 Driver Version: 384.130 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 980 Off | 00000000:01:00.0 On | N/A |
| 50% 77C P2 109W / 180W | 2911MiB / 4036MiB | 41% Default |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1068 G /usr/lib/xorg/Xorg 213MiB |
| 0 1745 G compiz 120MiB |
| 0 6114 C python2 74MiB |
| 0 8186 C /home//caffe/build/tools/caffe 2489MiB |
±----------------------------------------------------------------------------+
Sat Jun 23 21:14:30 2018
±----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130 Driver Version: 384.130 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 980 Off | 00000000:01:00.0 On | N/A |
| 50% 76C P2 95W / 180W | 2911MiB / 4036MiB | 42% Default |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1068 G /usr/lib/xorg/Xorg 213MiB |
| 0 1745 G compiz 120MiB |
| 0 6114 C python2 74MiB |
| 0 8186 C /home//caffe/build/tools/caffe 2489MiB |
±----------------------------------------------------------------------------+
Sat Jun 23 21:14:31 2018
±----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130 Driver Version: 384.130 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 980 Off | 00000000:01:00.0 On | N/A |
| 50% 77C P2 101W / 180W | 2911MiB / 4036MiB | 44% Default |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1068 G /usr/lib/xorg/Xorg 213MiB |
| 0 1745 G compiz 120MiB |
| 0 6114 C python2 74MiB |
| 0 8186 C /home//caffe/build/tools/caffe 2489MiB |
±----------------------------------------------------------------------------+
Sat Jun 23 21:14:32 2018
±----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130 Driver Version: 384.130 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 980 Off | 00000000:01:00.0 On | N/A |
| 50% 73C P2 62W / 180W | 2911MiB / 4036MiB | 1% Default |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1068 G /usr/lib/xorg/Xorg 213MiB |
| 0 1745 G compiz 120MiB |
| 0 6114 C python2 74MiB |
| 0 8186 C /home//caffe/build/tools/caffe 2489MiB |
±----------------------------------------------------------------------------+
Sat Jun 23 21:14:33 2018
±----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130 Driver Version: 384.130 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 980 Off | 00000000:01:00.0 On | N/A |
| 50% 72C P2 60W / 180W | 422MiB / 4036MiB | 32% Default |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1068 G /usr/lib/xorg/Xorg 213MiB |
| 0 1745 G compiz 120MiB |
| 0 6114 C python2 74MiB |
±----------------------------------------------------------------------------+
Sat Jun 23 21:14:34 2018
±----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130 Driver Version: 384.130 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 980 Off | 00000000:01:00.0 On | N/A |
| 50% 71C P2 59W / 180W | 422MiB / 4036MiB | 0% Default |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1068 G /usr/lib/xorg/Xorg 213MiB |
| 0 1745 G compiz 120MiB |
| 0 6114 C python2 74MiB |
±----------------------------------------------------------------------------+
Sat Jun 23 21:14:35 2018
±----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130 Driver Version: 384.130 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 980 Off | 00000000:01:00.0 On | N/A |
| 49% 71C P2 59W / 180W | 422MiB / 4036MiB | 0% Default |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1068 G /usr/lib/xorg/Xorg 213MiB |
| 0 1745 G compiz 120MiB |
| 0 6114 C python2 74MiB |
±----------------------------------------------------------------------------+
Sat Jun 23 21:14:36 2018
±----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130 Driver Version: 384.130 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 980 Off | 00000000:01:00.0 On | N/A |
| 49% 69C P5 23W / 180W | 422MiB / 4036MiB | 1% Default |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1068 G /usr/lib/xorg/Xorg 213MiB |
| 0 1745 G compiz 120MiB |
| 0 6114 C python2 74MiB |
±----------------------------------------------------------------------------+

Output in DIGITS:

Train net output #0: loss_bbox = 0.678703 (* 2 = 1.35741 loss)
Train net output #1: loss_coverage = 0.851304 (* 1 = 0.851304 loss)
Iteration 1536, lr = 2.40195e-05
Snapshotting to binary proto file snapshot_iter_1544.caffemodel
Snapshotting solver state to binary proto file snapshot_iter_1544.solverstate
Iteration 1544, Testing net (#0)
Ignoring source layer train_data
Ignoring source layer train_label
Ignoring source layer train_transform

Error Message in Caffe Log (now a different one?!):

*** Aborted at 1529781271 (unix time) try “date -d @1529781271” if you are using GNU date ***
PC: @ 0x7f17c281f39a cv::MatExpr::type()
*** SIGSEGV (@0x7f1500010000) received by PID 8186 (TID 0x7f17e5527740) from PID 65536; stack trace: ***
@ 0x7f17e31354b0 (unknown)
@ 0x7f17c281f39a cv::MatExpr::type()
@ 0x7f17c268abc9 cv::_InputArray::type()
@ 0x7f17c26ca808 cv::Mat::copyTo()
@ 0x7f17bd0be2e6 (unknown)
@ 0x7f17bd153ea8 (unknown)
@ 0x7f17e3d60971 PyEval_EvalFrameEx
@ 0x7f17e3d5f044 PyEval_EvalFrameEx
@ 0x7f17e3d5f044 PyEval_EvalFrameEx
@ 0x7f17e3e9605c PyEval_EvalCodeEx
@ 0x7f17e3dec370 (unknown)
@ 0x7f17e3dbf273 PyObject_Call
@ 0x7f17e3e333ac (unknown)
@ 0x7f17e3dbf273 PyObject_Call
@ 0x7f17e3e95487 PyEval_CallObjectWithKeywords
@ 0x7f17e3df3fa7 PyEval_CallFunction
@ 0x7f1634e11065 caffe::PythonLayer<>::Forward_cpu()
@ 0x7f17e4ca6847 caffe::Net<>::ForwardFromTo()
@ 0x7f17e4ca6bb7 caffe::Net<>::Forward()
@ 0x7f17e4e4a072 caffe::Solver<>::Test()
@ 0x7f17e4e4acee caffe::Solver<>::TestAll()
@ 0x7f17e4e4dad9 caffe::Solver<>::Step()
@ 0x7f17e4e4e829 caffe::Solver<>::Solve()
@ 0x40c6f6 train()
@ 0x408708 main
@ 0x7f17e3120830 __libc_start_main
@ 0x408e79 _start
@ 0x0 (unknown)

I did today another test (dataset from the object detection KITTI tutorial). Same error:

OpenCV Error: The function/feature is not implemented (Unknown/unsupported array type) in type, file /home/om/test/opencv/opencv/modules/core/src/matrix.cpp, line 2034
OpenCV Error: The function/feature is not implemented (Unknown/unsupported array type) in type, file /home/om/test/opencv/opencv/modules/core/src/matrix.cpp, line 2034
Traceback (most recent call last):
File “/home//caffe/python/caffe/layers/detectnet/clustering.py”, line 133, in forward
bbox = cluster(self, data0, bottom[1].data)
File “/home//caffe/python/caffe/layers/detectnet/clustering.py”, line 227, in cluster
boxes_cur_image = vote_boxes(propose_boxes, propose_cvgs, mask, self)
File “/home//caffe/python/caffe/layers/detectnet/clustering.py”, line 193, in vote_boxes
self.gridbox_rect_eps)
cv2.error: /home/om/test/opencv/opencv/modules/core/src/matrix.cpp:2034: error: (-213) Unknown/unsupported array type in function type

I noticed that my label shapes are different than in the tutorial:

I have (1, 25, 16) for train_db and (1, 19, 16) for val_db.

Is this the cause?

Hi,

Guess that issue is caused by invalid data or label. Ex, out of boundary on certain setting.
Could you share why you use the different label dimension between training and validation set?

Thanks

I am only using the data, which is provided on this website:

http://www.cvlibs.net/datasets/kitti/eval_object.php

  • Left color images of object data set data_object_image_2.zip
  • Training labels of object data set data_object_label_2.zip
  • Object development kit devkit_object.zip
  • I followed the Object-Detection tutorial -> https://github.com/NVIDIA/DIGITS/tree/master/examples/object-detection

    All I am doing is loading the labels etc. into DIGITS. No clue why they differ so much from the tutorial. Last time the machine crashed after the second epoch.

    Maybe the dataset changed? Is there a possibility to fix this or set label dimensions manually? Or can you provide anohter object detection tutorial? (https://github.com/dusty-nv/jetson-inference#locating-object-coordinates-using-detectnet also does not work on my machine)

    Thanks for your feedback.
    We will verify this tutorial again and update information with you.

    Hi,

    Could you apply this command and try it again?

    sudo pip install opencv-python
    

    In case the OpenCV library is messed up in your environment.
    Thanks.

    Thank you! Finally it is working (Job is finished without errors)
    I don’t know why but I had to adjust the python reference manually.

    What I did:

    1.
    As you suggested -> sudo pip install opencv-python
    2. OpenCV-Python Version 3 was installed

    3. Next I checked if Python is really using Version 3.
    In the terminal:

    python
    import cv2
    cv2.__version__
    Output was -> '2.X.Y'
    

    -> Python was still using the wrong version!

    4. So I executed this in the terminal:

    [i]sudo apt-get remove python-opencv
    [/i]
    

    5. Checking again:

    python
    import cv2
    cv2.__version__
    Output was -> '3.4.1'
    

    Python is now using the correct lib!

    As mentioned before DIGITS is now finishing the ObjectDetection job :)

    Just in addition:

    The training curves look a bit odd, but I think this is because of the model.

    Hi,

    Good to know this. : )
    Also, thanks to updating this information with us.