Problem with OpenCV Isaac SDK

Hello,

I have some problems trying to use OpenCV in my custom codelet. I wrote a simple codelet to save images with OpenCV. However, when I tried running the application, the following error happens:

2020-06-25 00:33:44.880 INFO  engine/gems/scheduler/scheduler.cpp@290: Scheduler execution groups are:
2020-06-25 00:33:44.880 INFO  engine/gems/scheduler/scheduler.cpp@299: __BlockerGroup__: Cores = [9, 10, 11], Workers = No
2020-06-25 00:33:44.880 INFO  engine/gems/scheduler/scheduler.cpp@299: __WorkerGroup__: Cores = [0, 1, 2, 3, 4, 5, 6, 7, 8], Workers = Yes
free(): invalid pointer
====================================================================================================
|                            Isaac application terminated unexpectedly                             |
====================================================================================================
#01 /home/arc/.cache/bazel/_bazel_arc/3ea1431de7d06df19bbc8ecc65d8fa51/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/packages/freespace_dnn/apps/freespace_dnn_training.runfiles/com_nvidia_isaac/engine/pyalice/bindings.so(+0x1d08fa) [0x7f6957a658fa]
#02 google_breakpad::ExceptionHandler::GenerateDump(google_breakpad::ExceptionHandler::CrashContext*) /home/arc/.cache/bazel/_bazel_arc/3ea1431de7d06df19bbc8ecc65d8fa51/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/packages/freespace_dnn/apps/freespace_dnn_training.runfiles/com_nvidia_isaac/engine/pyalice/bindings.so(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3f0) [0x7f6957b55d20]
#03 google_breakpad::ExceptionHandler::SignalHandler(int, siginfo_t*, void*) /home/arc/.cache/bazel/_bazel_arc/3ea1431de7d06df19bbc8ecc65d8fa51/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/packages/freespace_dnn/apps/freespace_dnn_training.runfiles/com_nvidia_isaac/engine/pyalice/bindings.so(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xc0) [0x7f6957b56090]
#04 /lib/x86_64-linux-gnu/libc.so.6(+0x3ef20) [0x7f69596eaf20]
#05 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7) [0x7f69596eae97]
#06 /lib/x86_64-linux-gnu/libc.so.6(abort+0x141) [0x7f69596ec801]
#07 /lib/x86_64-linux-gnu/libc.so.6(+0x89897) [0x7f6959735897]
#08 /lib/x86_64-linux-gnu/libc.so.6(+0x9090a) [0x7f695973c90a]
#09 /lib/x86_64-linux-gnu/libc.so.6(cfree+0x4cc) [0x7f6959743e1c]
#10 /home/arc/.cache/bazel/_bazel_arc/3ea1431de7d06df19bbc8ecc65d8fa51/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/packages/freespace_dnn/apps/freespace_dnn_training.runfiles/com_nvidia_isaac//packages/image_extractor/../../_solib_k8/_U@opencv_Ux86_U64_S_S_Copencv_Ux86_U64___Uexternal_Sopencv_Ux86_U64_Slib/libopencv_dnn.so.3.3(+0x22e49e) [0x7f68dc72e49e]
#11 /home/arc/.cache/bazel/_bazel_arc/3ea1431de7d06df19bbc8ecc65d8fa51/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/packages/freespace_dnn/apps/freespace_dnn_training.runfiles/com_nvidia_isaac//packages/image_extractor/../../_solib_k8/_U@opencv_Ux86_U64_S_S_Copencv_Ux86_U64___Uexternal_Sopencv_Ux86_U64_Slib/libopencv_dnn.so.3.3(+0x237ee6) [0x7f68dc737ee6]
#12 /home/arc/.cache/bazel/_bazel_arc/3ea1431de7d06df19bbc8ecc65d8fa51/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/packages/freespace_dnn/apps/freespace_dnn_training.runfiles/com_nvidia_isaac//packages/image_extractor/../../_solib_k8/_U@opencv_Ux86_U64_S_S_Copencv_Ux86_U64___Uexternal_Sopencv_Ux86_U64_Slib/libopencv_dnn.so.3.3(+0x2392b7) [0x7f68dc7392b7]
#13 /home/arc/.cache/bazel/_bazel_arc/3ea1431de7d06df19bbc8ecc65d8fa51/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/packages/freespace_dnn/apps/freespace_dnn_training.runfiles/com_nvidia_isaac//packages/image_extractor/../../_solib_k8/_U@opencv_Ux86_U64_S_S_Copencv_Ux86_U64___Uexternal_Sopencv_Ux86_U64_Slib/libopencv_dnn.so.3.3(+0x1e43e6) [0x7f68dc6e43e6]
#14 /home/arc/.cache/bazel/_bazel_arc/3ea1431de7d06df19bbc8ecc65d8fa51/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/packages/freespace_dnn/apps/freespace_dnn_training.runfiles/com_nvidia_isaac//packages/image_extractor/../../_solib_k8/_U@opencv_Ux86_U64_S_S_Copencv_Ux86_U64___Uexternal_Sopencv_Ux86_U64_Slib/libopencv_dnn.so.3.3(+0x245a06) [0x7f68dc745a06]
#15 /home/arc/.cache/bazel/_bazel_arc/3ea1431de7d06df19bbc8ecc65d8fa51/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/packages/freespace_dnn/apps/freespace_dnn_training.runfiles/com_nvidia_isaac//packages/image_extractor/../../_solib_k8/_U@opencv_Ux86_U64_S_S_Copencv_Ux86_U64___Uexternal_Sopencv_Ux86_U64_Slib/libopencv_dnn.so.3.3(+0x1f3058) [0x7f68dc6f3058]
#16 /home/arc/.cache/bazel/_bazel_arc/3ea1431de7d06df19bbc8ecc65d8fa51/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/packages/freespace_dnn/apps/freespace_dnn_training.runfiles/com_nvidia_isaac//packages/image_extractor/../../_solib_k8/_U@opencv_Ux86_U64_S_S_Copencv_Ux86_U64___Uexternal_Sopencv_Ux86_U64_Slib/libopencv_dnn.so.3.3(+0x479ee) [0x7f68dc5479ee]
#17 /home/arc/.cache/bazel/_bazel_arc/3ea1431de7d06df19bbc8ecc65d8fa51/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/packages/freespace_dnn/apps/freespace_dnn_training.runfiles/com_nvidia_isaac//packages/image_extractor/../../_solib_k8/_U@opencv_Ux86_U64_S_S_Copencv_Ux86_U64___Uexternal_Sopencv_Ux86_U64_Slib/libopencv_dnn.so.3.3(+0x1ea88d) [0x7f68dc6ea88d]
#18 /home/arc/.cache/bazel/_bazel_arc/3ea1431de7d06df19bbc8ecc65d8fa51/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/packages/freespace_dnn/apps/freespace_dnn_training.runfiles/com_nvidia_isaac//packages/image_extractor/../../_solib_k8/_U@opencv_Ux86_U64_S_S_Copencv_Ux86_U64___Uexternal_Sopencv_Ux86_U64_Slib/libopencv_dnn.so.3.3(+0x47893) [0x7f68dc547893]
#19 /lib64/ld-linux-x86-64.so.2(+0x10733) [0x7f6959aad733]
#20 /lib64/ld-linux-x86-64.so.2(+0x151ff) [0x7f6959ab21ff]
#21 /lib/x86_64-linux-gnu/libc.so.6(_dl_catch_exception+0x6f) [0x7f69598132df]
#22 /lib64/ld-linux-x86-64.so.2(+0x147ca) [0x7f6959ab17ca]
#23 /lib/x86_64-linux-gnu/libdl.so.2(+0xf96) [0x7f6959289f96]
#24 /lib/x86_64-linux-gnu/libc.so.6(_dl_catch_exception+0x6f) [0x7f69598132df]
#25 /lib/x86_64-linux-gnu/libc.so.6(_dl_catch_error+0x2f) [0x7f695981336f]
#26 /lib/x86_64-linux-gnu/libdl.so.2(+0x1735) [0x7f695928a735]
#27 /lib/x86_64-linux-gnu/libdl.so.2(dlopen+0x71) [0x7f695928a051]
#28 /home/arc/.cache/bazel/_bazel_arc/3ea1431de7d06df19bbc8ecc65d8fa51/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/packages/freespace_dnn/apps/freespace_dnn_training.runfiles/com_nvidia_isaac/engine/pyalice/bindings.so(+0x1d41e4) [0x7f6957a691e4]
#29 isaac::alice::ModuleManager::Module::open(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /home/arc/.cache/bazel/_bazel_arc/3ea1431de7d06df19bbc8ecc65d8fa51/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/packages/freespace_dnn/apps/freespace_dnn_training.runfiles/com_nvidia_isaac/engine/pyalice/bindings.so(_ZN5isaac5alice13ModuleManager6Module4openERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS9_EERKS9_SF_+0x7df) [0x7f6957a6beff]
#30 isaac::alice::ModuleManager::loadModule(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /home/arc/.cache/bazel/_bazel_arc/3ea1431de7d06df19bbc8ecc65d8fa51/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/packages/freespace_dnn/apps/freespace_dnn_training.runfiles/com_nvidia_isaac/engine/pyalice/bindings.so(_ZN5isaac5alice13ModuleManager10loadModuleERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0xfb) [0x7f6957a6ecbb]
#31 isaac::alice::ModuleManager::loadModules(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) /home/arc/.cache/bazel/_bazel_arc/3ea1431de7d06df19bbc8ecc65d8fa51/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/packages/freespace_dnn/apps/freespace_dnn_training.runfiles/com_nvidia_isaac/engine/pyalice/bindings.so(_ZN5isaac5alice13ModuleManager11loadModulesERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS8_EE+0x18e) [0x7f6957a6f3fe]
#32 isaac::alice::Application::createMore(isaac::alice::ApplicationJsonLoader const&) /home/arc/.cache/bazel/_bazel_arc/3ea1431de7d06df19bbc8ecc65d8fa51/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/packages/freespace_dnn/apps/freespace_dnn_training.runfiles/com_nvidia_isaac/engine/pyalice/bindings.so(_ZN5isaac5alice11Application10createMoreERKNS0_21ApplicationJsonLoaderE+0x6c) [0x7f6957a260cc]
#33 isaac::alice::Application::createApplication(isaac::alice::ApplicationJsonLoader const&) /home/arc/.cache/bazel/_bazel_arc/3ea1431de7d06df19bbc8ecc65d8fa51/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/packages/freespace_dnn/apps/freespace_dnn_training.runfiles/com_nvidia_isaac/engine/pyalice/bindings.so(_ZN5isaac5alice11Application17createApplicationERKNS0_21ApplicationJsonLoaderE+0x339) [0x7f6957a27799]
#34 isaac::alice::Application::Application(isaac::alice::ApplicationJsonLoader const&) /home/arc/.cache/bazel/_bazel_arc/3ea1431de7d06df19bbc8ecc65d8fa51/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/packages/freespace_dnn/apps/freespace_dnn_training.runfiles/com_nvidia_isaac/engine/pyalice/bindings.so(_ZN5isaac5alice11ApplicationC1ERKNS0_21ApplicationJsonLoaderE+0xff) [0x7f6957a27b3f]
#35 isaac::alice::PybindApplication::PybindApplication(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /home/arc/.cache/bazel/_bazel_arc/3ea1431de7d06df19bbc8ecc65d8fa51/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/packages/freespace_dnn/apps/freespace_dnn_training.runfiles/com_nvidia_isaac/engine/pyalice/bindings.so(_ZN5isaac5alice17PybindApplicationC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES9_+0x3f4) [0x7f69579ef784]
#36 /home/arc/.cache/bazel/_bazel_arc/3ea1431de7d06df19bbc8ecc65d8fa51/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/packages/freespace_dnn/apps/freespace_dnn_training.runfiles/com_nvidia_isaac/engine/pyalice/bindings.so(+0x15ace8) [0x7f69579efce8]
#37 /home/arc/.cache/bazel/_bazel_arc/3ea1431de7d06df19bbc8ecc65d8fa51/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/packages/freespace_dnn/apps/freespace_dnn_training.runfiles/com_nvidia_isaac/engine/pyalice/bindings.so(+0x14fd4e) [0x7f69579e4d4e]
#38 /usr/bin/python3(_PyCFunction_FastCallDict+0x35c) [0x5669ac]
#39 /usr/bin/python3() [0x5949a1]
#40 /usr/bin/python3() [0x54a005]
#41 /usr/bin/python3() [0x5513d1]
#42 /usr/bin/python3(_PyObject_FastCallKeywords+0x19c) [0x5a9cbc]
#43 /usr/bin/python3() [0x50a5c3]
#44 /usr/bin/python3(_PyEval_EvalFrameDefault+0x444) [0x50bfb4]
#45 /usr/bin/python3() [0x507d64]
#46 /usr/bin/python3(_PyFunction_FastCallDict+0x2e2) [0x509042]
#47 /usr/bin/python3() [0x594931]
#48 /usr/bin/python3() [0x549e5f]
#49 /usr/bin/python3() [0x5513d1]
#50 /usr/bin/python3(_PyObject_FastCallKeywords+0x19c) [0x5a9cbc]
#51 /usr/bin/python3() [0x50a5c3]
#52 /usr/bin/python3(_PyEval_EvalFrameDefault+0x1226) [0x50cd96]
#53 /usr/bin/python3() [0x507d64]
#54 /usr/bin/python3() [0x509a90]
#55 /usr/bin/python3() [0x50a48d]
#56 /usr/bin/python3(_PyEval_EvalFrameDefault+0x444) [0x50bfb4]
#57 /usr/bin/python3() [0x507d64]
#58 /usr/bin/python3(PyEval_EvalCode+0x23) [0x50ae13]
#59 /usr/bin/python3() [0x634c82]
#60 /usr/bin/python3(PyRun_FileExFlags+0x97) [0x634d37]
#61 /usr/bin/python3(PyRun_SimpleFileExFlags+0x17f) [0x6384ef]
#62 /usr/bin/python3(Py_Main+0x591) [0x639091]
#63 /usr/bin/python3(main+0xe0) [0x4b0d00]
#64 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f69596cdb97]
#65 /usr/bin/python3(_start+0x2a) [0x5b250a]
====================================================================================================
Minidump written to: /tmp/9ecd18c0-a1d8-4e8c-11deca96-97d7ee63.dmp
Segmentation fault (core dumped)`

The corresponding codelet’s BUILD file is:

load("//engine/build:isaac.bzl", "isaac_app", "isaac_cc_module", "isaac_subgraph")
isaac_cc_module(
    name="Image_Extractor_module",
    srcs=["image_extractor.cpp"],
    hdrs=["image_extractor.hpp"],
    visibility = ["//visibility:public"],
    deps=[
        "//third_party:opencv",
        "//engine/core/image",
        "//engine/core/math",
        "//engine/core/tensor",
        "//engine/gems/sight",
    ]
)

and I added the freespace_dnn_training app’s BUILD

py_binary(
    name = "freespace_dnn_training",
    srcs = [
        "freespace_dnn_training.py",
    ],
    data = [
        "__init__.py",
        "freespace_dnn_training.config.json",
        "freespace_dnn_training_freeze_model.py",
        "freespace_dnn_training_models.py",
        "freespace_dnn_training_unity3d.app.json",
        "freespace_dnn_training_unity3d_building_k.config.json",
        "freespace_dnn_training_unity3d_medium_warehouse.config.json",
        "freespace_dnn_training_unity3d_rng_warehouse.config.json",
        "freespace_dnn_training_unity3d_sidewalk.config.json",
        ":freespace_dnn_training_subgraph",
        "//packages:py_init",
        "//packages/freespace_dnn:py_init",
        "//packages/map:libmap_module.so",
        "//packages/ml:libml_module.so",
        "//packages/navsim/apps:navsim_subgraph",
        "//packages/viewers:libviewers_module.so",
       "//packages/image_extractor:libImage_Extractor_module_module.so" <==== ADDED
    ],
    visibility = ["//visibility:public"],
    deps = [
        "//engine/pyalice",
        "//packages/ml:pyml",
    ]
)

System Configuration:

  • Ubuntu 18.04
  • Dell G7
  • Isaac SDK 2019.3

I don’t think there is a problem with the opencv because I could run the EdgeDetector tutorial application. If I comment out the OpenCV dependency in the codelet module BUILD file and only log a message, it works. However, same error happens when I added the opencv in the codelet BUILD deps. I’m not sure whether I added the opencv deps wrongly.

Any help will be greatly appreciated.

Thank you.

Hi, @31431 ,

Reading the crashing stack, I guess maybe it is due to the libtbbmalloc.so which is used by OpenCV. When memory allocated by libtbbmalloc.so is returned and de-allocated by regular glibc, we have such issue.

Maybe 2020.1 would make it.