Loading Yolov5 model into DeepStream

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.
• Hardware Platform (Jetson / GPU) Jetson Xavier AGX
• DeepStream Version 6.0.1
• JetPack Version (valid for Jetson only) 4.6
• TensorRT Version 8.2.1

I am trying to change the pgie from the tracking sample gpubootcamp/Introduction_to_Multi-DNN_pipeline.ipynb at a647a2c3fc75828cbbf1cbd5ab29f865c491a35c · openhackathons-org/gpubootcamp · GitHub . It uses a resnet10 Caffe model and I want to change it to a customized Yolov5 model (from Ultralitics GitHub - ultralytics/yolov5: YOLOv5 🚀 in PyTorch > ONNX > CoreML > TFLite).

There are so many issues, that I became confused:

  1. My custom Yolov5 model was trained outside Nvidia environment, at a Intel CPU server. Is it an issue, or a model created at a HPC server can be use in Nvidia Jetson?

  2. The produced model is a Pytorch file .pt. There is an exporting utility that converts to .onnx and .pb. Which format is recommended?

  3. Once the file model is choosen, are there a config.txt file to be used as a sample of what must be changed on DeepStream?

Thanks in advance,

Flávio Mello

3

Deepstream supportes model trained in other framework. Deepstream supports model from caffe and onnx model file, you can refer to the nvinfer config file description: Gst-nvinfer — DeepStream 6.2 Release documentation (nvidia.com).
The config file should be adapted, you can refer to the config file of the samples. You can also find a config example in this example yolov5_gpu_optimization/deepstream-sample at main · NVIDIA-AI-IOT/yolov5_gpu_optimization · GitHub.

4

So, there are 3 features at the config yolov5_gpu_optimization/deepstream-sample at main · NVIDIA-AI-IOT/yolov5_gpu_optimization · GitHub that I need assistance.

  1. Can’t find anywhere that shared library from:
    custom-lib-path=./yolov5_decode.so

  2. The example links to:
    parse-bbox-func-name=NvDsInferParseYolo
    I see this method is compiled with the C++ source code, but I am in Pyhton. Is there any alternative for Python?

  3. How to produce a .engine file from:
    model-engine-file=./yolov5n.onnx_b1_gpu1_fp16.engine

5
  1. you need to compile the plugin and deepstream parser, please refer to build
  2. as you know, deepstream is a c lib, ptyhon is using python binding to develop deepstream sdk. this NvDsInferParseYolo is used to parse the inference outputs, there is no python version.
  3. at the first run, the application will generate an engine, you only need to set the correct configuration, especially the onnx-file. after the first run, the application will load the engine directly.
6

Understood itens 2 and 3.

I am stucked on item 1, that is, on pip3 install -r requirement_export.txt. I tries to install onnx-graphsurgeon, but it fails with an error associated to ‘transform’ not being a member of ‘std’ from c++ compiler. How to correctly install those requirements?

ubuntu@ubuntu:~/edge/yolo_install/yolov5$ sudo pip3 install onnx-graphsurgeon
[sudo] password for ubuntu: 
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting onnx-graphsurgeon
  Downloading https://developer.download.nvidia.com/compute/redist/onnx-graphsurgeon/onnx_graphsurgeon-0.3.27-py2.py3-none-any.whl (42 kB)
     |████████████████████████████████| 42 kB 6.8 MB/s            
Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from onnx-graphsurgeon) (1.19.5)
Collecting onnx
  Downloading onnx-1.14.0.tar.gz (11.3 MB)
     |████████████████████████████████| 11.3 MB 3.9 MB/s            
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Collecting protobuf>=3.20.2
  Downloading protobuf-4.21.0-py3-none-any.whl (291 kB)
     |████████████████████████████████| 291 kB 7.9 MB/s            
Requirement already satisfied: typing-extensions>=3.6.2.1 in /home/ubuntu/.local/lib/python3.6/site-packages (from onnx->onnx-graphsurgeon) (4.1.1)
WARNING: The candidate selected for download or install is a yanked version: 'protobuf' candidate (version 4.21.0 at https://files.pythonhosted.org/packages/27/82/986065ef305c0989c99d8ef3f29e58a03fac6e64bb2c36ffe64500cc6955/protobuf-4.21.0-py3-none-any.whl#sha256=4e78116673ba04e01e563f6a9cca2c72db0be8a3e1629094816357e81cc39d36 (from https://pypi.org/simple/protobuf/))
Reason for being yanked: Required python version not configured correctly (https://github.com/protocolbuffers/protobuf/issues/10076)
Building wheels for collected packages: onnx
  Building wheel for onnx (pyproject.toml) ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 /home/ubuntu/.local/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmp41d6h93y
       cwd: /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687
  Complete output (206 lines):
  fatal: not a git repository (or any of the parent directories): .git
  running bdist_wheel
  running build
  running build_py
  running create_version
  running cmake_build
  Using cmake args: ['/usr/local/bin/cmake', '-DPYTHON_INCLUDE_DIR=/usr/include/python3.6m', '-DPYTHON_EXECUTABLE=/usr/bin/python3', '-DBUILD_ONNX_PYTHON=ON', '-DCMAKE_EXPORT_COMPILE_COMMANDS=ON', '-DONNX_NAMESPACE=onnx', '-DPY_EXT_SUFFIX=.cpython-36m-aarch64-linux-gnu.so', '-DCMAKE_BUILD_TYPE=Release', '-DONNX_ML=1', '/tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687']
  -- The C compiler identification is GNU 7.5.0
  -- The CXX compiler identification is GNU 7.5.0
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Check for working C compiler: /usr/bin/cc - skipped
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  -- Check for working CXX compiler: /usr/bin/c++ - skipped
  -- Detecting CXX compile features
  -- Detecting CXX compile features - done
  -- Found PythonInterp: /usr/bin/python3 (found version "3.6.9")
  -- Found PythonLibs: /usr/lib/aarch64-linux-gnu/libpython3.6m.so (found version "3.6.9")
  -- Found Protobuf: /usr/lib/aarch64-linux-gnu/libprotobuf.a;-lpthread (found version "3.0.0")
  Generated: /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/.setuptools-cmake-build/onnx/onnx-ml.proto
  Generated: /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/.setuptools-cmake-build/onnx/onnx-operators-ml.proto
  Generated: /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/.setuptools-cmake-build/onnx/onnx-data.proto
  -- Could NOT find pybind11 (missing: pybind11_DIR)
  -- pybind11 v2.10.3
  -- Found PythonInterp: /usr/bin/python3 (found suitable version "3.6.9", minimum required is "3.6")
  -- Found PythonLibs: /usr/lib/aarch64-linux-gnu/libpython3.6m.so
  -- Performing Test HAS_FLTO
  -- Performing Test HAS_FLTO - Success
  --
  -- ******** Summary ********
  --   CMake version             : 3.26.4
  --   CMake command             : /usr/local/lib/python3.6/dist-packages/cmake/data/bin/cmake
  --   System                    : Linux
  --   C++ compiler              : /usr/bin/c++
  --   C++ compiler version      : 7.5.0
  --   CXX flags                 :  -Wnon-virtual-dtor
  --   Build type                : Release
  --   Compile definitions       : __STDC_FORMAT_MACROS
  --   CMAKE_PREFIX_PATH         :
  --   CMAKE_INSTALL_PREFIX      : /usr/local
  --   CMAKE_MODULE_PATH         :
  --
  --   ONNX version              : 1.14.0
  --   ONNX NAMESPACE            : onnx
  --   ONNX_USE_LITE_PROTO       : OFF
  --   USE_PROTOBUF_SHARED_LIBS  : OFF
  --   Protobuf_USE_STATIC_LIBS  : ON
  --   ONNX_DISABLE_EXCEPTIONS   : OFF
  --   ONNX_WERROR               : OFF
  --   ONNX_BUILD_TESTS          : OFF
  --   ONNX_BUILD_BENCHMARKS     : OFF
  --
  --   Protobuf compiler         : /usr/bin/protoc
  --   Protobuf includes         : /usr/include
  --   Protobuf libraries        : /usr/lib/aarch64-linux-gnu/libprotobuf.a;-lpthread
  --   BUILD_ONNX_PYTHON         : ON
  --     Python version        :
  --     Python executable     : /usr/bin/python3
  --     Python includes       : /usr/include/python3.6m
  -- Configuring done (1.9s)
  -- Generating done (0.0s)
  -- Build files have been written to: /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/.setuptools-cmake-build
  [  1%] Running gen_proto.py on onnx/onnx.in.proto
  Processing /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/onnx/onnx.in.proto
  Writing /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/.setuptools-cmake-build/onnx/onnx-ml.proto
  Writing /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/.setuptools-cmake-build/onnx/onnx-ml.proto3
  generating /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/.setuptools-cmake-build/onnx/onnx_pb.py
  [  2%] Running C++ protocol buffer compiler on /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/.setuptools-cmake-build/onnx/onnx-ml.proto
  Failed to generate mypy stubs: No module named 'google'
  [  2%] Built target gen_onnx_proto
  [  4%] Running gen_proto.py on onnx/onnx-data.in.proto
  [  5%] Running gen_proto.py on onnx/onnx-operators.in.proto
  Processing /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/onnx/onnx-operators.in.proto
  Writing /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/.setuptools-cmake-build/onnx/onnx-operators-ml.proto
  Writing /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/.setuptools-cmake-build/onnx/onnx-operators-ml.proto3
  generating /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/.setuptools-cmake-build/onnx/onnx_operators_pb.py
  Processing /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/onnx/onnx-data.in.proto
  Writing /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/.setuptools-cmake-build/onnx/onnx-data.proto
  Writing /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/.setuptools-cmake-build/onnx/onnx-data.proto3
  generating /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/.setuptools-cmake-build/onnx/onnx_data_pb.py
  [  8%] Running C++ protocol buffer compiler on /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/.setuptools-cmake-build/onnx/onnx-data.proto
  [  8%] Running C++ protocol buffer compiler on /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/.setuptools-cmake-build/onnx/onnx-operators-ml.proto
  Failed to generate mypy stubs: No module named 'google'
  Failed to generate mypy stubs: No module named 'google'
  [  8%] Built target gen_onnx_operators_proto
  [  8%] Built target gen_onnx_data_proto
  [ 10%] Building CXX object CMakeFiles/onnx_proto.dir/onnx/onnx-ml.pb.cc.o
  [ 11%] Building CXX object CMakeFiles/onnx_proto.dir/onnx/onnx-operators-ml.pb.cc.o
  [ 13%] Building CXX object CMakeFiles/onnx_proto.dir/onnx/onnx-data.pb.cc.o
  [ 14%] Linking CXX static library libonnx_proto.a
  [ 23%] Built target onnx_proto
  [ 26%] Building CXX object CMakeFiles/onnx.dir/onnx/common/assertions.cc.o
  [ 26%] Building CXX object CMakeFiles/onnx.dir/onnx/checker.cc.o
  [ 27%] Building CXX object CMakeFiles/onnx.dir/onnx/common/interned_strings.cc.o
  [ 29%] Building CXX object CMakeFiles/onnx.dir/onnx/common/path.cc.o
  [ 30%] Building CXX object CMakeFiles/onnx.dir/onnx/defs/attr_proto_util.cc.o
  [ 32%] Building CXX object CMakeFiles/onnx.dir/onnx/common/status.cc.o
  [ 35%] Building CXX object CMakeFiles/onnx.dir/onnx/common/model_helpers.cc.o
  [ 35%] Building CXX object CMakeFiles/onnx.dir/onnx/common/ir_pb_converter.cc.o
  [ 36%] Building CXX object CMakeFiles/onnx.dir/onnx/defs/controlflow/defs.cc.o
  [ 38%] Building CXX object CMakeFiles/onnx.dir/onnx/defs/controlflow/old.cc.o
  [ 39%] Building CXX object CMakeFiles/onnx.dir/onnx/defs/controlflow/utils.cc.o
  In file included from /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/onnx/defs/schema.h:25:0,
                   from /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/onnx/defs/function.h:16,
                   from /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/onnx/checker.h:10,
                   from /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/onnx/checker.cc:5:
  /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/onnx/defs/shape_inference.h: In function ‘void onnx::adjustNegativeAxes(Axes&, int)’:
  /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/onnx/defs/shape_inference.h:828:8: error: ‘transform’ is not a member of ‘std’
     std::transform(
          ^~~~~~~~~
  In file included from /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/onnx/defs/schema.h:25:0,
                   from /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/onnx/defs/function.h:16,
                   from /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/onnx/checker.h:10,
                   from /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/onnx/common/model_helpers.cc:6:
  /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/onnx/defs/shape_inference.h: In function ‘void onnx::adjustNegativeAxes(Axes&, int)’:
  /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/onnx/defs/shape_inference.h:828:8: error: ‘transform’ is not a member of ‘std’
     std::transform(
          ^~~~~~~~~
  [ 41%] Building CXX object CMakeFiles/onnx.dir/onnx/defs/data_type_utils.cc.o
  In file included from /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/onnx/defs/schema.h:25:0,
                   from /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/onnx/defs/controlflow/utils.h:7,
                   from /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/onnx/defs/controlflow/defs.cc:8:
  /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/onnx/defs/shape_inference.h: In function ‘void onnx::adjustNegativeAxes(Axes&, int)’:
  /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/onnx/defs/shape_inference.h:828:8: error: ‘transform’ is not a member of ‘std’
     std::transform(
          ^~~~~~~~~
  In file included from /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/onnx/defs/schema.h:25:0,
                   from /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/onnx/defs/controlflow/utils.h:7,
                   from /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/onnx/defs/controlflow/old.cc:7:
  /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/onnx/defs/shape_inference.h: In function ‘void onnx::adjustNegativeAxes(Axes&, int)’:
  /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/onnx/defs/shape_inference.h:828:8: error: ‘transform’ is not a member of ‘std’
     std::transform(
          ^~~~~~~~~
  In file included from /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/onnx/defs/schema.h:25:0,
                   from /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/onnx/defs/math/utils.h:7,
                   from /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/onnx/defs/controlflow/utils.cc:5:
  /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/onnx/defs/shape_inference.h: In function ‘void onnx::adjustNegativeAxes(Axes&, int)’:
  /tmp/pip-install-_f9bz3go/onnx_220296baf1f04f36a37198b3e706c687/onnx/defs/shape_inference.h:828:8: error: ‘transform’ is not a member of ‘std’
     std::transform(
          ^~~~~~~~~
  CMakeFiles/onnx.dir/build.make:131: recipe for target 'CMakeFiles/onnx.dir/onnx/common/model_helpers.cc.o' failed
  make[2]: *** [CMakeFiles/onnx.dir/onnx/common/model_helpers.cc.o] Error 1
  make[2]: *** Waiting for unfinished jobs....
  CMakeFiles/onnx.dir/build.make:187: recipe for target 'CMakeFiles/onnx.dir/onnx/defs/controlflow/defs.cc.o' failed
  make[2]: *** [CMakeFiles/onnx.dir/onnx/defs/controlflow/defs.cc.o] Error 1
  CMakeFiles/onnx.dir/build.make:201: recipe for target 'CMakeFiles/onnx.dir/onnx/defs/controlflow/old.cc.o' failed
  make[2]: *** [CMakeFiles/onnx.dir/onnx/defs/controlflow/old.cc.o] Error 1
  CMakeFiles/onnx.dir/build.make:215: recipe for target 'CMakeFiles/onnx.dir/onnx/defs/controlflow/utils.cc.o' failed
  make[2]: *** [CMakeFiles/onnx.dir/onnx/defs/controlflow/utils.cc.o] Error 1
  CMakeFiles/onnx.dir/build.make:75: recipe for target 'CMakeFiles/onnx.dir/onnx/checker.cc.o' failed
  make[2]: *** [CMakeFiles/onnx.dir/onnx/checker.cc.o] Error 1
  CMakeFiles/Makefile2:212: recipe for target 'CMakeFiles/onnx.dir/all' failed
  make[1]: *** [CMakeFiles/onnx.dir/all] Error 2
  Makefile:135: recipe for target 'all' failed
  make: *** [all] Error 2
  Traceback (most recent call last):
    File "/home/ubuntu/.local/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
      main()
    File "/home/ubuntu/.local/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/home/ubuntu/.local/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 262, in build_wheel
      metadata_directory)
    File "/tmp/pip-build-env-yed7_gva/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 231, in build_wheel
      wheel_directory, config_settings)
    File "/tmp/pip-build-env-yed7_gva/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 215, in _build_with_temp_dir
      self.run_setup()
    File "/tmp/pip-build-env-yed7_gva/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 268, in run_setup
      self).run_setup(setup_script=setup_script)
    File "/tmp/pip-build-env-yed7_gva/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 158, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 365, in <module>
      "backend-test-tools = onnx.backend.test.cmd_tools:main",
    File "/tmp/pip-build-env-yed7_gva/overlay/lib/python3.6/site-packages/setuptools/__init__.py", line 153, in setup
      return distutils.core.setup(**attrs)
    File "/usr/lib/python3.6/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/lib/python3.6/distutils/dist.py", line 955, in run_commands
      self.run_command(cmd)
    File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-env-yed7_gva/overlay/lib/python3.6/site-packages/wheel/bdist_wheel.py", line 299, in run
      self.run_command('build')
    File "/usr/lib/python3.6/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/usr/lib/python3.6/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/usr/lib/python3.6/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "setup.py", line 236, in run
      self.run_command("cmake_build")
    File "/usr/lib/python3.6/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "setup.py", line 230, in run
      subprocess.check_call(build_args)
    File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['/usr/local/bin/cmake', '--build', '.', '--', '-j', '8']' returned non-zero exit status 2.
  ----------------------------------------
  ERROR: Failed building wheel for onnx
Failed to build onnx
ERROR: Could not build wheels for onnx, which is required to install pyproject.toml-based projects
7

yolov5_gpu_optimization is a sample to use yolov5, yolov5_decode.so encapsulates model parsing function and postprocess function. if using your yolov5 model, the app dose not need model parsing function because the engine was already generated. you only need to compile postprocess function to a so lib. please refer to post_processor to compile the custom function NvDsInferParseYolo.

8

OK, so I don’t need to run export because I already have yolov5s.onnx.

I sucessfully generated the yolov5_decode.so using nvcc.

Also, copied the onnx file to yolov5_gpu_optimization/deepstream-sample/

ubuntu@ubuntu:~/edge/yolo_install/yolov5_gpu_optimization-main/deepstream-sample$ ls
README.md  config  nvdsparsebbox_Yolo.cpp  yoloForward_nc.cu  yoloPlugins.cpp  yoloPlugins.h  yolov5_decode.so  yolov5s.onnx

But when I tried to run inference using deepstream-app, I got an error in the config file:

ubuntu@ubuntu:~/edge/yolo_install/yolov5_gpu_optimization-main/deepstream-sample/config$ deepstream-app -c config/deepstream_app_config_save_video.txt

(gst-plugin-scanner:3254): GStreamer-WARNING **: 16:23:05.116: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_udp.so': librivermax.so.0: cannot open shared object file: No such file or directory

(gst-plugin-scanner:3254): GStreamer-WARNING **: 16:23:05.119: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_inferserver.so': libtritonserver.so: cannot open shared object file: No such file or directory

(gst-plugin-scanner:3254): GStreamer-WARNING **: 16:23:05.215: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libcustom2d_preprocess.so': /usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libcustom2d_preprocess.so: undefined symbol: NvBufSurfTransformAsync

(gst-plugin-scanner:3254): GStreamer-WARNING **: 16:23:05.238: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_preprocess.so': /usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_preprocess.so: undefined symbol: NvBufSurfTransformAsync
** ERROR: <parse_config_file:536>: parse_config_file failed
** ERROR: <main:630>: Failed to parse config file 'config/deepstream_app_config_save_video.txt'
Quitting
App run failed

This is the deepstream_app_config_save_video.txt config file:

[application]
enable-perf-measurement=1
perf-measurement-interval-sec=5

[tiled-display]
enable=1
rows=1
columns=1
width=1280
height=720
gpu-id=0
nvbuf-memory-type=0

[source0]
enable=1
type=3
uri=file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4
num-sources=1
gpu-id=0
cudadec-memtype=0

[sink0]
enable=1
type=3
sync=0
gpu-id=0
container=1
codec=1
enc-type=1
bitrate=4000000
output-file=./out.mp4
nvbuf-memory-type=0

[osd]
enable=1
gpu-id=0
border-width=5
text-size=15
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Serif
show-clock=0
clock-x-offset=800
clock-y-offset=820
clock-text-size=12
clock-color=1;0;0;0
nvbuf-memory-type=0

[streammux]
gpu-id=0
live-source=0
batch-size=1
batched-push-timeout=40000
width=1920
height=1080
enable-padding=0
nvbuf-memory-type=0

[primary-gie]
enable=1
gpu-id=0
gie-unique-id=1
nvbuf-memory-type=0
config-file=config/config_infer_primary_yoloV5.txt

[tests]
file-loop=1

I checked /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4, it do exists in such path.

And this is the config_infer_primary_yoloV5.txt:

[property]
gpu-id=0
net-scale-factor=0.0039215697906911373
model-color-format=0
onnx-file=./yolov5s.onnx
model-engine-file=./yolov5s.onnx_b1_gpu1_fp16.engine
infer-dims=3;640;640
labelfile-path=config/labels.txt
batch-size=1
workspace-size=1024
network-mode=2
num-detected-classes=80
interval=0
gie-unique-id=1
process-mode=1
network-type=0
cluster-mode=2
maintain-aspect-ratio=1
parse-bbox-func-name=NvDsInferParseYolo
custom-lib-path=./yolov5_decode.so

[class-attrs-all]
nms-iou-threshold=0.45
pre-cluster-threshold=0.25
topk=300
9
  1. noticing there is error “undefined symbol: NvBufSurfTransformAsync”, can you run the simplest sample deepstream-test1 successfully?
  2. could you share more logs? please do “export GST_DEBUG=6” first to modify Gstreamer’s log level, then run again, you can redirect the logs to a file.
  3. from the log “parse_config_file failed”, it failed when decoding the configuration file, please compare the files with deepstream sample files.
10
  1. Yes deepstream-test1 runs successfully.
  2. The Debug log level 6 is in attachment
  3. I added the full path for every file cited in both config files, and bypassed the config file errors, but the error “undefined symbol: NvBufSurfTransformAsync” remains and there is another one concerning the engine creation.
ubuntu@ubuntu:~/edge/yolo_install/yolov5_gpu_optimization-main/deepstream-sample$ deepstream-app -c config/deepstream_app_config_save_video.txt

(gst-plugin-scanner:21268): GStreamer-WARNING **: 13:14:01.919: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_udp.so': librivermax.so.0: cannot open shared object file: No such file or directory

(gst-plugin-scanner:21268): GStreamer-WARNING **: 13:14:01.923: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_inferserver.so': libtritonserver.so: cannot open shared object file: No such file or directory

(gst-plugin-scanner:21268): GStreamer-WARNING **: 13:14:02.021: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libcustom2d_preprocess.so': /usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libcustom2d_preprocess.so: undefined symbol: NvBufSurfTransformAsync

(gst-plugin-scanner:21268): GStreamer-WARNING **: 13:14:02.046: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_preprocess.so': /usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_preprocess.so: undefined symbol: NvBufSurfTransformAsync
ERROR: Deserialize engine failed because file path: /home/ubuntu/edge/yolo_install/yolov5_gpu_optimization-main/deepstream-sample/config/yolov5s.onnx_b1_gpu1_fp16.engine open error
0:00:02.873275630 21267     0x1b2446d0 WARN                 nvinfer gstnvinfer.cpp:635:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1889> [UID = 1]: deserialize engine from file :/home/ubuntu/edge/yolo_install/yolov5_gpu_optimization-main/deepstream-sample/config/yolov5s.onnx_b1_gpu1_fp16.engine failed
0:00:02.873381075 21267     0x1b2446d0 WARN                 nvinfer gstnvinfer.cpp:635:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1996> [UID = 1]: deserialize backend context from engine from file :/home/ubuntu/edge/yolo_install/yolov5_gpu_optimization-main/deepstream-sample/config/yolov5s.onnx_b1_gpu1_fp16.engine failed, try rebuild
0:00:02.873405812 21267     0x1b2446d0 INFO                 nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1914> [UID = 1]: Trying to create engine from model files
WARNING: [TRT]: onnx2trt_utils.cpp:364: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
ERROR: [TRT]: ModelImporter.cpp:720: While parsing node number 210 [Range -> "349"]:
ERROR: [TRT]: ModelImporter.cpp:721: --- Begin node ---
ERROR: [TRT]: ModelImporter.cpp:722: input: "347"
input: "346"
input: "348"
output: "349"
name: "Range_217"
op_type: "Range"

ERROR: [TRT]: ModelImporter.cpp:723: --- End node ---
ERROR: [TRT]: ModelImporter.cpp:726: ERROR: builtin_op_importers.cpp:3172 In function importRange:
[8] Assertion failed: inputs.at(0).isInt32() && "For range operator with dynamic inputs, this version of TensorRT only supports INT32!"
ERROR: Failed to parse onnx file
ERROR: failed to build network since parsing model errors.

log.txt (30.9 MB)

11

from the log, the app created TensorRT engine failed. please use trtexec to test first. here is a sample command, especially please use the correct model resolution.
/usr/src/tensorrt/bin/trtexec --fp16 --onnx=yolov5s.onnx
–saveEngine=yolov5s.onnx_b4_gpu0_fp16.engine --minShapes=images:1x3x672x672
–optShapes=images:4x3x672x672 --maxShapes=images:4x3x672x672 --shapes=images:4x3x672x672 --workspace=10000

12

This is what I get:

ubuntu@ubuntu:~/edge/yolo_install/yolov5_gpu_optimization-main/deepstream-sample$ /usr/src/tensorrt/bin/trtexec --fp16 --onnx=yolov5s.onnx --saveEngine=yolov5s.onnx_b4_gpu0_fp16.engine --minShapes=images:1x3x672x672 --optShapes=images:4x3x672x672 --maxShapes=images:4x3x672x672 --shapes=images:4x3x672x672 --workspace=10000
&&&& RUNNING TensorRT.trtexec [TensorRT v8001] # /usr/src/tensorrt/bin/trtexec --fp16 --onnx=yolov5s.onnx --saveEngine=yolov5s.onnx_b4_gpu0_fp16.engine --minShapes=images:1x3x672x672 --optShapes=images:4x3x672x672 --maxShapes=images:4x3x672x672 --shapes=images:4x3x672x672 --workspace=10000
[06/29/2023-11:39:36] [I] === Model Options ===
[06/29/2023-11:39:36] [I] Format: ONNX
[06/29/2023-11:39:36] [I] Model: yolov5s.onnx
[06/29/2023-11:39:36] [I] Output:
[06/29/2023-11:39:36] [I] === Build Options ===
[06/29/2023-11:39:36] [I] Max batch: explicit
[06/29/2023-11:39:36] [I] Workspace: 10000 MiB
[06/29/2023-11:39:36] [I] minTiming: 1
[06/29/2023-11:39:36] [I] avgTiming: 8
[06/29/2023-11:39:36] [I] Precision: FP32+FP16
[06/29/2023-11:39:36] [I] Calibration: 
[06/29/2023-11:39:36] [I] Refit: Disabled
[06/29/2023-11:39:36] [I] Sparsity: Disabled
[06/29/2023-11:39:36] [I] Safe mode: Disabled
[06/29/2023-11:39:36] [I] Restricted mode: Disabled
[06/29/2023-11:39:36] [I] Save engine: yolov5s.onnx_b4_gpu0_fp16.engine
[06/29/2023-11:39:36] [I] Load engine: 
[06/29/2023-11:39:36] [I] NVTX verbosity: 0
[06/29/2023-11:39:36] [I] Tactic sources: Using default tactic sources
[06/29/2023-11:39:36] [I] timingCacheMode: local
[06/29/2023-11:39:36] [I] timingCacheFile: 
[06/29/2023-11:39:36] [I] Input(s)s format: fp32:CHW
[06/29/2023-11:39:36] [I] Output(s)s format: fp32:CHW
[06/29/2023-11:39:36] [I] Input build shape: images=1x3x672x672+4x3x672x672+4x3x672x672
[06/29/2023-11:39:36] [I] Input calibration shapes: model
[06/29/2023-11:39:36] [I] === System Options ===
[06/29/2023-11:39:36] [I] Device: 0
[06/29/2023-11:39:36] [I] DLACore: 
[06/29/2023-11:39:36] [I] Plugins:
[06/29/2023-11:39:36] [I] === Inference Options ===
[06/29/2023-11:39:36] [I] Batch: Explicit
[06/29/2023-11:39:36] [I] Input inference shape: images=4x3x672x672
[06/29/2023-11:39:36] [I] Iterations: 10
[06/29/2023-11:39:36] [I] Duration: 3s (+ 200ms warm up)
[06/29/2023-11:39:36] [I] Sleep time: 0ms
[06/29/2023-11:39:36] [I] Streams: 1
[06/29/2023-11:39:36] [I] ExposeDMA: Disabled
[06/29/2023-11:39:36] [I] Data transfers: Enabled
[06/29/2023-11:39:36] [I] Spin-wait: Disabled
[06/29/2023-11:39:36] [I] Multithreading: Disabled
[06/29/2023-11:39:36] [I] CUDA Graph: Disabled
[06/29/2023-11:39:36] [I] Separate profiling: Disabled
[06/29/2023-11:39:36] [I] Time Deserialize: Disabled
[06/29/2023-11:39:36] [I] Time Refit: Disabled
[06/29/2023-11:39:36] [I] Skip inference: Disabled
[06/29/2023-11:39:36] [I] Inputs:
[06/29/2023-11:39:36] [I] === Reporting Options ===
[06/29/2023-11:39:36] [I] Verbose: Disabled
[06/29/2023-11:39:36] [I] Averages: 10 inferences
[06/29/2023-11:39:36] [I] Percentile: 99
[06/29/2023-11:39:36] [I] Dump refittable layers:Disabled
[06/29/2023-11:39:36] [I] Dump output: Disabled
[06/29/2023-11:39:36] [I] Profile: Disabled
[06/29/2023-11:39:36] [I] Export timing to JSON file: 
[06/29/2023-11:39:36] [I] Export output to JSON file: 
[06/29/2023-11:39:36] [I] Export profile to JSON file: 
[06/29/2023-11:39:36] [I] 
[06/29/2023-11:39:36] [I] === Device Information ===
[06/29/2023-11:39:36] [I] Selected Device: Xavier
[06/29/2023-11:39:36] [I] Compute Capability: 7.2
[06/29/2023-11:39:36] [I] SMs: 8
[06/29/2023-11:39:36] [I] Compute Clock Rate: 1.377 GHz
[06/29/2023-11:39:36] [I] Device Global Memory: 31920 MiB
[06/29/2023-11:39:36] [I] Shared Memory per SM: 96 KiB
[06/29/2023-11:39:36] [I] Memory Bus Width: 256 bits (ECC disabled)
[06/29/2023-11:39:36] [I] Memory Clock Rate: 1.377 GHz
[06/29/2023-11:39:36] [I] 
[06/29/2023-11:39:36] [I] TensorRT version: 8001
[06/29/2023-11:39:38] [I] [TRT] [MemUsageChange] Init CUDA: CPU +354, GPU +0, now: CPU 372, GPU 12168 (MiB)
[06/29/2023-11:39:38] [I] Start parsing network model
[06/29/2023-11:39:38] [I] [TRT] ----------------------------------------------------------------
[06/29/2023-11:39:38] [I] [TRT] Input filename:   yolov5s.onnx
[06/29/2023-11:39:38] [I] [TRT] ONNX IR version:  0.0.7
[06/29/2023-11:39:38] [I] [TRT] Opset version:    12
[06/29/2023-11:39:38] [I] [TRT] Producer name:    pytorch
[06/29/2023-11:39:38] [I] [TRT] Producer version: 1.10
[06/29/2023-11:39:38] [I] [TRT] Domain:           
[06/29/2023-11:39:38] [I] [TRT] Model version:    0
[06/29/2023-11:39:38] [I] [TRT] Doc string:       
[06/29/2023-11:39:38] [I] [TRT] ----------------------------------------------------------------
[06/29/2023-11:39:38] [W] [TRT] onnx2trt_utils.cpp:364: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[06/29/2023-11:39:38] [E] [TRT] ModelImporter.cpp:720: While parsing node number 210 [Range -> "349"]:
[06/29/2023-11:39:38] [E] [TRT] ModelImporter.cpp:721: --- Begin node ---
[06/29/2023-11:39:38] [E] [TRT] ModelImporter.cpp:722: input: "347"
input: "346"
input: "348"
output: "349"
name: "Range_217"
op_type: "Range"

[06/29/2023-11:39:38] [E] [TRT] ModelImporter.cpp:723: --- End node ---
[06/29/2023-11:39:38] [E] [TRT] ModelImporter.cpp:726: ERROR: builtin_op_importers.cpp:3172 In function importRange:
[8] Assertion failed: inputs.at(0).isInt32() && "For range operator with dynamic inputs, this version of TensorRT only supports INT32!"
[06/29/2023-11:39:38] [E] Failed to parse onnx file
[06/29/2023-11:39:38] [I] Finish parsing network model
[06/29/2023-11:39:38] [E] Parsing model failed
[06/29/2023-11:39:38] [E] Engine creation failed
[06/29/2023-11:39:38] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v8001] # /usr/src/tensorrt/bin/trtexec --fp16 --onnx=yolov5s.onnx --saveEngine=yolov5s.onnx_b4_gpu0_fp16.engine --minShapes=images:1x3x672x672 --optShapes=images:4x3x672x672 --maxShapes=images:4x3x672x672 --shapes=images:4x3x672x672 --workspace=10000
13

from the log above, the Input format is fp32:CHW. converting failed becuase TensorRT only supports Int32.

14

Hi @flavio.mello ,
From error log below, it run into ONNX paser check in https://github.com/onnx/onnx-tensorrt/blob/8.2-GA/builtin_op_importers.cpp#L3268 for ‘Range’ operator, so you need to modify ONNX to make sure the inputs.at(0) of “Range_217” node to be INT32 format.

[06/29/2023-11:39:38] [E] [TRT] ModelImporter.cpp:722: input: “347”
input: “346”
input: “348”
output: “349”
name: “Range_217”
op_type: “Range”

[06/29/2023-11:39:38] [E] [TRT] ModelImporter.cpp:723: — End node —
[06/29/2023-11:39:38] [E] [TRT] ModelImporter.cpp:726: ERROR: builtin_op_importers.cpp:3172 In function importRange:
[8] Assertion failed: inputs.at(0).isInt32() && “For range operator with dynamic inputs, this version of TensorRT only supports INT32!”

15

So, I am hooked many days trying to convert onnx from fp32 to int32. Tried ultralytics export.py.

Is there any library/script to convert from fp32 to int32?
Or should I try to upgrade Jetson TensorRT ?

18

Hi @flavio.mello,
Sorry! I missed your update.

Could youj try GitHub - ZhangGe6/onnx-modifier: A tool to modify ONNX models in a visualization fashion, based on Netron and Flask. ?

And, can you open your ONNX file with Netron and share the screenshot of the Range node like below?

image
image913×640 26.9 KB

19

So, I couldn’t find “Range_217”, but I found “output 349”. This is what I got:

Screenshot from 2023-07-19 15-29-02
Screenshot from 2023-07-19 15-29-02924×721 97.5 KB

20

This node does not match below logs.

21

I can’t find the error node with this viewer. Maybe we are in the wrong path.

Let me change it a bit: where must I train my model so that it can be consumed by Deepstream easily/native?

22

Is it possible to share me your onnx by dropping a private messge?

23

I have just sent to you