JetPack 4.4 support

Please provide complete information as applicable to your setup.
• Hardware Platform (Jetson / GPU)
JetPack 4.4
• DeepStream Version
5.0.0-1
• JetPack Version (valid for Jetson only)
4.4 GA
• TensorRT Version
7,1

I’m currently paused on the DeepStream based project I’ve been workign on for the past month of so since DeepStream seems not to work anymore on any fully updated platform I have. On AGX, and NX I get a segfault when doing a gst-inspect-1.0 nvinfer

 $ GST_DEBUG=4 gdb --args gst-inspect-1.0 nvinfer
GNU gdb (Ubuntu 8.1-0ubuntu3.2) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "aarch64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from gst-inspect-1.0...(no debugging symbols found)...done.
(gdb) run
Starting program: /usr/bin/gst-inspect-1.0 nvinfer
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
0:00:00.000136263 12123   0x5555580a00 INFO                GST_INIT gst.c:586:init_pre: Initializing GStreamer Core Library version 1.14.5
0:00:00.000262734 12123   0x5555580a00 INFO                GST_INIT gst.c:587:init_pre: Using library installed in /usr/lib/aarch64-linux-gnu
0:00:00.000336594 12123   0x5555580a00 INFO                GST_INIT gst.c:607:init_pre: Linux motherbird 4.9.140-tegra #1 SMP PREEMPT Thu Jun 25 21:22:12 PDT 2020 aarch64
0:00:00.001253764 12123   0x5555580a00 INFO                GST_INIT gstmessage.c:127:_priv_gst_message_initialize: init messages
0:00:00.002809658 12123   0x5555580a00 INFO                GST_INIT gstcontext.c:84:_priv_gst_context_initialize: init contexts
0:00:00.003432092 12123   0x5555580a00 INFO      GST_PLUGIN_LOADING gstplugin.c:317:_priv_gst_plugin_initialize: registering 0 static plugins
0:00:00.003642503 12123   0x5555580a00 INFO      GST_PLUGIN_LOADING gstplugin.c:225:gst_plugin_register_static: registered static plugin "staticelements"
0:00:00.003709355 12123   0x5555580a00 INFO      GST_PLUGIN_LOADING gstplugin.c:227:gst_plugin_register_static: added static plugin "staticelements", result: 1
0:00:00.003778447 12123   0x5555580a00 INFO            GST_REGISTRY gstregistry.c:1727:ensure_current_registry: reading registry cache: /home/anzu/.cache/gstreamer-1.0/registry.aarch64.bin
0:00:00.040040916 12123   0x5555580a00 INFO            GST_REGISTRY gstregistrybinary.c:621:priv_gst_registry_binary_read_cache: loaded /home/anzu/.cache/gstreamer-1.0/registry.aarch64.bin in 0.036179 seconds
0:00:00.040180444 12123   0x5555580a00 INFO            GST_REGISTRY gstregistry.c:1583:scan_and_update_registry: Validating plugins from registry cache: /home/anzu/.cache/gstreamer-1.0/registry.aarch64.bin
0:00:00.042614273 12123   0x5555580a00 INFO            GST_REGISTRY gstregistry.c:1685:scan_and_update_registry: Registry cache has not changed
0:00:00.042651683 12123   0x5555580a00 INFO            GST_REGISTRY gstregistry.c:1762:ensure_current_registry: registry reading and updating done, result = 1
0:00:00.042675684 12123   0x5555580a00 INFO                GST_INIT gst.c:807:init_post: GLib runtime version: 2.56.4
0:00:00.042698534 12123   0x5555580a00 INFO                GST_INIT gst.c:809:init_post: GLib headers version: 2.56.4
0:00:00.042713446 12123   0x5555580a00 INFO                GST_INIT gst.c:810:init_post: initialized GStreamer successfully
[New Thread 0x7f97ffb1f0 (LWP 12127)]
[New Thread 0x7f977fa1f0 (LWP 12128)]
[New Thread 0x7f96ff91f0 (LWP 12129)]
[New Thread 0x7f967f81f0 (LWP 12130)]
[New Thread 0x7f95ff71f0 (LWP 12131)]
[New Thread 0x7f957f61f0 (LWP 12132)]
[New Thread 0x7f93a5b1f0 (LWP 12133)]
0:00:00.651035772 12123   0x5555580a00 INFO      GST_PLUGIN_LOADING gstplugin.c:901:_priv_gst_plugin_load_file_for_registry: plugin "/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_infer.so" loaded
0:00:00.651147171 12123   0x5555580a00 INFO     GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element "nvinfer"
0:00:00.651759172 12123   0x5555580a00 INFO        GST_ELEMENT_PADS gstelement.c:670:gst_element_add_pad:<GstBaseTransform@0x5555a39940> adding pad 'sink'
0:00:00.651842185 12123   0x5555580a00 INFO        GST_ELEMENT_PADS gstelement.c:670:gst_element_add_pad:<GstBaseTransform@0x5555a39940> adding pad 'src'
Factory Details:
  Rank                     primary (256)
  Long-name                NvInfer plugin
  Klass                    NvInfer Plugin
  Description              Nvidia DeepStreamSDK TensorRT plugin
  Author                   NVIDIA Corporation. Deepstream for Tesla forum: https://devtalk.nvidia.com/default/board/209

Plugin Details:
  Name                     nvdsgst_infer
  Description              NVIDIA DeepStreamSDK TensorRT plugin
  Filename                 /usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_infer.so
  Version                  5.0.0
  License                  Proprietary
  Source module            nvinfer
  Binary package           NVIDIA DeepStreamSDK TensorRT plugin
  Origin URL               http://nvidia.com/

GObject
 +----GInitiallyUnowned
       +----GstObject
             +----GstElement
                   +----GstBaseTransform
                         +----GstNvInfer

Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw(memory:NVMM)
                 format: { (string)NV12, (string)RGBA }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
  
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-raw(memory:NVMM)
                 format: { (string)NV12, (string)RGBA }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]

Element has no clocking capabilities.
Element has no URI handling capabilities.

Pads:
  SINK: 'sink'
    Pad Template: 'sink'
  SRC: 'src'
    Pad Template: 'src'

Element Properties:
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "nvinfer0"
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  qos                 : Handle Quality-of-Service events
                        flags: readable, writable
                        Boolean. Default: false
  unique-id           : Unique ID for the element. Can be used to identify output of the element
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 4294967295 Default: 15 
  process-mode        : Infer processing mode
                        flags: readable, writable, changeable only in NULL or READY state
                        Enum "GstNvInferProcessModeType" Default: 1, "primary"
                           (1): primary          - Primary (Full Frame)
                           (2): secondary        - Secondary (Objects)
  config-file-path    : Path to the configuration file for this instance of nvinfer
                        flags: readable, writable, changeable in NULL, READY, PAUSED or PLAYING state
                        String. Default: ""
  infer-on-gie-id     : Infer on metadata generated by GIE with this unique ID.
			Set to -1 to infer on all metadata.
                        flags: readable, writable, changeable only in NULL or READY state
                        Integer. Range: -1 - 2147483647 Default: -1 
  infer-on-class-ids  : Operate on objects with specified class ids
			Use string with values of class ids in ClassID (int) to set the property.
			 e.g. 0:2:3
                        flags: readable, writable, changeable only in NULL or READY state
                        String. Default: ""
  filter-out-class-ids: Ignore metadata for objects of specified class ids
			Use string with values of class ids in ClassID (int) to set the property.
			 e.g. 0;2;3
                        flags: readable, writable, changeable only in NULL or READY state
                        String. Default: ""
  model-engine-file   : Absolute path to the pre-generated serialized engine file for the model
                        flags: readable, writable, changeable in NULL, READY, PAUSED or PLAYING state
                        String. Default: ""
  batch-size          : Maximum batch size for inference
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 1 - 1024 Default: 1 
  interval            : Specifies number of consecutive batches to be skipped for inference
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 2147483647 Default: 0 
  gpu-id              : Set GPU Device ID
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 4294967295 Default: 0 
  raw-output-file-write: Write raw inference output to file
                        flags: readable, writable, changeable only in NULL or READY state
                        Boolean. Default: false
  raw-output-generated-callback: Pointer to the raw output generated callback funtion
			(type: gst_nvinfer_raw_output_generated_callback in 'gstnvdsinfer.h')
                        flags: readable, writable, changeable only in NULL or READY state
                        Pointer.
  raw-output-generated-userdata: Pointer to the userdata to be supplied with raw output generated callback
                        flags: readable, writable, changeable only in NULL or READY state
                        Pointer.
  output-tensor-meta  : Attach inference tensor outputs as buffer metadata
                        flags: readable, writable, changeable only in NULL or READY state
                        Boolean. Default: false

Element Signals:
  "model-updated" :  void user_function (GstElement* object,
                                         gint arg0,
                                         gchararray arg1,
                                         gpointer user_data);
0:00:00.656294301 12123   0x5555580a00 INFO         GST_REFCOUNTING gstelement.c:3251:gst_element_dispose:<nvinfer0> 0x5555a39940 dispose
0:00:00.656323774 12123   0x5555580a00 INFO        GST_ELEMENT_PADS gstelement.c:789:gst_element_remove_pad:<nvinfer0> removing pad 'sink'
0:00:00.656377441 12123   0x5555580a00 INFO        GST_ELEMENT_PADS gstelement.c:789:gst_element_remove_pad:<nvinfer0> removing pad 'src'
0:00:00.656407523 12123   0x5555580a00 INFO         GST_REFCOUNTING gstelement.c:3297:gst_element_dispose:<nvinfer0> 0x5555a39940 parent class dispose
0:00:00.656432292 12123   0x5555580a00 INFO         GST_REFCOUNTING gstelement.c:3328:gst_element_finalize:<nvinfer0> 0x5555a39940 finalize
0:00:00.656452134 12123   0x5555580a00 INFO         GST_REFCOUNTING gstelement.c:3334:gst_element_finalize:<nvinfer0> 0x5555a39940 finalize parent
[Thread 0x7f93a5b1f0 (LWP 12133) exited]
[Thread 0x7f95ff71f0 (LWP 12131) exited]
[Thread 0x7f96ff91f0 (LWP 12129) exited]
[Thread 0x7f957f61f0 (LWP 12132) exited]
[Thread 0x7f967f81f0 (LWP 12130) exited]
[Thread 0x7f977fa1f0 (LWP 12128) exited]
[Thread 0x7f97ffb1f0 (LWP 12127) exited]

Thread 1 "gst-inspect-1.0" received signal SIGSEGV, Segmentation fault.
__GI___pthread_mutex_lock (mutex=0x8) at pthread_mutex_lock.c:65
65	pthread_mutex_lock.c: No such file or directory.
(gdb) bt
#0  0x0000007fb7a996f0 in __GI___pthread_mutex_lock (mutex=0x8) at pthread_mutex_lock.c:65
#1  0x0000007f98241344 in  () at /usr/lib/aarch64-linux-gnu/libEGL_mesa.so.0

Apart from reflashing with JetPack 4.4 DP and not updating to the GA, is there a way to get DeepStream running on a fully updated JetPack 4.4?

Reflashing means a fair amount of work. Can a patched package, not necessarily a final GA, please be pushed so those who updated to JetPack 4.4 can continue to work on DeepStream projects?

Usually I would just fall back to x86, but that stopped working around the same time as well for unrelated reasons. Solving either one of these issues would be very much appreciated.

Hi @mdegans,

DeepStream seems not to work anymore on any fully updated platform

What does “fully updated platform” mean? How did you get the “fully updated platform”, by flashing the system with SDKManager+JP4.4?

We have tried JP4.4 (not JP4.4DP) + DS5.0DP, we don’t see issue about nvinfer so far.

Thanks!

@mchi
My AGX was originally flashed to 4.3 and updated to 4.4 using this procedure here. My NX was flashed to 4.4 DP and upgraded to GA with apt as well. Both systems segfault when trying to use nvinfer. Are there any known issues with this upgrade path?

Could you follow https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide%2Fquick_start.html%23wwpID0EVHA to upgrade from Jetpack4.3 to Jetpack4.4? Before upgrding, we need to make sure there is enough free disk space.
Could refer to https://docs.nvidia.com/jetson/jetpack/install-jetpack/index.html#upgrade-jetpack to remove some components to free the disk space.

Thanks!

@mchi

That would be step one of the link in my own post, so I did that plus installing the nvidia-jetpack package as outlined in the upgrade instructions. Both Tegra devices were upgraded this way. There is plenty of free disk space on both devices and I got no errors from apt on upgrade about the disk being full. Any ideas?

Please check below note, which is from https://developer.nvidia.com/embedded/jetpack

JetPack 4.4 supports the upcoming DeepStream 5.0 release

  • DeepStream 5.0 Developer Preview is only supported with JetPack 4.4 Developer Preview.
1 Like

I’m getting mixed messages here. So basically there is no working version of DeepStream on the current JetPack? Meaning: don’t update your system, which is really easy to do because the Ubuntu parts of l4t will keep reminding you. Is there any workaround for this? Details on what doesn’t work and why? Possible packages that can be pinned to older versions?

This is correct and official message to community.
Sorry for keep you waiting. We will have the next release in coming weeks to support JP4.4 GA, please wait for our announcement.

1 Like