Running containerized on Jetson Xavier NX is failing and error messages are not helpful

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

FYI:I have edited out messages that give away private IP.
I have a graph that runs fine on ubuntu both on the “metal” and containerized but fails to run containerized on the Jetson Xavier NX. The messages that it gives don’t point me at all where to look… Are there additional log files or other things I can do to determine what’s failing?

Also, what is the equivalent dGPU command to jetson_release -v?

Running on ubuntu:
2022-08-24 23:27:39.692 INFO gxf/gxe/gxe.cpp@230: Creating context
2022-08-24 23:27:39.707 INFO gxf/gxe/gxe.cpp@291: Initializing…
2022-08-24 23:27:40.027 INFO extensions/nvdsbase/nvds_scheduler.cpp@265: This program is linked against GStreamer 1.16.2
2022-08-24 23:27:40.027 INFO extensions/nvdsmuxdemux/nvstreammux.hpp@27: initialize: nvstreammux nv_ds_stream_mux7…f8d0
2022-08-24 23:27:40.027 INFO extensions/nvdstracker/nvtrackerbin.hpp@23: initialize: nvtrackerbin nv_ds_tracker15…f208
2022-08-24 23:27:40.027 INFO extensions/nvdsvisualization/nvosdbin.hpp@24: initialize: nvosdbin nv_ds_osd31…5940
2022-08-24 23:27:40.033 INFO extensions/nvdsinference/nvinferbin.hpp@24: initialize: nvinferbin nv_ds_infer_video0
2022-08-24 23:27:40.033 INFO extensions/nvdsvisualization/nvtilerbin.hpp@38: initialize: nvtilerbin nv_ds_tiler4
2022-08-24 23:27:40.033 INFO extensions/nvdsoutputsink/nvvideoencfilesinkbin.hpp@23: initialize: nvvideoencfilesinkbin nv_ds_file_out0
2022-08-24 23:27:40.033 INFO gxf/gxe/gxe.cpp@298: Running…
2022-08-24 23:27:40.033 DEBUG extensions/nvdsbase/nvds_scheduler.cpp@102: NvDsScheduler::runAsync_abi
2022-08-24 23:27:40.033 INFO extensions/nvdsbase/nvds_scheduler.cpp@116: Scheduling 7 elements and 3 components
2022-08-24 23:27:40.033 INFO extensions/nvdsmuxdemux/nvstreammux.hpp@37: create_element: nvstreammux nv_ds_stream_mux7…f8d0
2022-08-24 23:27:40.066 INFO extensions/nvdsmuxdemux/nvstreammux.hpp@61: bin_add: nvstreammux nv_ds_stream_mux7…f8d0
2022-08-24 23:27:40.066 INFO extensions/nvdstracker/nvtrackerbin.hpp@31: create_element: nvtrackerbin nv_ds_tracker15…f208
2022-08-24 23:27:40.092 INFO extensions/nvdstracker/nvtrackerbin.hpp@55: bin_add: nvtrackerbin nv_ds_tracker15…f208
2022-08-24 23:27:40.092 INFO extensions/nvdsvisualization/nvosdbin.hpp@32: create_element: nvosdbin nv_ds_osd31…5940
2022-08-24 23:27:40.093 INFO extensions/nvdsvisualization/nvosdbin.hpp@56: bin_add: nvosdbin nv_ds_osd31…5940
2022-08-24 23:27:40.093 INFO extensions/nvdsinference/nvinferbin.hpp@32: create_element: nvinferbin nv_ds_infer_video0
2022-08-24 23:27:40.093 INFO extensions/nvdsinference/nvinferbin.hpp@56: bin_add: nvinferbin nv_ds_infer_video0
2022-08-24 23:27:40.093 INFO extensions/nvdsvisualization/nvtilerbin.hpp@46: create_element: nvtilerbin nv_ds_tiler4
2022-08-24 23:27:40.093 INFO extensions/nvdsvisualization/nvtilerbin.hpp@70: bin_add: nvtilerbin nv_ds_tiler4
2022-08-24 23:27:40.093 INFO extensions/nvdssource/multi_uri_src_bin.cpp@343: create_element: NvDsMultiSrcInput nv_ds_multi_src_input18
2022-08-24 23:27:40.093 INFO extensions/nvdssource/multi_uri_src_bin.cpp@389: bin_add: bin nv_ds_multi_src_input18
2022-08-24 23:27:40.093 INFO extensions/nvdsoutputsink/nvvideoencfilesinkbin.hpp@31: create_element: nvvideoencfilesinkbin nv_ds_file_out0
2022-08-24 23:27:40.093 INFO extensions/nvdsoutputsink/nvvideoencfilesinkbin.hpp@49: bin_add: nvvideoencfilesinkbin nv_ds_file_out0
2022-08-24 23:27:40.093 DEBUG extensions/nvdsbase/nvds_connection.cpp@46: nv_ds_static_output9…f9e8 (NvDsStreamMux…0048/nv_ds_stream_mux7…f8d0:src) → nv_ds_static_input2 (NvDsInferVideo/nv_ds_infer_video0:sink) link successful
2022-08-24 23:27:40.093 DEBUG extensions/nvdsbase/nvds_connection.cpp@46: nv_ds_static_output33…8b00 (NvDsOSD…3e48/nv_ds_osd31…5940:src) → nv_ds_static_input1 (NvDsFileOut/nv_ds_file_out0:sink) link successful
2022-08-24 23:27:40.093 DEBUG extensions/nvdsbase/nvds_connection.cpp@46: nv_ds_static_output1 (NvDsInferVideo/nv_ds_infer_video0:src) → nv_ds_static_input16…6a90 (OBJECT TRACKER…3898/nv_ds_tracker15…f208:sink) link successful
2022-08-24 23:27:40.093 DEBUG extensions/nvdsbase/nvds_connection.cpp@46: nv_ds_static_output17…6b38 (OBJECT TRACKER…3898/nv_ds_tracker15…f208:src) → nv_ds_static_input5 (NvDsTiler/nv_ds_tiler4:sink) link successful
2022-08-24 23:27:40.093 DEBUG extensions/nvdsbase/nvds_connection.cpp@46: nv_ds_static_output6 (NvDsTiler/nv_ds_tiler4:src) → nv_ds_static_input32…afd0 (NvDsOSD…3e48/nv_ds_osd31…5940:sink) link successful
2022-08-24 23:27:40.093 DEBUG extensions/nvdsbase/nvds_io.cpp@207: Setting ‘pad-added’ and ‘pad-removed’ signal handler on ‘NvDsMultiSrcInput/nv_ds_multi_src_input18’ for template ‘video_%u’
2022-08-24 23:27:40.094 DEBUG extensions/nvdsbase/nvds_scheduler.cpp@174: NvDsScheduler::runAsync_abi
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
gstnvtracker: Batch processing is ON
gstnvtracker: Past frame output is OFF
2022-08-24 23:27:43.784 INFO gxf/gxe/gxe.cpp@320: Deinitializing…
2022-08-24 23:27:43.785 INFO gxf/gxe/gxe.cpp@327: Destroying context
2022-08-24 23:27:43.785 INFO gxf/gxe/gxe.cpp@333: Context destroyed.

RunningJetson:
2022-08-24 23:06:37.029 INFO gxf/gxe/gxe.cpp@230: Creating context
2022-08-24 23:06:37.097 INFO gxf/gxe/gxe.cpp@291: Initializing…
2022-08-24 23:06:37.165 INFO extensions/nvdsbase/nvds_scheduler.cpp@265: This program is linked against GStreamer 1.16.2
2022-08-24 23:06:37.166 INFO extensions/nvdsmuxdemux/nvstreammux.hpp@27: initialize: nvstreammux nv_ds_stream_mux7…f8d0
2022-08-24 23:06:37.166 INFO extensions/nvdstracker/nvtrackerbin.hpp@23: initialize: nvtrackerbin nv_ds_tracker15…f208
2022-08-24 23:06:37.167 INFO extensions/nvdsvisualization/nvosdbin.hpp@24: initialize: nvosdbin nv_ds_osd31…5940
2022-08-24 23:06:37.184 INFO extensions/nvdsinference/nvinferbin.hpp@24: initialize: nvinferbin nv_ds_infer_video0
2022-08-24 23:06:37.185 INFO extensions/nvdsvisualization/nvtilerbin.hpp@38: initialize: nvtilerbin nv_ds_tiler4
2022-08-24 23:06:37.186 INFO extensions/nvdsoutputsink/nvvideoencfilesinkbin.hpp@23: initialize: nvvideoencfilesinkbin nv_ds_file_out0
2022-08-24 23:06:37.187 INFO gxf/gxe/gxe.cpp@298: Running…
2022-08-24 23:06:37.187 DEBUG extensions/nvdsbase/nvds_scheduler.cpp@102: NvDsScheduler::runAsync_abi
2022-08-24 23:06:37.187 INFO extensions/nvdsbase/nvds_scheduler.cpp@116: Scheduling 7 elements and 3 components
2022-08-24 23:06:37.187 INFO extensions/nvdsmuxdemux/nvstreammux.hpp@37: create_element: nvstreammux nv_ds_stream_mux7…f8d0

2022-08-24 23:06:37.187 ERROR extensions/nvdsmuxdemux/nvstreammux.hpp@41: Could not create GStreamer element ‘nvstreammux’
2022-08-24 23:06:37.187 ERROR gxf/std/program.cpp@337: Couldn’t run async. Deactivating…
2022-08-24 23:06:37.190 ERROR gxf/core/runtime.cpp@1157: Graph run failed with error: GXF_FAILURE
2022-08-24 23:06:37.190 ERROR gxf/gxe/gxe.cpp@301: GxfGraphRunAsync Error: GXF_FAILURE

Ubuntu: (Better command for Deepstream and GXF config?)
±----------------------------------------------------------------------------+
| NVIDIA-SMI 510.85.02 Driver Version: 510.85.02 CUDA Version: 11.6 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce … On | 00000000:07:00.0 On | N/A |
| 0% 35C P8 29W / 370W | 202MiB / 12288MiB | 0% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+

Jetson:

  • NVIDIA Jetson Xavier NX (Developer Kit Version)
    • Jetpack 4.6 [L4T 32.6.1]
    • NV Power Mode: MODE_20W_6CORE - Type: 8
    • jetson_stats.service: active
  • Board info:
    • Type: Xavier NX (Developer Kit Version)
    • SOC Family: tegra194 - ID:25
    • Module: P3668 - Board: P3509-000
    • Code Name: jakku
    • CUDA GPU architecture (ARCH_BIN): 7.2
    • Serial Number: 1424420038627
  • Libraries:
    • CUDA: 10.2.300
    • cuDNN: 8.2.1.32
    • TensorRT: 8.0.1.6
    • Visionworks: 1.6.0.501
    • OpenCV: 4.1.1 compiled CUDA: NO
    • VPI: ii libnvvpi1 1.1.12 arm64 NVIDIA Vision Programming Interface library
    • Vulkan: 1.2.70
  • jetson-stats:
    • Version 3.1.2
    • Works on Python 3.6.9
3

Seems extension not synced well. you can run registry repo list [args] to check repositories after repo sync for extension. Registry Command Line Interface — DeepStream 6.0 Release documentation

4

Ok, so run the registry sync inside the container? I thought the container builder was supposed to create a container with the needed extensions? They are well synced and working on the dGPU machine.

5

@yingliu Why did you remove graph composer from this thread? It’s a container built by graph composer.

6

Thanks for reminding, I had thought it is a pure container issue. tag added.

7

Sorry for the late.
If the docker container build include graph coposer installation, you can run above command.
Yes. it synced the extensions, but from the log, seems it did not synced well.
Which Jetpack version you are using?

8

5.0.2

9

The graph you used in based on which graph composer version?

10

2.0 but I am now working on getting 2.0.1 working with 6.1.1.

Getting different errors and working through them.

11

You get it running, right?

12

Not yet, still working on it…

13

Can you run success with original graph like test1 graph with DS 6.1.1 and graph composer 2.0,1?

14

We have not had a consistent enough run on the actual Jetson to bother trying a container yet under 6.1.1… We have something going on that takes out network name resolution which so far the only fix has been to reimage, which of course takes a significant amount of time…

Hopefully next week I will have a second NVMe drive so that we can keep a copy of the file system so we don’t have to go and do the whole reimage process every time that we take out the stack.

15

Hello @Nerdx86 Do you still need support for this topic? Or should we close it? Thanks.

16

Still waiting on getting Jetpack 5.0.2 stable enough to try to test a container… I have reloaded it about 10 or 15 times in the last week. (Keep losing network name resolution, and am getting very close to certainty that is not Anything on our side)

So, yes please leave this open until we are able to actually run containers again to finish testing it.

17

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

OK, let us know if you have more update.

19

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.