gi.repository.ElementFactory.make("nvstreammux", "Stream-muxer") failed

bridge · June 9, 2020, 10:34pm

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
• DeepStream Version
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)

Jetson TX2
DeepStream 5.0
JetPack 4.4
TensorRT 7.1.0.16

Application packages:
deepstream_python_v0.9.tbz2
deepstream_sdk_v5.0.0_jetson.tbz2

Running deepstream_imagedata-multistream.py got failed at creating nvstreammux instance.
Gst.ElementFactory.make(“nvstreammux”, “Stream-muxer”)

Creating streamux

 Unable to create NvStreamMux
Traceback (most recent call last):
  File "deepstream_imagedata-multistream.py", line 415, in <module>
    sys.exit(main(sys.argv))
  File "deepstream_imagedata-multistream.py", line 266, in main
    pipeline.add(streammux)
  File "/usr/lib/python3/dist-packages/gi/overrides/Gst.py", line 64, in add
    if not Gst.Bin.add(self, arg):
TypeError: Argument 1 does not allow None as a value

It seems gstreamer doesn’t have plugin nvstreammux. The question is how to install this plugin, I searched around didn’t figure out yet. I installed all the packages with JetPack 4.4 and SDKmanager 1.1, isn’t that supposed to install everything needed for deepstream 5.0 applications?

amycao · June 10, 2020, 6:04am

Can you find nvstreammux plugin?
gst-inspect-1.0 /usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_multistream.so
If you choose install deepstream by sdkmanager, it will automatically install the package needed.

bridge · June 10, 2020, 12:54pm

Thanks @amycao. I do install deepstream 5.0 by sdkmanager 1.1, the nvstreammux library is there, yet the date is not the installation date, it seems the date when the plugin was generated by nvidia?

tx2@tx2-desktop:~$ gst-inspect-1.0 /usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_multistream.so
cuInit failed : 100

tx2@tx2-desktop:~$ ll /usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_multistream.so
-rw-r–r-- 1 root root 558304 Apr 27 10:05 /usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_multistream.so

amycao · June 10, 2020, 1:51pm

tx2@tx2-desktop:~$ gst-inspect-1.0 /usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_multistream.so
cuInit failed : 100

can you run below cuda sample?
/usr/local/cuda/samples/1_Utilities/deviceQuery

bridge · June 10, 2020, 1:54pm

Just built it and ran it.

tx2@tx2-desktop:/usr/local/cuda/samples/1_Utilities/deviceQuery$ ./deviceQuery
./deviceQuery Starting…

CUDA Device Query (Runtime API) version (CUDART static linking)

cudaGetDeviceCount returned 100
-> no CUDA-capable device is detected
Result = FAIL

ll /usr/local/cuda lrwxrwxrwx 1 root root 9 Jun 9 15:31 /usr/local/cuda -> cuda-10.2/ ll /usr/local/cuda/
total 60
drwxr-xr-x 12 root root 4096 Jun 8 22:45 ./
drwxr-xr-x 11 root root 4096 Jun 9 15:31 …/
drwxr-xr-x 3 root root 4096 Jun 8 22:45 bin/
drwxr-xr-x 5 root root 4096 Jun 8 22:45 doc/
drwxr-xr-x 4 root root 4096 Jun 8 22:43 extras/
lrwxrwxrwx 1 root root 29 Oct 29 2019 include -> targets/aarch64-linux/include/
lrwxrwxrwx 1 root root 25 Oct 29 2019 lib64 -> targets/aarch64-linux/lib/
-rw-r–r-- 1 root root 366 Oct 29 2019 LICENSE
drwxr-xr-x 3 root root 4096 Jun 8 22:45 nvml/
drwxr-xr-x 7 root root 4096 Jun 8 22:43 nvvm/
drwxr-xr-x 7 root root 4096 Jun 8 22:43 nvvmx/
-rw-r–r-- 1 root root 366 Oct 29 2019 README
drwxr-xr-x 12 root root 4096 Jun 10 09:53 samples/
drwxr-xr-x 3 root root 4096 Jun 8 22:43 share/
drwxr-xr-x 3 root root 4096 Jun 8 22:43 targets/
drwxr-xr-x 2 root root 4096 Jun 8 22:45 tools/
-rw-r–r-- 1 root root 21 Oct 29 2019 version.txt

amycao · June 10, 2020, 2:37pm

Please check dmesg to see if have GPU error.

cudaErrorNoDevice = 100

This indicates that no CUDA-capable devices were detected by the installed CUDA driver.

bridge · June 10, 2020, 3:21pm

Nothing with ‘dmesg | grep -i gpu’, below is what I observed in /var/log/syslog/. It seems the gpu card was not initialized properly. This is TX2 development kit, it’s supposed to be supported by the kernel in JetPack4.4, right?

Jun 10 10:58:43 tx2-desktop nvpmodel[6090]: NVPM ERROR: Error opening /sys/devices/gpu.0/power/control: 2
Jun 10 10:58:43 tx2-desktop nvpmodel[6090]: NVPM ERROR: failed to write PARAM GPU_POWER_CONTROL_ENABLE: ARG GPU_PWR_CNTL_EN: PATH: /sys/devices/gpu.0/power/control VAL: on
Jun 10 10:58:43 tx2-desktop nvpmodel[6090]: NVPM ERROR: Error opening /sys/devices/17000000.gp10b/devfreq/17000000.gp10b/available_frequencies: 2
Jun 10 10:58:43 tx2-desktop nvpmodel[6090]: NVPM ERROR: failed to read PARAM GPU: ARG FREQ_TABLE: PATH /sys/devices/17000000.gp10b/devfreq/17000000.gp10b/available_frequencies
Jun 10 10:58:43 tx2-desktop nvpmodel[6090]: NVPM ERROR: Error opening /sys/devices/gpu.0/power/control: 2
Jun 10 10:58:43 tx2-desktop nvpmodel[6090]: NVPM ERROR: failed to write PARAM GPU_POWER_CONTROL_DISABLE: ARG GPU_PWR_CNTL_DIS: PATH: /sys/devices/gpu.0/power/control VAL: auto
Jun 10 10:58:43 tx2-desktop nvpmodel[6090]: NVPM ERROR: failed to set power mode!
Jun 10 10:58:43 tx2-desktop nvpmodel[6090]: NVPM ERROR: optMask is 2, no request for power mode
Jun 10 10:58:43 tx2-desktop NetworkManager[5682]: [1591801123.1146] manager: (l4tbr0): new Bridge device (/org/freedesktop/NetworkManager/Devices/4)
Jun 10 10:58:43 tx2-desktop NetworkManager[5682]: [1591801123.1173] manager: (rndis0): new Ethernet device (/org/freedesktop/NetworkManager/Devices/5)
Jun 10 10:58:43 tx2-desktop NetworkManager[5682]: [1591801123.1196] manager: (usb0): new Ethernet device (/org/freedesktop/NetworkManager/Devices/6)
Jun 10 10:58:43 tx2-desktop NetworkManager[5682]: [1591801123.1341] devices added (path: /sys/devices/virtual/net/l4tbr0, iface: l4tbr0)
Jun 10 10:58:43 tx2-desktop NetworkManager[5682]: [1591801123.1342] device added (path: /sys/devices/virtual/net/l4tbr0, iface: l4tbr0): no ifupdown configuration found.
Jun 10 10:58:43 tx2-desktop NetworkManager[5682]: [1591801123.1404] devices added (path: /sys/devices/3550000.xudc/gadget/net/usb0, iface: usb0)
Jun 10 10:58:43 tx2-desktop NetworkManager[5682]: [1591801123.1404] device added (path: /sys/devices/3550000.xudc/gadget/net/usb0, iface: usb0): no ifupdown configuration found.
Jun 10 10:58:43 tx2-desktop NetworkManager[5682]: [1591801123.1413] devices added (path: /sys/devices/3550000.xudc/gadget/net/rndis0, iface: rndis0)
Jun 10 10:58:43 tx2-desktop NetworkManager[5682]: [1591801123.1414] device added (path: /sys/devices/3550000.xudc/gadget/net/rndis0, iface: rndis0): no ifupdown configuration found.
Jun 10 10:58:43 tx2-desktop NetworkManager[5682]: [1591801123.1490] modem-manager: ModemManager available
Jun 10 10:58:43 tx2-desktop systemd[1]: nvpmodel.service: Main process exited, code=exited, status=255/n/a
Jun 10 10:58:43 tx2-desktop systemd[1]: nvpmodel.service: Failed with result ‘exit-code’.
Jun 10 10:58:43 tx2-desktop systemd[1]: Failed to start nvpmodel service.

bridge · June 11, 2020, 1:15pm

I tried the same application on nano, it passed this step. The problem nano got is some segmentation fault.

Any reason why the GPU card could not be initialized properly?

tx2@tx2-desktop:~$ sudo jetson_clocks --show
SOC family:tegra186 Machine:quill
Online CPUs: 0,3-5
CPU Cluster Switching: Disabled
cpu0: Online=1 Governor=schedutil MinFreq=345600 MaxFreq=2035200 CurrentFreq=960000 IdleStates: C1=1 c7=1
cpu1: Online=0 Governor=schedutil MinFreq=345600 MaxFreq=2035200 CurrentFreq=2035200 IdleStates: C1=1 c6=1 c7=1
cpu2: Online=0 Governor=schedutil MinFreq=345600 MaxFreq=2035200 CurrentFreq=2035200 IdleStates: C1=1 c6=1 c7=1
cpu3: Online=1 Governor=schedutil MinFreq=345600 MaxFreq=2035200 CurrentFreq=960000 IdleStates: C1=1 c7=1
cpu4: Online=1 Governor=schedutil MinFreq=345600 MaxFreq=2035200 CurrentFreq=806400 IdleStates: C1=1 c7=1
cpu5: Online=1 Governor=schedutil MinFreq=345600 MaxFreq=2035200 CurrentFreq=960000 IdleStates: C1=1 c7=1
/usr/bin/jetson_clocks: line 243: /sys/class/devfreq/*/device/of_node/name: No such file or directory
> Error! Unknown GPU!

tx2@tx2-desktop:~$ tegrastats
RAM 348/7860MB (lfb 1361x4MB) CPU [1%@345,off,off,0%@345,0%@345,0%@345] EMC_FREQ 0% PLL@48C MCPU@48C PMIC@100C Tboard@46C GPU@47C BCPU@48C thermal@47.8C Tdiode@45.75C
RAM 348/7860MB (lfb 1361x4MB) CPU [2%@345,off,off,0%@345,0%@345,1%@345] EMC_FREQ 0% PLL@48C MCPU@48C PMIC@100C Tboard@46C GPU@47.5C BCPU@48C thermal@47.8C Tdiode@45.75C
RAM 348/7860MB (lfb 1361x4MB) CPU [1%@345,off,off,0%@345,0%@345,0%@345] EMC_FREQ 0% PLL@48C MCPU@48C PMIC@100C Tboard@46C GPU@47C BCPU@48C thermal@47.6C Tdiode@45.75C
RAM 348/7860MB (lfb 1361x4MB) CPU [0%@345,off,off,0%@345,0%@345,0%@345] EMC_FREQ 0% PLL@48C MCPU@48C PMIC@100C Tboard@46C GPU@47C BCPU@48C thermal@47.6C Tdiode@45.75C

bridge · June 11, 2020, 1:21pm

I think the root cause to this issue is that the GPU card is broken. I even tried flashing JetPack 4.2.3 to this TX2, it got the same error, failed to initialize the GPU card.

Jun 11 09:17:07 tx2-desktop /usr/lib/gdm3/gdm-x-session[17430]: (==) Automatically adding GPU devices
Jun 11 09:17:07 tx2-desktop /usr/lib/gdm3/gdm-x-session[17430]: (==) Automatically binding GPU devices
Jun 11 09:17:07 tx2-desktop /usr/lib/gdm3/gdm-x-session[17430]: (II) NVIDIA Unified Driver for all Supported NVIDIA GPUs
Jun 11 09:17:07 tx2-desktop /usr/lib/gdm3/gdm-x-session[17430]: (EE) NVIDIA(GPU-0): Failed to initialize the NVIDIA graphics device!

Don’t know how easy the TX2 dev kit may get broken.

amycao · June 23, 2020, 1:34am

Can you do one more test to try with Jetpack 3.3? it’s an earlier jetpack version. I flashed with this version, cuda sample work well, if your device still can not initialize GPU properly, then it should be GPU broken.