Nvidia Deepstream SDK module fails to run on Azure IoT Edge

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
Jetson Nano Develpor Kit
• DeepStream Version
DeepStream 6.0.1
• JetPack Version (valid for Jetson only)
JetPack 4.6.3
• TensorRT Version
TensorRT 8.2
• NVIDIA GPU Driver Version (valid for GPU only)
N/A
• Issue Type( questions, new requirements, bugs)
Bugs
• How to reproduce the issue ? (This is for bugs. Including which sample app is used, the configuration files content, the command line used and other details for reproducing)

Hey guys, I am Joe, a newcomer to the Jetson developer community. Currently, I am following the wiki of NVIDIA Deepstream + Azure IoT Edge on an NVIDIA Jetson Nano provided by Azure to achieve real-time streaming analysis using my new Jetson Nano Developer Kit:

Following the guide, I flashed my Jetson Nano with JetPack 4.6.3 and Deepstream 6.0.1, installed Azure IoT Edge and its runtime 1.4.9, and registered and hosted my device on Azure IoT Edge. Then I created and deployed an Azure IoT Edge solution with the NVIDIA DeepStream SDK module from Azure Marketplace using Visual Code Studio.

However, after I deployed and started to monitor the streaming event and data flow, the NVIDIA DeepStream SDK module on my Azure IoT Edge logger reported an error message as below:

(Argus) Error FileOperationFailed: Connecting to nvargus-daemon failed: No such file or directory (in src/rpc/socket/client/SocketClientDispatch.cpp, function openSocketConnection(), line 205)
(Argus) Error FileOperationFailed: Cannot create camera provider (in src/rpc/socket/client/SocketClientDispatch.cpp, function createCameraProvider(), line 106)

(gst-plugin-scanner:7): GStreamer-WARNING **: 04:32:29.808: Failed to load plugin ‘/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_osd.so’: /usr/lib/aarch64-linux-gnu/libnvinfer.so.7: file too short

(gst-plugin-scanner:7): GStreamer-WARNING **: 04:32:29.906: Failed to load plugin ‘/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_infer.so’: /usr/lib/aarch64-linux-gnu/libnvparsers.so.7: file too short

(gst-plugin-scanner:7): GStreamer-WARNING **: 04:32:30.032: Failed to load plugin ‘/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_inferserver.so’: /usr/lib/aarch64-linux-gnu/libnvinfer.so.7: file too short

(gst-plugin-scanner:7): GStreamer-WARNING **: 04:32:30.123: Failed to load plugin ‘/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_inferaudio.so’: /usr/lib/aarch64-linux-gnu/libnvparsers.so.7: file too short
** ERROR: <create_osd_bin:58>: Failed to create ‘nvosd0’
** ERROR: <create_osd_bin:100>: create_osd_bin failed
** ERROR: <create_processing_instance:802>: create_processing_instance failed
** ERROR: <create_pipeline:1294>: create_pipeline failed
** ERROR: main:1419: Failed to create pipeline
Quitting

I suspect that the Deepstream module is not working properly. Since it runs the sample test 5 based on the default configuration file, I log into the Jetson Nano device and locally tested the sample test-5 provided on path /opt/nvidia/deepstream/deepstream-6.0/sources/apps/sample_apps/deepstream-test5, using command deepstream-test5-app -c configs/test5_config_file_src_infer_azure_iotedge.txt

The test failed and reported an error message as below:

(deepstream-test5-app:13756): GLib-CRITICAL **: 20:42:27.414: g_strrstr: assertion ‘haystack != NULL’ failed
Error: Time:Sun Apr 16 20:42:27 2023 File:/home/nvidia/azure/azure-iot-sdk-c/iothub_client/src/iothub_client_core_ll.c Func:retrieve_edge_environment_variabes Line:177 Environment IOTEDGE_AUTHSCHEME not set
Error: Time:Sun Apr 16 20:42:27 2023 File:/home/nvidia/azure/azure-iot-sdk-c/iothub_client/src/iothub_client_core_ll.c Func:IoTHubClientCore_LL_CreateFromEnvironment Line:1186 retrieve_edge_environment_variabes failed
Error: Time:Sun Apr 16 20:42:27 2023 File:/home/nvidia/azure/azure-iot-sdk-c/iothub_client/src/iothub_client_core.c Func:create_iothub_instance Line:924 Failure creating iothub handle
ERROR: iotHubModuleClientHandle is NULL! connect failed
** ERROR: main:1455: Failed to set pipeline to PAUSED
Quitting
ERROR from sink_sub_bin_sink2: Could not configure supporting library.
Debug info: /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvmsgbroker/gstnvmsgbroker.cpp(402): legacy_gst_nvmsgbroker_start (): /GstPipeline:pipeline/GstBin:sink_sub_bin2/GstNvMsgBroker:sink_sub_bin_sink2:
unable to connect to broker library
ERROR from sink_sub_bin_sink2: GStreamer error: state change failed and some element failed to post a proper error message with the reason for the failure.
Debug info: gstbasesink.c(5265): gst_base_sink_change_state (): /GstPipeline:pipeline/GstBin:sink_sub_bin2/GstNvMsgBroker:sink_sub_bin_sink2:
Failed to start
App run failed

I have read quite a lot of the previous posts which describe similar issue as mine and tried to fix it by following a similar procedure, but no luck so far:

  1. Deepstream module fails with AZure IoTEdge

  2. IOTEDGE_AUTHSCHEME not set error running deepstream-test5-app

What did I do wrong here? Can anyone share their insights with me, please?

PS:
I have followed the README file specified at path /opt/nvidia/deepstream/deepstream-6.0/sources/libs/azure_protocol_adaptor/module_clients to set up proper dependencies, but I noticed one thing:

The latest Azure IoT Edge uses configuration files specified at path /etc/aziot/config.toml for connection string and other IoT device parameters, but the README file specified by Nvidia stated that Azure IoT edge configuration setting must be specified at path /etc/iotedge/config.yaml

When I was installing the Azure IoT edge and runtime on my Jetson Nano following the documentation by Azure:

The device configuration file (at path /etc/aziot/config.toml) is automatically created by Azure Iot Edge using the command sudo iotedge config mp --connection-string ‘PASTE_DEVICE_CONNECTION_STRING_HERE’

so I wonder if this is a version compatibility issue between the latest Azure IoT Edge 1.4.9 and Deepstream SDK 6.0.1. If so, where should I make the change so I can make it work?

Sincerely,
Joe

4

could you share the output of “gst-inspect-1.0 |grep nvdsosd”?

5

sure, Thank you for your helping hand!

The command “gst-inspect-1.0 | grep nvdsosd” produce the following:

nvdsgst_osd: nvdsosd: NvDsOsd plugin

I think I do have the GStreamer NvDsOsd plugin on my Jetson Nano.

6

from the error, it created nvdsosd element failed, can the simplest sample deepstream-test run successfully? it will also use nvdsosd.

7

Yes, I ran the deepstream-test1-app using the sample stream sample_720p.h264, and it was successful and smooth. The following are the commands I used to perform the test:

cd /opt/nvidia/deepstream/deepstream-6.0/sources/apps/sample_apps/deepstream-test1

deepstream-test1-app /opt/nvidia/deepstream/deepstream-6.0/samples/streams/sample_720p.h264

However, I have not figured out how to run deepstream-test5-app with the proper configuration yet. I think there is something wrong with setting up the sink in the configuration .txt file, like broker connection string, and host local Kafka so that I can test it locally.

Any idea?

8

please refer to this topic IOTEDGE_AUTHSCHEME not set error running deepstream-test5-app

9

Yes, I have mentioned the same topic in my original post, and I have followed the documentation provided there and set up the needed dependencies:

https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvmsgbroker.html#azure-mqtt-protocol-adapter-libraries

However, the issue persists.

Also, the Nvidia documentation (in the section Connection Details for the Module Client Adapter) mentioned that:

Leave the connection string empty, since the Azure IoT Edge library automatically fetches the connection string from the file /etc/iotedge/config.yaml.

As I have noticed on the latest Azure documentation and its runtime behavior on configuration file auto-generation, it no longer creates nor uses /etc/iotedge/config.yaml; instead, it auto-creates and uses /etc/aziot/config.toml, which is causing the confusion here, and highly likely the reason that causes bugs. I have mentioned it in the PS section of my post.

Again, I have set up both configuration files /etc/iotedge/config.yaml and /etc/aziot/config.toml with the proper connection string info of the Azure IoT device, but it does not work, and I am trying to figure out why.

I am pretty sure it is a bug that can reproduce. Is it possible for you to take a closer look at this issue? Do you need the actual content of my configuration files?

10

yes, please share the configuration file, and please use /opt/nvidia/deepstream/deepstream/sources/libs/azure_protocol_adaptor/module_client/ to debug this issue instead of test5.

11

Sure, I will be using /opt/nvidia/deepstream/deepstream/sources/libs/azure_protocol_adaptor/module_client/ to debug the issue for now and will report the result after testing.

The following are the configuration files I used specifically for Azure IoT Edge:

  1. This config.toml file is automatically generated by the command sudo iotedge config mp --connection-string ‘PASTE_DEVICE_CONNECTION_STRING_HERE’
    config.toml (886 Bytes)

  2. This config.yaml file is converted from the above config.tomal file using a toml to yaml auto file format converter.
    config.yaml (847 Bytes)

Please let me know if there are any incorrect configuration settings on these files, I am more than happy to correct them and test the test 5 app again.

12

Hi Fanzh, I have followed the README file from /opt/nvidia/deepstream/deepstream/sources/libs/azure_protocol_adaptor/module_client/ to deploy and test the test_async sample module, and sadly it failed as well, the following is the corresponding screenshots:

4
42060×938 48.6 KB

There are no connectivity and configuration issues based on the output of command iotedge check (The output content is quite long, so I did not post it). EdgeAgent and edgeHub are up and running, but test_async failed due to an unknown error on unable to open a shared library.

I followed the README file step by step and monitor each step’s output and did not find anything wrong. I can also find the shared library (.so) files of Azure IoT SDK lib (libiothub_client.so) and protocol bridge (libnvds_azure_edge_proto.so) available under the directory /opt/nvidia/deepstream/deepstream-6.0/sources/lib/ which mentioned in the README file, so I am not sure what exactly is going on here about the error.

Any idea about debugging this issue? Are you able to reproduce the bugs that I encounter so far?

13

please refer to opt\nvidia\deepstream\deepstream\sources\libs\azure_protocol_adaptor\module_client\README, as the doc shown, “This deepstream release ships with precompiled azure c sdk lib(v1.2.8)”, please use the corresponding Azure version.

14

Hey fanzh! I Got some good news here. I debugged and fixed the issue where the test_async sample module failed to run by fixing the /etc/iotedge/config.yaml file:

To make it work, we need to change the name of the connection string key from
connection_string: “YOUR_IOT_DEVICE_CONNECTION_STRING”
to
device_connection_string: “YOUR_IOT_DEVICE_CONNECTION_STRING”

After that, the test_async module ran successfully and smoothly even with Azure IoT Edge 1.4.9:

5
52336×1012 165 KB

I also tested the sample module (named deep_stream on my iotedge list) provided by Nvidia DeepStream SDK from Azure Marketplace. Although it failed (as expected), the current error message is slightly different from before. It no longer shows “Failed to create ‘nvosd0’” but shows “Failed to parse config file ‘test5_config_file_src_infer_azure_iotedge.txt’” instead:

6
62690×1162 223 KB

I am not sure why the parse config function failed to parse the file, and will look into that and try to figure it out. Do you have any idea?

15

thanks for your sharing, from the error, it is because of parsing configuration failed, you can compare with a working configuration file, or comment out some lines, then run again to check which line causes the error.

16

Okay, I am currently going through each property defined in test5_config_file_src_infer_azure_iotedge.txt and trying to figure out which one is causing the problem.

Honestly, I did not find anything suspicious, so I am kinda lost here. I feel like randomly commenting on some lines wouldn’t really help either. Would you mind taking a closer look at this file and see what would be a better move, please?

17

did you modify the configuration file? if yes, please share the new file.

18

No, I used the original configuration file and did not change anything from it, so I am not sure where it went wrong.

19

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

Sorry for the late reply, Is this still an DeepStream issue to support?

21

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.