Could not receive message from the DeepStreamSDK Module on Azure IOT Edge (sink type =6)

• Hardware Platform (Jetson Nano 4G)
• DeepStream Version Deepstream 5.1 Azure Container Registry | Microsoft Azure
• JetPack Version 4.5.1
• TensorRT Version 7.1.3
• Issue Type( questions)
• Azure Iot Agent Version: azureiotedge-agent:1.2.10
• Azure Iot Hub Version: azureiotedge-hub:1.2.10

Hi,
I am trying to route the output from the DeepStreamSDK 5.1 module on in the azure iot edge (imported from the microsoft Marketplace ) to the cloud or another module, but I failed.

I found that the logs in DS module is :

Message sent : {
  "version" : "4.0",
  "id" : 339,
  "@timestamp" : "2022-08-29T20:44:36.044Z",
  "sensorId" : "HWY_20_AND_LOCUST__WBA__4_11_2018_4_59_59_379_AM_UTC-07_00",
  "objects" : [
    "17|470.4|468.75|518.4|611.25|Person",
    "98|64|457.5|188.8|791.25|Person",
    "102|684.8|483.75|844.8|603.75|Car",
    "106|393.6|472.5|425.6|566.25|Person",
    "132|569.6|472.5|604.8|498.75|Car",
    "133|614.4|487.5|688|540|Car",
    "134|492.8|457.5|534.4|487.5|Car",
    "139|976|465|1011.2|487.5|Car",
    "140|1017.6|468.75|1049.6|483.75|Car",
    "141|553.6|472.5|588.8|495|Car",
    "142|1187.2|487.5|1379.2|566.25|Car",
    "144|438.4|472.5|467.2|577.5|Person"
  ]
}
....

and a lot of error :

send_async_callback : send async failed with error code 1
send_async_callback : send async failed with error code 1
send_async_callback : send async failed with error code 1

At the same time, I found no output in cloud and the ModulePower module in which I handle the message using the following code

        if message.input_name == "light" or message.input_name == "led" or message.input_name == "environment":
            await telemetryHandler(message)
        elif message.input_name == "DS51":
            logging.warning(f"Received DS51 msg {message}")
        else:
            logging.info(f"Unhandled message input name {message.input_name}")

By the way, Here are my configurations and other environments

  • Iot edge version:
DS51             running          Up 20 seconds    marketplace.azurecr.io/nvidia/deepstream51-l4t:latest
edgeAgent        running          Up 3 minutes     mcr.microsoft.com/azureiotedge-agent:1.2.10
edgeHub          running          Up 2 minutes     mcr.microsoft.com/azureiotedge-hub:1.2.10
......
  • cat test5_config_file_src_infer_azure_iotedge_edited.txt

[sink1]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=UDPSink 5=nvoverlaysink 6=MsgConvBroker
type=6
msg-conv-config=dstest5_msgconv_sample_config.txt
#(0): PAYLOAD_DEEPSTREAM - Deepstream schema payload
#(1): PAYLOAD_DEEPSTREAM_MINIMAL - Deepstream schema payload minimal
#(256): PAYLOAD_RESERVED - Reserved type
#(257): PAYLOAD_CUSTOM   - Custom schema payload
msg-conv-payload-type=1
msg-broker-proto-lib=/opt/nvidia/deepstream/deepstream-5.1/lib/libnvds_azure_edge_proto.so
topic=DS51
#Optional:
#msg-broker-config=../../../../libs/azure_protocol_adaptor/module_client/cfg_azure.txt
#msg-broker-config=cfg_azure.txt

  • DeepStream config in the deploy template json
          "DS51": {
            "version": "1.5-20220829",
            "type": "docker",
            "status": "running",
            "restartPolicy": "always",
            "settings": {
              "image": "marketplace.azurecr.io/nvidia/deepstream51-l4t:latest",
              "createOptions": {
                "ExposedPorts": {
                  "8554/tcp": {}
                },
                "HostConfig": {
                  "PortBindings": {
                    "8554/tcp": [
                      {
                        "HostPort": "8554"
                      }
                    ]
                  },
                  "Binds": [
                    "/etc/aziot/:/etc/aziot/",
                    "/var/deepstream/custom_configs/:/opt/nvidia/deepstream/deepstream-5.1/sources/apps/sample_apps/deepstream-test5/custom_configs/"
                  ],
                  "runtime": "nvidia"
                },
                "WorkingDir": "/opt/nvidia/deepstream/deepstream-5.1/sources/apps/sample_apps/deepstream-test5/custom_configs/",
                "Entrypoint": [
                  "/usr/bin/deepstream-test5-app",
                  "-c",
                  "test5_config_file_src_infer_azure_iotedge_edited.txt"
                ]
              }
            }
          }
  • Route config:
            "DeepstreamToIoTHub": "FROM /messages/modules/DS51/* INTO $upstream",
            "DS51ToModulePower": "FROM /messages/modules/DS51/* INTO BrokeredEndpoint(\"/modules/ModulePower/inputs/DS51\")",
            "PowerTelemetry": "FROM /messages/modules/ModulePower/outputs/cloudtelemetry INTO $upstream"

When I tested a azure module with example /opt/nvidia/deepstream/deepstream-5.1/sources/libs/azure_protocol_adaptor/module_client/test_azure_iotedge_sync

I got the following error logs

In sample prog: connect success 
nvds_msgapi_connect : connect success
main: after connect
Error: Time:Tue Aug 30 20:54:30 2022 File:/home/nvidia/azure/azure-iot-sdk-c/provisioning_client/adapters/hsm_client_http_edge.c Func:on_edge_hsm_http_recv Line:364 executing HTTP request fails, status=400, response_buffer={"message":"missing request body"}nten
Error: Time:Tue Aug 30 20:54:30 2022 File:/home/nvidia/azure/azure-iot-sdk-c/provisioning_client/adapters/hsm_client_http_edge.c Func:send_http_workload_request Line:483 send_request_to_edge_workload failed
Error: Time:Tue Aug 30 20:54:30 2022 File:/home/nvidia/azure/azure-iot-sdk-c/provisioning_client/adapters/hsm_client_http_edge.c Func:hsm_client_http_edge_sign_data Line:576 send_http_workload_request failed
Error: Time:Tue Aug 30 20:54:30 2022 File:/home/nvidia/azure/azure-iot-sdk-c/provisioning_client/src/iothub_auth_client.c Func:iothub_device_auth_generate_credentials Line:339 Failure generate hash from tpm.
Error: Time:Tue Aug 30 20:54:30 2022 File:/home/nvidia/azure/azure-iot-sdk-c/iothub_client/src/iothub_client_authorization.c Func:IoTHubClient_Auth_Get_SasToken Line:348 failure getting credentials from device auth module
Error: Time:Tue Aug 30 20:54:30 2022 File:/home/nvidia/azure/azure-iot-sdk-c/iothub_client/src/iothubtransport_mqtt_common.c Func:SendMqttConnectMsg Line:1944 failure getting sas token from IoTHubClient_Auth_Get_SasToken.
Error: Time:Tue Aug 30 20:54:30 2022 File:/home/nvidia/azure/azure-iot-sdk-c/provisioning_client/adapters/hsm_client_http_edge.c Func:on_edge_hsm_http_recv Line:364 executing HTTP request fails, status=400, response_buffer={"message":"missing request body"}nten
Error: Time:Tue Aug 30 20:54:30 2022 File:/home/nvidia/azure/azure-iot-sdk-c/provisioning_client/adapters/hsm_client_http_edge.c Func:send_http_workload_request Line:483 send_request_to_edge_workload failed
Error: Time:Tue Aug 30 20:54:30 2022 File:/home/nvidia/azure/azure-iot-sdk-c/provisioning_client/adapters/hsm_client_http_edge.c Func:hsm_client_http_edge_sign_data Line:576 send_http_workload_request failed
Error: Time:Tue Aug 30 20:54:30 2022 File:/home/nvidia/azure/azure-iot-sdk-c/provisioning_client/src/iothub_auth_client.c Func:iothub_device_auth_generate_credentials Line:339 Failure generate hash from tpm.
Error: Time:Tue Aug 30 20:54:30 2022 File:/home/nvidia/azure/azure-iot-sdk-c/iothub_client/src/iothub_client_authorization.c Func:IoTHubClient_Auth_Get_SasToken Line:348 failure getting credentials from device auth module
Error: Time:Tue Aug 30 20:54:30 2022 File:/home/nvidia/azure/azure-iot-sdk-c/iothub_client/src/iothubtransport_mqtt_common.c Func:SendMqttConnectMsg Line:1944 failure getting sas token from IoTHubClient_Auth_Get_SasToken.
nvds_msgapi_send: Exceeds max wait time to receive send confirmation. Discarding message
Thread [8] , Message num 0 : send failed
nvds_msgapi_send: Exceeds max wait time to receive send confirmation. Discarding message
Thread [10] , Message num 0 : send failed
nvds_msgapi_send: Exceeds max wait time to receive send confirmation. Discarding message
Thread [7] , Message num 0 : send failed
nvds_msgapi_send: Exceeds max wait time to receive send confirmation. Discarding message
Thread [9] , Message num 0 : send failed
nvds_msgapi_send: Exceeds max wait time to receive send confirmation. Discarding message
Thread [6] , Message num 0 : send failed
Error: Time:Tue Aug 30 20:54:32 2022 File:/home/nvidia/azure/azure-iot-sdk-c/provisioning_client/adapters/hsm_client_http_edge.c Func:on_edge_hsm_http_recv Line:364 executing HTTP request fails, status=400, response_buffer={"message":"missing request body"}nten
Error: Time:Tue Aug 30 20:54:32 2022 File:/home/nvidia/azure/azure-iot-sdk-c/provisioning_client/adapters/hsm_client_http_edge.c Func:send_http_workload_request Line:483 send_request_to_edge_workload failed
Error: Time:Tue Aug 30 20:54:32 2022 File:/home/nvidia/azure/azure-iot-sdk-c/provisioning_client/adapters/hsm_client_http_edge.c Func:hsm_client_http_edge_sign_data Line:576 send_http_workload_request failed
Error: Time:Tue Aug 30 20:54:32 2022 File:/home/nvidia/azure/azure-iot-sdk-c/provisioning_client/src/iothub_auth_client.c Func:iothub_device_auth_generate_credentials Line:339 Failure generate hash from tpm.
Error: Time:Tue Aug 30 20:54:32 2022 File:/home/nvidia/azure/azure-iot-sdk-c/iothub_client/src/iothub_client_authorization.c Func:IoTHubClient_Auth_Get_SasToken Line:348 failure getting credentials from device auth module
Error: Time:Tue Aug 30 20:54:32 2022 File:/home/nvidia/azure/azure-iot-sdk-c/iothub_client/src/iothubtransport_mqtt_common.c Func:SendMqttConnectMsg Line:1944 failure getting sas token from IoTHubClient_Auth_Get_SasToken.
nvds_msgapi_send: Exceeds max wait time to receive send confirmation. Discarding message
Thread [7] , Message num 1 : send failed
nvds_msgapi_send: Exceeds max wait time to receive send confirmation. Discarding message

Did you use device connection string or module connection string?

//For azure module client - the edge device connection details should be mentioned in /etc/iotedge/config.yaml
NvDsMsgApiHandle ah = nvds_msgapi_connect_ptr(NULL, connect_cb, (char *)"/root/cfg_azure.txt");

Hi @Amycao
Should I create a new /etc/iotedge/config.yaml file ?
I didn’t found /etc/iotedge/config.yaml on the host environment. Microsoft now is supposed to use /etc/aziot/config.toml to store the connection string.

I even looked into the docker container of the iot edge . I found the edge module only get connection information from OS envrionment variables which is passed from the host OS.

Actually, I have ever mounted /etc/aziot/config.toml on host to the docker container, but it also didn’t work. I’m wondering if I’m experiencing compatibility issues between the latest host azure IoT edge environments and our azure edge adapter (msg-broker-proto-lib=/opt/nvidia/deepstream/deepstream-5.1/lib/libnvds_azure_edge_proto.so).

Did you ever run success with older azure IoT edge and ds azure edge adapter?

I never worked on older versions of agents .
we always expect the latest version of agent due to the cloud system.

Did you build libiothub_client.so manually?

This issue was sovled when I installed the DeepStream 6.0.1

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.