IPC Connection Issue with DOCA Flow Inspector on BlueField2 DPU

Infrastructure & Networking BlueField
1

Hello,

I am currently deploying the DOCA Flow Inspector service on a BlueField2 DPU following the instructions in the official guide: DOCA Flow Inspector Service Guide.

After following the steps, the DOCA Flow Inspector pod and container are both running normally. Here are the outputs:

sudo crictl pods
POD ID              CREATED             STATE               NAME                                           NAMESPACE           ATTEMPT             RUNTIME
73df4e7a1602a       2 days ago          Ready               doca-flow-inspector-localhost.localdomain      default             0                   (default)
4ec138f3a9fbe       6 weeks ago         Ready               doca-telemetry-service-localhost.localdomain   default             0                   (default)

sudo crictl ps
CONTAINER           IMAGE               CREATED             STATE               NAME                     ATTEMPT             POD ID              POD
277d2e477afca       9093f339c4f1b       2 days ago          Running             doca-flow-inspector      0                   73df4e7a1602a       doca-flow-inspector-localhost.localdomain
fa1a005359039       0d17e540ea5b6       6 weeks ago         Running             doca-telemetry-service   0                   4ec138f3a9fbe       doca-telemetry-service-localhost.localdomain

However, I noticed that the output file from DPS doesn’t show any content from DOCA Flow Inspector. Upon checking the logs of the DOCA Flow Inspector container, I found the following warning:

2024-11-13 04:21:30 - flow_inspector - FI     - WARNING  - IPC connection status -1

It seems like there is an issue with the IPC connection between the DOCA Flow Inspector container and DTS.

Has anyone encountered this issue before? Any suggestions on how to resolve the IPC communication problem between the Flow Inspector and DTS?

Thank you!

2

Hi,

Could you please provide the exact versions involved so we can better triage the issue on our end?

  • Version of DOCA Flow Inspector - Listed in the YAML used for the contaienr deployment
  • Version of DOCA Telemetry Service (DTS) - Given that it runs for 6 weeks it is most probably the built-in image that is part of the BFB, so it should be covered by ‘crictl images’.
  • BFB Version - ‘cat /etc/mlnx-release’

Thanks.

4

Hi,

Thank you for your reply.

Here are the versions as requested:

  • DOCA Flow Inspector: 2.5.0 (specified in the YAML used for container deployment)
  • DOCA Telemetry Service (DTS): 2.8.0 (as listed by crictl images)
  • BFB Version: bf-bundle-2.8.0-98_24.07-ubuntu-22.04_prod (from cat /etc/mlnx-release)

Thanks.

5

Hi,

Thanks for the provided details. We have checked these versions and the deployment did succeed on our end.

Please note that the log indicated above is part of a periodic connection check, which means that if it shows only once/twice it might just be part of the initialization but later the connection is established successfully. In addition, it is important to verify that the /dev/shm/telemetry directory exists (as showed in Flow Inspector’s YAML file):

  name: shared-memory
  hostPath:
    path: /dev/shm/telemetry
    type: DirectoryOrCreate

On top of that, as mentioned in the DTS’s guide “Data write is disabled by default due to BlueField storage restriction”. For enabling the Data write, please follow the instructions from the relevant section in the guide.

6

Hi,

Thank you for your help.

After redeploying everything with the updated configurations, it worked successfully. The issue is now resolved.

Thanks again!