Does Nsight Operator supports IPv6 EKS cluster?

Developer Tools Nsight Cloud
1

I am trying to use the NVIDIA Nsight Operator to enable GPU profiling for workloads running in our EKS clusters.

This works correctly on our IPv4 clusters. We enable profiling by adding the following label to the Argo Rollout resource:

nvidia-nsight-profile: enabled

However, the same workflow does not work on an IPv6-only, single-stack EKS cluster.

The Nsight tenant pods enter CrashLoopBackOff, including:

nsight-operator-analysis
nsight-operator-cloud-ui
nsight-operator-tenant-operator

In addition, the workload’s wait-for-dependencies init step reports that the coordinator is “not reachable” and eventually times out after 300 seconds. As a result, profiling never starts.

From what I can tell, the root cause appears to be that the Nsight tenant components only listen on IPv4 and do not respond on the cluster’s IPv6 addresses. This causes their health probes to fail, which leads to repeated pod restarts, and the profiling agent in the workload is unable to reach the coordinator.

I have not found a documented way to customize this behavior or configure the Nsight components to listen on IPv6.

Is there a supported method or workaround to make the NVIDIA Nsight Operator GPU profiling workflow work on an IPv6-only, single-stack Kubernetes cluster?

Any guidance would be appreciated. Thanks.

2

@mpopov can you comment

3

@davidl3 , Nsight Operator 26.3.1 was released. It adds support for IPv6 and dual-stack clusters.

See the Release Notes.

Thanks,

Mike

4

Nice, thanks for the update

5

Hi @mpopov , I upgraded 2 EKS clusters from 26.2.1 to 26.3.1. It works for the ipv6 cluster now. But I found a huge increase of CPU and memory usage of the nsight-operator pod. I am not sure whether it’s expected. Can you help to check? Thanks.

AI told me that the nsight-operator manager pod’s memory is dominated by its cluster-wide informer caches — it holds an in-memory copy of every Secret, ConfigMap, Service, Deployment, PVC, Role, RoleBinding, and ServiceAccount across all namespaces in the cluster.

In the ipv4 cluster with more resources, below is the difference before and after the upgrade:

nsight-operator-86558b88c-7xnbk (pre-upgrade, 26.2.1) : CPU usage(~1.5-3m), Memory usage(~156-160Mi)

nsight-operator-78c88744f5-d6sz6 (post-upgrade, 26.3.1): CPU usage(~1000-1160m), Memory usage(~800-880Mi)

In the ipv6 cluster with less resources, below is the difference before and after the upgrade:

nsight-operator-55499bbf5b-fh4rf (pre-upgrade) CPU usage(~1m), Memory usage(~29-34Mi)

nsight-operator-6b8768b49b-qnvg2 (post-upgrade) CPU usage(~265-274m), Memory usage(~64-66Mi)

6

Hi @davidl3 , thank you for the detailed information and metrics, and for confirming that the Operator works on the IPv6 cluster.

Such a significant increase in CPU and memory usage after the 26.3.1 upgrade is not expected. We will investigate this further and update this thread once we have more information.

7

Hi @mpopov , is there any update for this topic? Are you able to recreate similar issue? Thanks.

8

@mpopov is actually out of office for a bit, so I am going to loop in @mhallock

9

Hi @davidl3,

We’re still working on it. I will provide more of an update next week.

10

Hi @mpopov @mhallock , is there any update on this? Are you able to recreate the same issue? Thanks.

11

@davidl3 , yes, we were able to reproduce the increased CPU and memory usage. We have several optimizations planned for the next release to address this. Thank you for reporting the issue!