Completely purge and reinstall nvidia gpu operator

[Morganh]

@ganinduN

I think in the dgx I need to use the toolkit for the changes to /etc/containerd/config.toml to take effect, because I seem to not be able to run gpu jobs even with the gpu-operator installed without the --set toolkit.enabled=false

So I gess we need to fix the nvidia-operator-validator-*** issue. even re installng seem not to fix things now. I’m not sure if this was caused by an update or I’ve missed some critical bit.

Can you please try with a DGX station A100 (where the driver and toolkit is already installed) instead of A40s? at this all I’m doing is installing the cluster , calico, add the dgx node and try to install the gpu-operator. (which I;ve done quite a few times without an issue)

Pretty mich I’m doing this (my notes) simplified k8 install instructions | DL Docs (should be straightforward to follow and replicate)

Hi Ganindu,
For the issue when reinstalling gpu-operator in DGX station A100, could you please create a topic in DGX User Forum - NVIDIA Developer Forums or use About the DGX User Forum / Note: this is not NVIDIA Enterprise Support to generate a ticket for enterprise support? Since the issue is related to the software stack in DGX, and also is out of the scope of TAO, so it is better for you to get help from DGX team directly for better hints. More, you can also search topics or create topics on Issues · NVIDIA/gpu-operator · GitHub.