Two power supplies and ethernet port died

PUSU3 and PSU4 on our dgx-1 failed simultaneously, as did both ethernet ports.
But management port still works and all sensor readings are green other than PSU3 and PSU4 (0 Watts)

We verified that our power outlets are working, reseated power plugs in DGX-1, swapped power supplies in PSU bays 1 and 2 with power supplies in bays 3 and 4. Result: power supplies that did not work in bays 3 and 4 did work in bays 1 and 2. And power supplies that formerly worked in bays 1 and 2 do NOT work in bays 3 and 4. Conclusion: all 4 power supply units are actually OK and problem lies elsewhere,

We lost network link to the dgx-1 at the same moment that the 2 PSUs failed and there are no lights on the ethernet ports. (We do see lights and have connectivity to the management ethernet port.)

A little surprised machine is running at all, as my understanding is that it is a “3 + 1” PSU setup meaning only 1 redundant PSU.

Any suggestions on how to revive the network and/or power supplies ?

You’ve correct that the DGX-1 is 3+1 redundant, but it still tries its best to start with >1 failure. Very weird that the 10GbE ports died at the same time, as those are off on a mezzanine card not really touching the PSUs.

This sounds like a perfect thing for NVIDIA Enterprise Support to solve @user158525 . :-)

Unfortunately enterprise support wont help because the machine went end of service life in September :(

Oh, shucks.

Seems like you need at least one more PSU to get back to a normal power state. Since it’s out of support and end-of-sale, we don’t have any PSUs to sell. The PSU should be a Quanta 1HY9ZZZ076Y , but I’m not sure those are available from anywhere. There’s always buying a used DGX-1 from somewhere, and using the PSUs and Ethernet mezzanine card from it. :-)

I’d get the PSUs whole before seeing what’s wrong with the Mezzanine card, but I’m not entirely sure what the 10GBaseT part number is. We had an SFP+ version which was a Quanta 1HY9ZZZ024N. The 10GBaseT version that shipped with the system may have been an https://www.qct.io/product/index/Storage/Storage-Accessory/OCP-Network-Mezzanine-For-Storage/ON-10GbE-X540-Storage#specifications but I’m not entirely sure of that).

Maybe this is the universe asking you to get a shiny new DGX A100? (Joking!)

Scott, thanks for your responses. But since the two seemingly dead PSUs started working again when moved to the PSU “sockets” formerly used by the two working PSUs, I’m skeptical that the issue would be resolved by buying replacement PSUs.

What might cause both PSU3 and PSU4 “socket” to fail at same time ? Is there some way to remotely test the functionality of the Mezzanine card, eg by attempting a firmware update ?

Aha! All the 4x PSUs are connected to a single PDB (Power Distribution Board) in the system. Maybe your failure is that the PDB has some failures (like, the slots where PSU 3 & 4 are)? the answer in that case would then be to replace the PDB. This is something the NVIDIA field support teams would do for systems under support, so it’s non-trivial, but possible.

It begs the question of “Where can I buy a PDB?”…unfortunately I think the answer would end up as “find a cheap for-parts DGX and get it from there”. :-(

Network card part number appears to be quanta-das2spc18b0 based on photo here: https://docs.nvidia.com/dgx/dgx1-user-guide/maintenance.html#task_replacing-10g-mezzanine-nic

If there is some damage to network card socket, is there any other way to add network connectivity to our dgx1, perhaps through usb-ethernet adapter ?

Oh, good find on the user guide! I totally forgot we had a picture of the underside of the card there!

Are you actively using all of the 4x NICs on the rear of the system? Those are dual-personality, and can be either InfiniBand or Ethernet, so depending on your setup you could use one of them to connect to your network. In theory USB-Ethernet would work, but performance wouldn’t be ideal - I’m sure fine for just SSH’ing in, but you wouldn’t want to access storage or other high-speed stuff that way.

Thanks Scott … Interesting about the InfiniBand ports, so those 4 connectors are standard SFP ? I didnt realize they could be converted to Ethernet, but I see procedure is described here https://docs.nvidia.com/dgx/dgx1-user-guide/configuring-managing-dgx1.html#infiniband-port-changing

One other thing, do you happen to know part # for the PDB (Power Distribution Board) ?

Those connectors are QSFP28, but you can turn them into SFP+ with something like an MAM1Q00A-QSA28 adapter.

I don’t know the PDB part number - we’ve only offered that as an NVIDIA RMA-able part (with an NVIDIA part numbers). I’m not sure if Quanta ever sold the PDB themselves as a standalone component. I reconfirmed with our support team that we don’t have the ability to sell DGX-1 components directly (only use them as part of an existing support contract).

Sorry. :-( The DGX-1 is still a really cool and useful system still, I feel bad that yours is out of support and having issues!