Low performance when training NN and power consumption bounded

Accelerated Computing CUDA CUDA Programming and Performance
1

I have a machine with 4 NVIDIA RTX A5000 GPUs. By checking the logs of previous users, this machine was able to train the same models 4x faster about a year ago.

When I train models now, I see that the power consumption stays always below 75W. By checking logs from previous users that trained about 1Y ago, this value was about 3x higher. Temperature stays always below 50C so it’s not overheating. Performance state is P2 while training and P8 while idle (switches automatically)

I also ran burn_gpu benchmark and obtained very poor results. Something is limiting my GPUs performance. Could it be a power issue? How can I diagnose it?

I checked via nvidia-smi -q -d POWER whether there is a power limit enforced and there is not. nvidia-smi -q -d PERFORMANCE does report a slowdown field:

    HW Slowdown                       : Active
        HW Thermal Slowdown           : Not Active
        HW Power Brake Slowdown       : Active

I really really appreciate any input.

2

It sounds power related, with approx. 75W being the limit able to be supplied by the PCIe bus.

Assuming all 4 cards are doing this and nobody has recently been inside the server, the PSU may be a prime suspect, as it appears there is inadequate power being supplied via the 8 pin connector each card has.

A voltmeter carefully used inside the rear of the connector, beside the wires, could be used to check.

3
4

The first thing to check is whether all cables with 8-pin connectors supplying auxiliary power to the GPUs are properly plugged in. Be sure to check at both, the GPU end and at the PSU end. No Y-splitters or 6-pin to 8-pin converters should be in use, and there should be no daisy chaining, i.e. there should be a dedicated cable running from each GPU to the PSU.

A single 8-pin PCIe connector is designed to supply 150W, while the PCIe slot itself can supply at most 75W, for a total of 225W, so with 230W design power, there is zero wiggle-room for the RTX A5000 when it comes to power supply. Deep learning workloads in particular tend to utilize the full power budget of a GPU.

A 1600W PSU mentioned in the text quoted in the post by Robert_Crovella should be adequate to power a system with four RTX A5000, but depending on other system components barely so. I would recommend using a PSU that is 80 PLUS Platinum compliant for best efficiency (reduced waste heat) and long-tern reliability.

PSUs and the semiconductors and passives in them are subject to significant thermal stress and are therefore often one of the system components that fail first.

5

The most likely cause of this in my opinion is that the motherboard is asserting the power brake input to the GPU. If that is the case, there is nothing you can do to fix it unless you take some sort of activity with the motherboard (get a different motherboard, discuss the issue with the motherboard vendor, tape off the relevant pin as discussed in the article I linked, etc.). The GPU and the GPU software/firmware cannot be overridden by an end-user to ignore this.

While we could possibly consider that a defective GPU might be involved (e.g. a broken or shorted trace on the GPU PCB), if the same report from nvidia-smi is present on all 4 GPUs, that seems systemic (i.e. motherboard related) and rules out the GPU unless you wish to entertain the idea that all 4 GPUs are defective in the same way.

If the aux power connector were unconnected or simply not delivering power/voltage, the GPU would complain about this in the vicinity of the boot cycle, either at boot time itself, or at the point of driver load. However double-checking the integrity of the aux power connector is generally a good idea as a best practice.

It’s also possible that there are different flavors of RTX A5000. For example, a PNY RTX A5000 (which is not necessarily shipped in a system) might be configured to ignore the power brake, whereas a RTX A5000 that originally shipped in an OEM workstation/server might be configured to monitor the power brake, since the OEM would know that that is their intent, and the system is designed to behave that way. I don’t know if this is the case or not, but its pretty much immaterial given that the system at hand displays the message.

If the RTX A5000 were purchased from a reputable vendor, such that the warranty is intact, then you could certainly discuss with the vendor where you acquired those, if it is not the system vendor. But the fact remains that the input is being declared as asserted, so the GPU is simply following the instructions is has received. The observation of low power draw and low performance is entirely expected behavior when the power brake is asserted or is being reported as asserted.

6

I have never seen power brake in action. My assumption is that this is a signal asserted by a “power good” circuit that measures whether there is adequate power supply. One method may involve detecting brown-outs / undervoltage.

(Temporarily) insufficient power supply could occur due to installing too many electrical consumers in a system, incorrect or sub-optimal cabling (I enumerated some of the possibilities), damaged or dirty connectors, or, I imagine, an aging PSU that is on its last legs.

The interesting part in OP’s description is the claim that the system used to work just fine in the present configuration. My #1 working hypothesis is that there were in fact changes to the system configuration of some kind or maybe #2 that the system is operating off a different electrical outlet than previously (maybe a 20A circuit vs a 15A circuit, assuming we are talking about 120V U.S. electrical supply).

7

my guess would also be changes in the system. I cannot explain any of this conclusively, I’m just offering opinions.

Power brake originated in server platforms as a desire to allow the server (perhaps in the context of a larger scope, i.e. a cluster) to deliver best possible performance when the total aggregate power available is less than the total aggregate power that the system (server or cluster, depending on how power delivery system is designed) could draw at peak demand.

The basic mechanism is that the the power delivery system (e.g. rack PDU or other cluster-level power indication) and/or server PSU would signal to the motherboard of one or more systems that the power was at or exceeding an aggregate level that the system could safely deliver. The system then asserts power brake, and the GPUs reduce their power draw, preventing sustained operation in an overload state.

The idea for the benefit, if we look at it at the single server level, is the possibility that I could design a system lets say with 4 GPUs, each of which can draw 300W, but I only have at most 1kW available for GPU power in my server. Knowing that GPUs don’t always run at their peak load, we could allow the GPUs to run normally until the aggregate demand was at or above 1kW, as determined by the PSU. The PSU then signals the motherboard (e.g. the server BMC), and the next step in the chain is that power brake is asserted. The GPUs reduce their power consumption.

You can imagine then that this scheme would allow two or possibly even 3 GPUs to run unconstrained, if the 4th was lightly loaded. Given that in a datacenter the loading across processors is not always equal, this is considered a useful system design strategy, in a certain light. Of course you could just prefer a system that did not do this and delivered 1.2kW to the GPUs. But sometimes server manufacturers make design tradeoffs that seem appropriate (to them, at least).

I don’t know if typical workstation platforms do this or not. But the RTX A5000 GPU is designed to be suitable for server applications as well as workstation applications (unlike, eg. a RTX A500 which would not have any typical server footprint, differentiating here between servers and rack workstations, but we are starting to get into the weeds now.)

And we should not conflate uncertainty in application description with uncertainty of diagnosis here. The fact remains that regardless of this discussion, the GPU is declaring that it believes the power brake input is asserted. And when the GPU believes that, it is completely normal for it to restrict clocks and power consumption. That part of the description here leaves little doubt, in my view.

Furthermore, without better description of the provenance of the system, I wouldn’t bet that this is anything other than a motherboard asserting a signal it should not (especially, for example, if power brake is asserted all the time, even when the GPUs are idle). I’m not suggesting there is an actual properly functioning power brake regime here, along the lines of the description I have given. Even if it were an actual properly designed power brake setup, and the system as a whole were behaving “as intended”, there isn’t anything that can be done at the GPU level. The issue must be clarified by working at (and understanding) the system level behavior.

8

That is entirely undisputed, I think.

I have done a bit of reading and it seems that the motherboards of today have a lot of complexity, using programmable logic devices and embedded microcontrollers (such as the baseboard management controller = BMC mentioned) to determine whether power brake should be asserted. Also, not all PCIe endpoints (loosely, PCIe slots) are necessarily subject to power brake assertion. In some systems the BMC seems to work off internal equipment databases (“power tables”), leading to power brake being asserted for PCIe devices that have not been validated by the system integrator for a particular platform and therefore have no corresponding table entry.

In consequence, and as suggested, any support requests in these cases should go to the system integrator who put the system together.

9

I understand near nothing about the power connections but here is a pic of the cables going into my GPUs, do you notice something odd? Should there be another connection that allows each GPU to draw >75W?

image
image540×720 59.2 KB

10

As far as I can see from the picture (only the GPU side of the cabling is completely visible), everything is good here.

(1) There is one cable with an 8-pin connector attached to each GPU. This is PCIe auxilliary power, where an 8-pin connector is specified to deliver 150W. In addition, each PCIe slot can deliver up to 75W by itself.

(2) The tabs on the 8-pin connectors seem to be engaged, meaning the connector was pushed into the socket all the way until it locked (audible click). This is as it should.

(3) It appears that each GPU is supplied by a separate cable running directly from the PSU. I see no evidence of power supply sharing, e.g. daisy chaining, splitters, or converters. Good.

To my knowledge, there is no standard governing the power monitoring and control in systems. Each system integrator (e.g. HPE, Dell) seems to have their own proprietary arrangement of sensors, controllers, and algorithms.

As has been pointed out, your GPUs run slow because they draw little power, and they draw little power because the system tells them to by asserting the “Power Brake” signal. This signal is generated by the system. So the issue to resolve is: “Why is power brake being asserted?” Your system integrator (that is the company that put this system together) should help you resolve this.

It could be due a fundamental problem like an under-dimensioned power supply: maybe the system originally had fewer GPUs and more were added later. There could be a problem with the sensors. There could be a problem with the power management algorithm or programming of the board management controller firmware which may be addressable with a system BIOS upgrade. There could be a problem with this specific GPU type lacking validation by the system integrator on this specific platform, so the controller lacks required power-management information and defaults to a safe setting.

The system integrator is who you should ask for support to resolve these potential issues (and others I do not know about or have forgotten to mention).

11

thank you very much everyone for your help.

Just to tell the end of the story, after contacting the integrator, they said that " this issue is a hardware defect of the PCIe lanes on the motherboard which causes the GPUs to not be able to draw enough power which is why nvidia-smi shows HW Power Brake Slowdown. "

12

That explanation seems plausible. It makes me wonder how such a condition would have developed over time, as you stated the system as initially delivered worked fine.

I assume the fix consists in the system integrator installing a new motherboard?

13

So, out of curiousity, do you know if the cards originally fitted, were a different model to that fitted now?

I ask, as I’m somewhat confused as to what changed, when you said: