Suggestions for motherboard, case and PS for dual M1060 setup Many questions about what system parts

Accelerated Computing CUDA CUDA Programming and Performance
1

Hi,

I recently bought a couple of M1060 cards that are HP parts meant for one of their big mondo and expensive servers, but I’m wanting to use them in a NORMAL desktop computer running Linux or maybe WIndows, but not Windows Server. I tried to put one in my current desktop system and found out that because of the huge heat sinks that are really close to the PCI edge connector, that the heat sink is in the way of some capacitors and other electronic parts that are close to the PCI connector on the motherboard, and cannot be physically inserted into the existing PCI express slot.

Has anyone used these HP M1060 cards in anything other than the ProLiant SL390S G7 system they were designed to go in (I’m guessing about this as I don’t know yet which HP system these were meant for since I can’t find the M1060 on the HP web site yet, only the M2050/M2070 cards)? Does anyone know exactly which HP system these cards were really meant for? Does anyone know where I can get any user guides or installation guides for these M1060’s?

If I’m trying to beat a dead horse and this is a hopeless project someone please tell me now - I hope not but realize it’s possible that these will only ever be able to be used in the HP ProLiant system they were designed for. If so would much rather find out now before a lot of time and expense are invested.

Will one GeForce 580 be a better/faster choice than two of these M1060’s?

So has anyone already solved this problem and know what motherboard, case and power supply would be needed to run two of these M1060’s and perhaps another nice Nvidia video card since the M1060’s have no video output (like maybe a 580 type) and maybe another couple of misc cards?

So what’s needed is a motherboard with at least 3 PCI Express x16 slots (and the two slots meant for the M1060 would have to be free of obstructions that my current motherboard isn’t), plus a couple of other regular PCI slots. The case has to have enough clearance to accept these really long HP M1060 cards as they have a funny metal bracket on the end furthest from the slot cover (the entire card with this bracket is about 12.375" long). These metal brackets can be removed but don’t think they should be as the card needs some support at that end because the cards are so heavy with that big honking heat sink, so had intended on having some sort of bracket fabricated to enable that end to be supported by attaching to the case somehow. Am guessing that the board would vibrate a bit with all that mass so far away from the PCI connector and might crack the card if not supported.

Am guessing that a 1000W power supply should be enough for two M1060’s plus a 580 graphics card plus the normal storage devices (a couple of DVD’s and a couple of internal drives and a couple of misc cards like a wireless network card or high speed ethernet or others along those lines. Perhaps even an 800-850W would be sufficient - one goal is to have as quite of a system as possible as this is in a home office not a lab or computer data center.

Also will it be possible to use any Nvidia graphics card with these M1060’s or are there only certain ones that will work?

I am just wanting to have a system to experiment with using CUDA and these GPU cards that I have, so don’t bother recommending I replace them with a single newer 2050 Fermi card. I got a really good deal on them and think they would be fine for what I’m trying to accomplish in getting my feet wet in the CUDA pool.

Thanks for any and all suggestions and recommendations.

Here are a couple of picture of what the card looks like in case anyone is not clear of what I’m trying to describe (HP part number is 900-20607-0350-000 and both were brand spanking new and never have been opened). The first picture shows the overall card (the bottom of the picture is really the top edge of the card)

M1060 card image

And here is a closeup showing the heat sink and how close it is to the PCI edge connector so you can see how little room it leaves for the area around the PCI express connector on the motherboard (it’s not as focused as I would like but can’t get my camera to focus if it’s any closer - this is a view of the bottom PCI connector and heat sink and the left side of the picture is the front slot cover part of the board):

M1060 PCI connector

Also have some questions about the different connectors on the M1060 board:

  1. There are two power connectors on the top right corner of the card - one is a 6-pin and the other is an 8-pin (I’m guessing they are power - are they?). Are these the same as other Nvidia graphics cards or are they wired any different? So will the standard 8-pin and 6-pin connectors from power supplies work fine?

  2. There are two 4-pin small connectors (one on the top left corner if the slot cover is on your left, the other is on the bottom right). Anyone know what are these style of connectors called? What are these for?

  3. There is a 20-pin connector on the top left of the board that looks like it takes some kind of a ribbon type cable. Anyone know what this is called and what’s it for?

  4. There is an edge style connector along the top edge of the card - am guessing this is for some sort of connector when using more than one board, ala SLI type connector? Anyone know what this is for and when it would be used?

  5. Are there any temperature sensors on this card and if so which connector is used for getting this temperature information?

There wasn’t any sort of documentation that came with this M1060 board. I’ve looked around on Nvidia’s and HP’s web site but haven’t found anything yet. Anybody know where I can find any sort of documentation for this board that gives any sort of details about it?

Also what about air flow for these two M1060 cards? These are passive designs so am guessing they radiate quite a bit of heat so what’s the best way to keep them cooled or perhaps is heat even an issue with them? Do I need to have a fan blowing directly across them to dissipate the heat? Which direction would be best for the air flow? What would be the best placement for the M1060 cards without another big heat generating Nvidia card or with one. Am guessing that the further apart these cards are the better? Or is placement really an issue if there is enough airflow?

What sort of CPU horsepower would be recommended for running apps on these boards? I know their double-precsion speed is pretty low compared to newer 2050/2070 cards, but for getting my feet wet with CUDA am willing to live with this for now. Someday, if I really get into this, I might be willing to get a 2050 type card - in that case would this make a difference for the system I put together today? In other words, if I were building the system to use two 2050 cards would that change anything in the parts specifications for case, power supply or video card or air flow since 2050/2070 cards come in both active and passive designs apparently (passive ones from HP and active ones from Nvidia???)?

Thanks for any and all help.

Anyone have any idea why the Nvidia registration would not work for this board - I used the part number on the box (900-20607-0350-000) and the page where I tried to register says that this is the correct style but it refuses to accept it and complains it’s not correct but yet it adheres to the form that it claims is correct, so not sure how to register these boards. Any suggestions? After I do get them registered are there any documentation that I can then get access to that without registering them you can’t?

Any other suggestions or recommendations from anyone who’s already been thru this exercise would be greatly appreciated.

Any other questions I should try and find out the answers to before proceeding to put the system together? Like are some Linux versions better/worse than others? Appreciate any heads up on the issues that I’ll be facing pretty soon.

Is it better to attach images to these posts or better to have links to where pictures can be seen on some other web site?

Apologize for all these disjoint questions - this is a bit overwhelming at first, but am looking forward to getting very comfortable with this whole GPU effort.

Thanks again…

-Bob

2

You won’t be able to use those cards in a standard ATX style case. The M1060 is intended only for 1U or 2U rack cases, lying on their sides with an array of blower fans behind them to move a large volume of air over them and out the rear of the case - the top and bottom of the type of rack case they are intended for plays in integral role in directing air across the heat sink on the board. The GPU on those cards has a TDP of over 150W, and using them without correctly designed cooling solution will probably damage them beyond repair. So you are looking at a rack case for these, otherwise you are out of luck.

That is only you first problem. Because these cards are designed to lie flat in a rack case, there are only a handful of specialized cases and motherboards that can accommodate two of them in one case, like this Supermicro one. That greatly complicates things because those cases and boards are effectively OEM only items intended for the HPC market, and they are expensive and not avaiable retail. If you decide to use a single card in a rack case with a standard ATX board (they exist but aren’t cheap or easy to find), then you are going to need a 1U power supply that can give about 40A on the 12v lines with a 6 or 8pin PCI- express output. Those are also not easy to find or or cheap. I seem to call you only need to connect either the 6pin or the 8pin but not both to power the board, so it is not beyond the realms of possibility to do it with a desktop board and a 90 degree PCI-e riser card in a 1U case, but it will be “custom” to say the least. I am 99.9% certain you would invalidate whatever warranty and support nvidia might offer on those cards by doing so.

If you were hoping to do this cheaply, I fear you are out of luck. It might be easier to sell these and find yourself a pair of C1060s, which are effectively the same gpu, but have build in cooling in the regular form factor and will work perfectly in a standard ATX tower case with a suitable desktop or workstation motherboard and PSU.

Caveat emptor, as they say…

3

If you can’t find a way to get the M1060 cards working, the GTX 580 is a decent alternative. The raw double precision performance of a GTX 580 is roughly equivalent to a pair of M1060 cards, and you don’t have to mess with multi-GPU programming. Plus, you get all the compute capability 2.0 benefits, like L1 and L2 caching.

The best Linux tip is to pick the exact distribution version from the supported OS list. There is a strong temptation to assume that when the release page says “Ubuntu 10.04”, they mean “Ubuntu >= 10.04”, but that is not true. You don’t want to be fighting weird gcc incompatibilities when you are first learning CUDA. Later, you can experiment and see if newer distribution versions still work.

4

I was able to return the two M1060 boards to where I got them for a complete refund, which was great.

I think I will try and follow your advice and go with a single GTX580 card and get my CUDA feet wet with it.

So now the question becomes will the GTX580 card work in my current PC if I upgrade the power supply, or do I need to go with an entirely new computer?

I think the thing to try is first upgrade the power supply to something on the order of 800-1000W and see if it works. If it doesn’t I can still use the new power supply and GTX 580 board in whatever new computer I would get. I’m on a tight budget for this so don’t want to go hog wild and get a bunch of items I don’t really have to have right now in order to decide if this is something that I’m able to get a lot out of or something that’s just a passing fancy.

Any recommendations or power supplies big enough in capacity but small enough to fit in a regular tower case?

Thanks again very much to all for the advice and insights.

-Bob

5

Quick rules for GPU PSU choices… they’re more strict than any other choice when building a system

  1. Reject any PSU that costs less than $150. bare minimum. Expect to pay more. Don’t skimp.
  2. Check the cables it supplies. It will make a HUGE difference if you go to dual or even triple GPU
  3. Thermaltake Toughpower is the solid choice for me. I’ve used Antec, Silverstone, BFG, Corsair, PowerPC&Cooling and the Toughpower.
    I’ve had failures on one Antec. The corsairs are OK but cables are limiting. BFG was a rebadged PSU, they don’t sell them any more.
    Toughpowers seem great… 4 systems, running 24/7, never a peep for a year.
    I use a Toughpower 1200W for my 3-GPU builds… I could likely reduce that to a 1000W since my wallsocket power draw is only about 750W.

This is not a bad guide for GPU system building, since it covers a dual-GPU system (for gaming, but it’s similar.). It uses too weak of a PSU though. I heartily endorse the FT02 case.