ConnectX-3 EN on an ARM platform

Hi,

I’ve been trying to get an ConnectX-3 EN (Dual port, 10GE) to behave nicely on an ARM-based platform but noticed that the NIC has some stability issues…

Is running on an ARM cpu supported? Have you ever tested it yourself?

I’m running on one of the latest stable Linux kernels, by the way (3.8.11).

Regards,

Stan

Hi Stan, thanks for clarifying your question. I’ll do some digging to see what I can find out for you. Can you tell me which board and ARM processor you’re experimenting with? Thanks.

I see. My question is though- Could it be that the expansion ROM is doing “something” (Some configuration change in the adapter card, I guess) that the O/S driver later relies on?

Like- PCIe link configuration, initialisations etc.

If so- that might explain why the card behaves so differently on a non-x86 system.

Hi Stan,

Just as an FYI, and probably not very helpful for your immediate issue, but check out this press release that mentions ARM Cortex-A9 processor and Mellanox InfiniBand. Pedraforca Cluster to be First to combine ARM CPUs, GPUs, and InfiniBand - insideHPC Pedraforca Cluster to be First to combine ARM CPUs, GPUs, and InfiniBand - insideHPC

  • I’ve tested the card on an x86 PC- works fine.

  • I did not check the power consumption but I doubt that’s the issue at hand (10GE ethernet cards from other vendors work fine on this board).

  • It’s a custom development board- not an off-the-shelf platform.

I’ve noticed the adapter card has an expansion ROM that’s probably executed at boot time by the PC’s BIOS (It’s an x86 byte code image)- any idea what it does?

Regards,

Stan

Hmmmm… no idea at all here, but if that card is supposed to work with powerpc stuff, then maybe it’s not x86 code in the expansion ROM?

From the Mellanox documentation, I see support for the following CPU architectures: x86_64, x86 and power-pc. I’m not sure if ARM is officially supported or not?

Hi,

Where you able to test your card on classic CPU architecture ?

Have you checked the power consumption of the adapter on the mainboard ?

May I ask what do you have as mainboard for your ARM-based platform ?

Regards

It’s definitely x86 code, but it might not have anything to do with the issues I’m experiencing… It might only be used for PXE, no idea.

Hi Stan, yes, the expansion ROM image presents itself to the BIOS as a boot device. As a result, the BIOS will add to the list of boot devices Mellanox FlexBoot for a ConnectX device. The priority of this list can be modified through BIOS setup. Please refer to section 1.1. of this document for a description of FlexBoot: http://www.mellanox.com/related-docs/prod_software/FlexBoot_user_manual.pdf http://www.mellanox.com/related-docs/prod_software/FlexBoot_user_manual.pdf . FlexBoot enables booting kernels or operating systems (OS) from remote servers in compliance with the PXE specification.

I’m not in liberty to say… Like I said- it’s an ARM-based custom development platform (ARMv7 architecture).

Thank you,

Stan

stan Infrastructure & Networking - NVIDIA Developer Forums - That Pedraforca article which branko Infrastructure & Networking - NVIDIA Developer Forums posted the URL to sounds potentially relevant.

IP issues aside, if you ping the technical guys that were involved in that, they might have some insight into your problem. The Mellanox guys here might be able to intro you. (no idea)

If not, I might be able to find a contact person for you through some old Dell connections. (or you could just Google for relevant stuff, might be just as effective)

Hope that helps.

(note - edited for typo fixes)