ISR9024D-M

Infrastructure & Networking Switches and Gateways InfiniBand/VPI Switch Systems
1

Hello,

As i mentioned on this thread: Infrastructure & Networking - NVIDIA Developer Forums I did get into the switch till i set it to be managed by IB instead of ether. Now i cannot connect to it as im having issues using the serial, which i think i may have figured out why. The serial card i bought had some jumpers on it (Startech PEX1S952) They only show two configs for the pins, (windows only and windows+dos). The pins were configured for neither… SO i’ll give it another shot here in a sec.

As for as nmap, i have never had to use it and i dont think im doing something right as it always responds with:

nmap -v -sn 192.168.2.0/24 -e ib1

Starting Nmap 6.00 ( http://nmap.org http://nmap.org ) at 2013-06-21 18:00 EDT

I cannot figure out what source address to use for device ib1, does it even exist?

QUITTING!

ib1 is the ingerface im using for IPoIB. The only DHCP server i have is from my router modded with DDWRT but the IP doesnt show up on the list of connected devices. (ARP table?)

I have tried the reset button. Held it for 3-4 seconds, tried holding it for 8 seconds, and even for 30 seconds, even while it was powered off.

Is there another un-documented way to hardware reset everything to defaults other than from cli?

2

Just found something out by accident.

I’ve logged into my 9024D-M switch as root, and ifconfig is showing the default IP address of the ipoib0 interface as “192.168.2.100”.

If you configure one of your Infiniband cards with IPoIB, and put it on the 192.168.2.x subnet, you might be able to get to 192.168.2.100 through it.

3

Wireshark shows me this everytime with the ISR directly connected to my second ether port on my desktop:

24250.097091000Heurikon_69:a1:a0BroadcastARP60Gratuitous ARP for 192.168.2.9 (Request)BroadcastHeurikon_69:a1:a0

which i looked up the Heurikon and it is part of the switch as i believe someone posted here on this site.

and the ARP info states:

hardware Type: Ethernet (1)

Protocol type: IP (0x0800)

Hardware size: 6

Protocol size: 4

Opcode: request (1)

[Is gratuitous: True]

Sender MAC address: Heurikon_69:a1:a0 (00:80:f9:69:a1:a0)

Sender IP Address: 192.168.2.9 (192.168.2.9)

Target MAC Address; Broadcast (ff:ff:ff:ff:ff:ff)

Target IP address: 192.168.2.9)

4

I set one interface on my pc to 192.168.2.9 and wireshark shows that it detected a duplicate ip. But no request from the ISR to fall back to another address… Do the infiniband interfaces have a different default IP? or so they get only manually assigned per server? Im still a little new to infiniband.

5

Hmmm, the instructions with that “multi-purpose” cable sound fairly complicated.

For my 9024D-M, I just plugged a standard everyday cat 5e ethernet cable into the front ethernet port (next to “I2C” DB9 connector), and plugged the other end into the 24 port 1GbE switch I’ve got all my lab hooked up to.

From there, I was able to ping 192.168.1.2 perfectly fine, connect to it with telnet, and connect to it with a web browser (http, tcp port 80).

Note, my (cheapo TP-LINK) 24 port switch is one of the MDI/MDIX sensing ones, so it automatically detects whether the cable plugged in is a cross over one or not and works fine either way. Most modern switches (last few years) are the same.

With nmap, try it like this:

$ sudo nmap -v -O 192.168.2.0/24

Starting Nmap 6.25 ( http://nmap.org http://nmap.org/ ) at 2013-06-21 23:47 BST

Initiating ARP Ping Scan at 23:47

Scanning 68 hosts [1 port/host]

Completed ARP Ping Scan at 23:47, 0.89s elapsed (68 total hosts)

Initiating Parallel DNS resolution of 68 hosts. at 23:47

Completed Parallel DNS resolution of 68 hosts. at 23:47, 0.00s elapsed

Nmap scan report for 192.168.2.0 [host down]

Nmap scan report for 192.168.2.1 [host down]

Nmap scan report for 192.168.2.3 [host down]

Nmap scan report for 192.168.2.4 [host down]

Nmap scan report for 192.168.2.5 [host down]

Nmap scan report for 192.168.2.6 [host down]

Nmap scan report for 192.168.2.7 [host down]

Nmap scan report for 192.168.2.8 [host down]

Nmap scan report for 192.168.2.9 [host down]

Nmap scan report for 192.168.2.10 [host down]

[etc]

I don’t have to specify an interface when I run it (from my OSX desktop), because my desktop’s default IP route goes to the 24 port switch mentioned, which that front cable on the 9024D-M is connected to.

Does this help?

(note - edited for typo fixes)

6

im going to have to try putting it on its own switch to eliminate complications. I tried the command above but it tells me:

route_dst_netlink: can’t find interface “vmbr0”

This is on a proxmox lunix system (like esxi but opensource). the vmbr0 is the virtual nic which has been given the ip 192,.168.2.10, and i can see that it does via ifconfig -a. I have a trendnet switch which i believe does the same thing you described.

I was able to access the switch via ether as you have, just not since i told it i wanted management to be done via infiniband. So im not sure if IB has a different default network address or if it gets the IP i assigned the ehternet port.

7

Well, here is some good news.

I took one of the Vm servers off the network, connected it to the IB switch using the CX4 cable and now i can ping 192.168.2.9 but when i try to ssh to it, i get nothing in return (destination host unreachable). But in order for me to ping it i have to specify an interface (ib1) to use for ping. Not sure how to do this with ssh, so im going to have to look at the routing table now.

8

I noticed in the directions for cabling it says this:

To connect multi-purpose cable of an ISR 9024-M (internally managed

unit):

Step 1 Connect the DB-9 connector and the RJ45 connector to each extremity of the

multipurpose Mng cable.

Step 2 Connect the DB-9 connector to the CLI port on the ISR 9024-M

Step 3 Connect the RJ45 connector to the Ethernet port either on the rear panel or on the

front panel (but not to both).

I am assuming (since i didnt get any cables with it) that the management cable is the patch cable? And the directions are a little confusing as it sounds like they want you to connect the db9 port to a db9-to-rj45 converter to a cat5e cable them plug that right into the ether port?

9

Sweet Im in and able to manage the switch vi the infiniband connection.

The issue was that one of the nics on the box has the same subnet as the switch (192.168.2.x). The SM on the switch itself is trying to manage that subnet so its conflicting with my router. So i changed the IP of the ether port, change the IP of the ib0 interface to be on 192.168.2.x and bam, access granted.

Thanks for your help with eliminating such a trivial problem. cant believe i didnt catch that sooner…

SO the IB interface does inherit the IP i set for the ether on the ISR switch. Good to know.

10

Interesting, so it sounds like it’s trying to use 192.168.2.9.

“Heurikon” is most likely correct too, as that’s how the board shows up in nmap as well:

Re: Voltaire ISR-9024D (NOT -M) Question

Are you able to ping 192.168.2.9, or maybe connect to that using telnet/http? You might need to configure the IP address of your second ethernet port to be in the same subnet (eg 192.168.2.10) for that to work.

It sounds like you’re making progress.

11

I have put the ether on the same subnet as the switch (192.168.2.8) but i cannot ping the ISR. when i connect the ISR to a trendnet TEG-S80g 1GB switch it doesnt seem to establish a connection (both activity light and link light just constantly blink) however if i connect it to a router or directly to the ether of my desktop the link light stays solid and the activity light only blinks a couple times (The ARP request).

When i try to ping it always says that the destination host is unreachable.

12

I’m not 100% sure. I’ve only used the default IP with my ISR 9024D-M, without never really needing to think outside of this particular box.

Out of curiosity, have you looked all through the GridVision manuals? I remember when I scanned through them quickly ages ago, they did mention some stuff like this.

If you don’t have them yet, you can get them using the FTP details here:

Re: Mellanox (old Voltaire) ISR9024D-M recover flash area Infrastructure & Networking - NVIDIA Developer Forums

13

Interesting. It’s obviously wanting to use the 192.168.2.9 address, but doesn’t seem to be actually going ahead and using it. Unfortunately it’s been ages since I had to work with IP(v4) network layer stuff in real depth, so I’m mostly out of sensible ideas.

For a non-sensible idea, I kind of wonder what happens if you already have something on the network using 192.168.2.9? ie What is its fallback behaviour?

If really, really, really lucky, the fallback behaviour might be to use the original 192.168.1.2 address. But, it could just as likely be “pick another address at random” or “stop working right here”.

Um, good luck?

14

Interesting. I’ll have to give it a shot tonight when i get home.

Thanks

15

Cool. And yep, good to know.