Mellanox (old Voltaire) ISR9024D-M recover flash area

I have 9024D-M, 4036 switches.

4036 works fine.

But 9024 have a problem some day before.

I connected console via serial cable.

But I can’t access safe mode.

I’m also tried bootd, bootp via tftp protocol.

backup firmware display inflate fail message everytime boot process.

How can I reprogramming flash area?

How can I recover my main SM switch 9024D-M?

I’m tried your 0.8.6 firmware.

Update was successful, but my console wasn’t return…

I’ll receive another 9024D-M switch next week.

I’ll try eburn command via root login to new switch.

Your log shows a write protect off EEPROM via eburn.

I’m also try it…

My corrupted 9024D-M can’t access serial console AND ethernet, too…

It doesn’t response pinging to default ip address 192.168.1.2.

Telnet/SSH connect was, too…

Hi!

I was completed that can I using my corrupted 9024D-M like 9024D…

And I found attached document via Google…

This document said internally managed 9024 switch can update firmware remotely (in-band)

another All 9024 switch (including managed or not)…

Below is example on the firmware release example

I’ll challenge…!

Thank you for your reply again!


Hi Guys,

i don’t know how but i got this procedure from some really dark area. i had to dust off few layers until i found this.

This procedure is good for cases where your switch OS is still functioning but the switch ASIC firmware itself got corrupted. for any other cases (e.g. you can’t even access the switch b/c the OS is corrupted), don’t wast your time.

And i am saying this upfront: USE IT AT YOUR OWN RISK !!!

This is an old procedure for recovery ISR9024D firmware from corrupted flash situation. Please read carefully and use technical and common sense :-)

let me start by showing the default env parameters for those of you who were looking to get things back to were they were originally:

=> printenv

loads_echo=1

baudrate=38400

filesize=235

bootargs=console=ttyS0,38400 root=/dev/ram0 rw

addip=setenv bootargs $(bootargs) ip=$(ipaddr):::$(netmask):$(hostname):eth1:none

sec=echo booting secondary, press ^c to abort; sleep 5; echo booting…; bootm d0a00000

prim=echo booting prim…; bootm d3500000

serial#=0

ecc=on

pci_enum=on

pci_listing=short

ethaddr=00.80.f9.69.fc.18

eth1addr=00.80.f9.69.17c.18

gatewayip=0.0.0.0

hostname=0.0.0.0

ipaddr=0.0.0.0

netmask=255.255.255.0

serverip=0.0.0.0

bootcmd=tftpboot;go

bootfile=path/file.bin

bootdelay=-1

loadaddr=80000

stdin=serial

stdout=serial

stderr=serial

Environment size: 602/2556 bytes

l

Next, the procedure itself:

if your switch is showing things like the following that procedure is the right for your situation. for other situations, use your own judgment:

Anafa not found

Start Firmware upgrade

\nError: Firmware upgrade FAILED (retry 1)

\nError: Firmware upgrade FAILED (retry 2)

\nError: Firmware upgrade FAILED (retry 3)

Firmware upgrade failed, call support

From attempted logins:

ISR-9024 login: admin

Password:

connecting

connect: No such file or directory

connect: Illegal seek

ConnectToServer::ERROR:Server not responding

connect: No such file or directory

connect: Illegal seek

ConnectToServer::ERROR:Server not responding

To recover from this failure:

  1. We must retrieve the GUID from a sticker inside the 9024. We do not have a way to retrieve the GUID once the switch is in this state without reading the sticker.

a. Disconnect power and open the cover. (three screws and the cover slides back)

b. Look on the bottom of the sheet metal case. You will see a sticker with 2 serial numbers. The 16 digit numbers will start with 0008F1xxxxxxxxxx.

c. Right down the number and put the cover back on the switch.

  1. Using a serial port connection to the 9024 login as root (default password is br6000)

  2. Assign an IP address to the eth1 interface:

a. ifconfig eth1 10.0.2.170 netmask 255.255.255.0

  1. Download the recovery files attached with this procedure (“ISR9024-i2c-upgrade.tar.gz”)

  2. Using FTP or WGET place the i2c-switch-burn.tar on the 9024

  3. Create a directory, (mkdir /temp) and expand the tar into it, (cd /temp ; tar -xvf …/i2c-switch-burn.tar)

  4. Make the files executable. (Chmod 777 /temp/*)

  5. Copy the files: eburn and i2c to /usr/Voltaire/bin. (cp eburn i2c /usr/voltaire/bin)

  6. Run gunzip against the firmware image.

gunzip firmware.ISR9024_SM.img.gz

  1. Run the script using the first number recorded from the sticker (sn#1)

./ibswlrm-fw-burn.sh 0008F1xxxxxxxxxx 0008F1xxxxxxxxxx firmware.ISR9024_SM.img.gz

./ibswlrm-fw-burn.sh 0008F1040041082A 0008F1040041082A firmware.ISR9024_SM.iIg

FW_VER = 0.8.6

sysimage_GUID = 0008f1040041082a

Node_GUID = 0008f1040041082a

Burning …

BURN -

100%

Verifying …

VERIFY - 100%

  1. Power cycle the 9024 (mandatory)

May the force be with you…

I think console configuration is locate another address on EEPROM.

I can run the printenv command on u-boot or ppc-boot (9024 or 4036)

How can I change that configuration?

=> printenv

loads_echo=1 ---------> this is problem

baudrate=38400 -------> this also problem

filesize=235 --------------> It’s also deleted

bootargs=console=ttyS0,38400 root=/dev/ram0 rw

addip=setenv bootargs $(bootargs) ip=$(ipaddr):::$(netmask):$(hostname):eth1:none

sec=echo booting secondary, press ^c to abort; sleep 5; echo booting…; bootm d0a00000

prim=echo booting prim…; bootm d3500000

stdin=serial

stdout=serial

stderr=serial

serial#=0

ecc=on

pci_enum=on

pci_listing=short

ethaddr=00.80.f9.69.fc.18 ------> This is also deleted…

eth1addr=00.80.f9.69.17c.18

gatewayip=0.0.0.0

hostname=0.0.0.0

ipaddr=0.0.0.0

netmask=255.255.255.0

bootcmd=tftpboot;go

bootfile=path/file.bin

bootdelay=-1

loadaddr=80000

My console show a below now…

What does mean that character?

Hmmm, so it seems like “printenv” shows the environment variables, “setenv” is used for setting them, and “saveenv” (hopefully) saves them to flash. It doesn’t seem to list an explicit way to delete them, which might mean that “saveenv” with no value might do that. Or using the “=” sign, as you mentioned in a post too.

If you create a brand new environment variable and save it, is it still there after you reboot the switch? Also, how to embed spaces in the value?

Something like:

=> setenv mytestvarA 123

=> setenv mytestvarB ‘123’

=> setenv mytestvarC “123”

=> setenv mytestvarD “123 456”

=> saveenv

Then reboot the switch and see if any of the “mytestvar*” variables show up with printenv.

With the corrupted 9024D-M, I’m not sure which environment variables need to be set. Experiment time.

(note - edited for typo fixes)

Hi yairi!

Thank you for your reply.

I’ll check it this weekend and reply on this thread…:)

Last post result is below and I was ask for help someday before

PPCBoot 1.1.6 (Nov 16 2004 - 08:21:51)

CPU: IBM PowerPC 440 Rev. C

Board: Artesyn Technologies PM/PPC-440gp

Artesyn Monitor Version 1.5.0

VCO: 800 MHz

CPU: 400 MHz

PLB: 133 MHz

OPB: 66 MHz

EPB: 66 MHz

I2C: ready

DRAM: ECC Init …Done 512 MB

FLASH: 64 MB

Initializing board as Monarch…

No PCI Enumeration due to environment variable pci_enum override

In: serial

Out: serial

Err: serial

Net: EMAC0, EMAC1

BEDBUG:ready

=>

  • At this step can’t proceed anymore…:(

But I could not connect console anymore in weekend test…:(

If I can’t connect console anymore then

can I remote In-Band firmware update via my another 4036 QDR switch?

I have also CX4 to QDR hybrid cable and my 4036 can find ISR9024D-M switch.

Best regard.

Thanks for reply…:)

My 4036 is works perfectly!

But 9024D-M wasn’t also serial console, too.

My 4036 report to me that 9024D-M’s firmware was corrupt!

I’ll purchase another ISR9024D-M on eBay.

And I’ll try update remote-firmware all command on Another 9024D-M.

(In-Band firmware update that descript on old manuals)

There is a final quiestion about my irresponsible 9024D-M.

If my 9024D-M switch is normal in hardware aspect then can I correct from another 9024D-M switch?

Thank you for your kind reply…

I love Grid Vision interface on 9024D-M that support 4036…

P.S

I’m also know that my 9024D-M wasn’t reached Safe mode.

If I type ‘reset’ command then switch was reboot and stop in same stage everytime.

Perhaps U-Boot firmware and configuration(env) was corrupt via unknown problems…

I don’t want to change boudrate to non default value 38400.

Recover the deleted boudrate to default value 38400 and another parameter value, too…

That’s the last method to factory-default reset via console access.

Firmware was recovered but I don’t know about change the boot parameter on EEPROM.

I was received another 9024D-M.

If recovering corrupted 9024D-M is possible then

I’ll make a fail-over switch fabric to training my IB skills…

Do you have any information about reset to default parameter on EEPROM for PPC-Bootup?

Hi!

I was test all command for my corrupted old InfiniScale III switch Voltaire ISR9024D-M.

Latest MFT 3.0 for Linux and Windows edition doesn’t support it anymore.

And I was build old CentOS 5.2 (with OFED 2.0 - the last ibspark embedded edition) system for recover my switch.

I found very interesting point…

flint, mstflint support Infiniscale IV switch, HCA only.

InfiniScale III was not…

mlxburn was support ISR9024D-M.

But whenever if you excute the mlxburn command that call the spark or ibspark (with -inband)

spark command is last option for ISR9024D-M now!

Because of MFT 2.1 or above will running in safeburn mode only when burnning on blank EEPROM

or corrupted…)

That caused my choice.

I’m also purchased another ISR9024D-M for export the normal firmware with normal u-boot configuration

from it.

(include baudrate, mac address, etc…)

fortunately spark support ri command to export current image to local disk.

But can’t support dc command like flint…

I’ll overwrite on corrupted EEROM with normal exported img file with spark ~ bb command.

(block burn. that’s all overwrite on EEPROM)

Next…

I’ll reburn with variety switch combination for recover original sysguid, nodeguide, bsn.

If my old one hasn’t hardware problem then I can recover it…

The last interesting thing is i2c port.

That’s for FAE or advanced users only interface.

I’ll be back again in near future…

Thank you for your reply.

Ok. All that’s left to do now is make the corrupted 9024D-M environment the same as the good one isn’t it?

The “bootcmd” variable looks both different and important. It uses spaces in the value, which may or may not need quoting. I have no idea, you’ll need to find out?

I noticed a JTAG connection on the boards inside the other day, when I was having a look around in my 9024D-M.

Back in Australia I had a JTAG - USB connector from when I was mucking around with Arduino stuff. I’m kind of curious to see what can be done with it. Maybe useful for setting values in stuff.

I’m recerived another ISR9024D-M that works good…

At first I’m connect to corrupted 9024D-M to new one then run the command

update remote-command and finished successfully.

But console was not return.

There isn’t update remote-software command…

I’m find reason for console problem.

It’s a cleard configuration and software on flash bank…

Below is normal printenv result via console


In:serial

Out: serial

Err: serial

Net: EMAC0, EMAC1

BEDBUG:ready

Hit any key to stop autoboot: 0

=> printenv

loads_echo=1

serial#=9998

ecc=on

pci_enum=on

pci_listing=short

ethaddr=00.80.f9.69.23.26

eth1addr=00.80.f9.69.a3.26

gatewayip=0.0.0.0

bootfile=path/file.bin

baudrate=38400

filesize=235

netmask=255.255.0.0

ipaddr=192.168.1.2

serverip=192.168.1.100

loadaddr=800000

hostname=0.0.0.0

bootdelay=2

bootargs=console=ttyS0,38400 root=/dev/ram0 rw

addip=setenv bootargs $(bootargs) ip=$(ipaddr):::$(netmask):$(hostname):eth1:none

sec=echo booting secondary, press ^c to abort; sleep 5; echo booting…; bootm d0a00000

prim=echo booting prim…; bootm d3500000

bootcmd=run addip sec prim

stdin=serial

stdout=serial

stderr=serial

Environment size: 619/2556 bytes


Below is last console log of corrupted 9024D-M


BEDBUG:ready

=> printenv

baudrate=38400

filesize=235

bootargs=console=ttyS0,38400 root=/dev/ram0 rw

addip=setenv bootargs $(bootargs) ip=$(ipaddr):::$(netmask):$(hostname):eth1:non

e

sec=echo booting secondary, press ^c to abort; sleep 5; echo booting…; bootm d0a00000

prim=echo booting prim…; bootm d3500000

serial#=0

ecc=on

pci_enum=on

pci_listing=short

eth1addr=00.80.f9.69.17c.18

bootcmd=tftpboot;go

bootdelay=-1

loadaddr=80000

stdin=serial

stdout=serial

stderr=serial

Environment size: 451/2556 bytes

=> setenv baudrate=38400

=> setenv filesize=235

Unknown command ‘setenv’ - try ‘help’

=> saveenv

Saving Enviroment to EEPROM…

=> setenv addip=setenv bootargs $(bootargs) ip=$(ipaddr):::$(netmask):$(hostname):eth1:non

=> e

Unknown command ‘e’ - try ‘help’

=> setenv addip=setenv bootargs $(bootargs) ip=$(ipaddr):::$(netmask):$(hostname):eth1:non

=> saveenv

Saving Enviroment to EEPROM…


When switch was reboot then all was gone…

(If I run the initenv command before reboot the switch there is no problem exists…)

setenv usage is very confused to me…

I find that => setenv baudrate=38400 command delete baudrate value.

Correct usage is setenv baudrate 38400

Also another value was deleted…

It’s a correct that configuration is located on EEPROM.

But there is nothing to correct the problem on user level.

If you have a solution via normal switch or etc, can you explain to me?

If initial configuration on EEPROM recover then can I re-install SM software on NAND Flash again?

I can’t access corrupted 9024D-M console…

Therefore I can input that commands…

If my threads help your work, that’s my pleasure…

yairi Infrastructure & Networking - NVIDIA Developer Forums - Is there any chance you guys still have the .ini files around used to generate the firmware for the old ISR 9024D-M switches, so people can try burning new firmware with customised settings?

Alternatively, is there a way to extract the original .ini file settings from generated firmware? I’ve looked through the mlxburn command but haven’t seen anything that would do it.

inbusiness Infrastructure & Networking - NVIDIA Developer Forums - Discovered something interesting.

I can log into my 9024D-M as root (not “admin”) using telnet/ssh. Default password is “123456”, same as for the admin user.

It gives shell access (ash), instead of the limited menu for the admin user.

You might be able to do something useful with this.

$ ssh -l root 192.168.1.2

root@192.168.1.2’s password: 123456

Welcome to Voltaire Switch ISR9024D-2200

BusyBox v0.60.5 (2008.02.25-06:43+0000) Built-in shell (ash)

Enter ‘help’ for a list of built-in commands.

You have logged in with root access.

Operations available in this mode may affect the system normal

operation and should only be performed on Voltaire Support approval only.

ls -al

drwxr-xr-x 14 root root 1024 Jun 20 03:22 .

drwxr-xr-x 14 root root 1024 Jun 20 03:22 …

lrwxrwxrwx 1 root root 14 Jun 20 03:22 .ssh → /mnt/jffs/.ssh

drwxrwxr-x 2 root root 2048 Feb 25 2008 bin

drwxr-xr-x 3 root root 6144 Jun 20 03:22 dev

drwxr-xr-x 8 root root 1024 Jun 20 03:22 etc

drwxr-xr-x 3 root root 1024 Feb 25 2008 home

drwxr-xr-x 4 root root 2048 Feb 25 2008 lib

drwxrwxr-x 2 root root 1024 Feb 25 2008 libexec

lrwxrwxrwx 1 root root 11 Feb 25 2008 linuxrc → bin/busybox

drwx------ 2 root root 12288 Feb 25 2008 lost+found

drwxr-xr-x 5 root root 1024 Jun 20 03:22 mnt

dr-xr-xr-x 65 root root 0 Jan 1 1970 proc

drwx------ 2 root root 1024 Feb 25 2008 sbin

lrwxrwxrwx 1 root root 11 Jun 20 03:22 tmp → /mnt/fs/tmp

drwxr-xr-x 8 root root 1024 Nov 10 2008 usr

drwxr-xr-x 9 root root 1024 Nov 10 2008 var

cd /sbin

ls -al

drwx------ 2 root root 1024 Feb 25 2008 .

drwxr-xr-x 14 root root 1024 Jun 20 03:22 …

-rwxrwxr-x 1 root root 26708 Feb 25 2008 arp

-rwxrwxr-x 1 root root 11580 Feb 25 2008 arping

lrwxrwxrwx 1 root root 14 Feb 25 2008 freeramdisk → …/bin/busybox

lrwxrwxrwx 1 root root 16 Feb 25 2008 getty → …/bin/tinylogin

lrwxrwxrwx 1 root root 14 Feb 25 2008 halt → …/bin/busybox

-rwxr-xr-x 1 root root 10032 Feb 25 2008 hostname

lrwxrwxrwx 1 root root 14 Feb 25 2008 ifconfig → …/bin/busybox

lrwxrwxrwx 1 root root 14 Feb 25 2008 init → …/bin/busybox

-rwxrwxr-x 1 root root 148188 Feb 25 2008 insmod

-rwxrwxr-x 1 root root 136500 Feb 25 2008 ip

-rwxrwxr-x 1 root root 22332 Feb 25 2008 klogd

lrwxrwxrwx 1 root root 14 Feb 25 2008 loadkmap → …/bin/busybox

lrwxrwxrwx 1 root root 14 Feb 25 2008 logread → …/bin/busybox

lrwxrwxrwx 1 root root 14 Feb 25 2008 losetup → …/bin/busybox

lrwxrwxrwx 1 root root 14 Feb 25 2008 lsmod → …/bin/busybox

lrwxrwxrwx 1 root root 14 Feb 25 2008 makedevs → …/bin/busybox

lrwxrwxrwx 1 root root 14 Feb 25 2008 mkswap → …/bin/busybox

lrwxrwxrwx 1 root root 14 Feb 25 2008 modprobe → …/bin/busybox

-r-xr-xr-x 1 root root 96488 Feb 25 2008 pam_console_apply

-rwxr-xr-x 1 root root 8480 Feb 25 2008 pam_tally

lrwxrwxrwx 1 root root 14 Feb 25 2008 poweroff → …/bin/busybox

-r-sr-xr-x 1 root root 16520 Feb 25 2008 pwdb_chkpwd

lrwxrwxrwx 1 root root 14 Feb 25 2008 reboot → …/bin/busybox

lrwxrwxrwx 1 root root 14 Feb 25 2008 rmmod → …/bin/busybox

lrwxrwxrwx 1 root root 14 Feb 25 2008 route → …/bin/busybox

lrwxrwxrwx 1 root root 16 Feb 25 2008 sulogin → …/bin/tinylogin

lrwxrwxrwx 1 root root 14 Feb 25 2008 swapoff → …/bin/busybox

lrwxrwxrwx 1 root root 14 Feb 25 2008 swapon → …/bin/busybox

-rwxrwxr-x 1 root root 29572 Feb 25 2008 syslogd

-rwxr-xr-x 1 root root 369236 Feb 25 2008 tcpdump

-r-sr-xr-x 1 root root 18364 Feb 25 2008 unix_chkpwd

lrwxrwxrwx 1 root root 14 Feb 25 2008 update → …/bin/busybox

-rwxrwxr-x 1 root root 67796 Feb 25 2008 visudo

lrwxrwxrwx 1 root root 14 Feb 25 2008 watchdog → …/bin/busybox

The last good known version for this product is version 5.1.1 that comes with FW 1.0.5 (i believe…)

Not long ago i placed a package for Justin Cliff with the recent code and documents:

ftp server: ftp.support.mellanox.com

user: 9024-d

pass: 9024-d_eol

Give it a shot.

Well Done! And good job. looks very good.

About the alert: what this switch is doing is: it sweeps the entire network and alerts if it finds a switch with an “different” FW revision then this current switch which is reasonable to see these days. So, pay attention to what’s going on around the switch and safely ignore this error when needed.

Mellanox Interconnect Community <Infrastructure & Networking - NVIDIA Developer Forums Infrastructure & Networking - NVIDIA Developer Forums >

You have been mentioned

by Justin Clift<Infrastructure & Networking - NVIDIA Developer Forums Infrastructure & Networking - NVIDIA Developer Forums > in Re: Re: Mellanox (old Voltaire) ISR9024D-M recover flash area in Mellanox Interconnect Community - View Justin Clift’s reference to you<Infrastructure & Networking - NVIDIA Developer Forums Infrastructure & Networking - NVIDIA Developer Forums >

Hi,

as for your 9024D-M - maybe the SW image is corrupted. still, you can use this switch as an externally managed one (run SM somewhere else) - the switch will still do the switching work fine. if you want to get another switch, that’s fine too.

as for updating the FW remotely - not sure if you can do that. i think that the ASIC on those switches is write protected for FW writes from remote but give it a try.