Flexboot failed to boot from iSCSI via iSER

Infrastructure & Networking Adapters and Cables InfiniBand/VPI Adapter Cards
, , , , , ,
1

Hi all,
I successfully installed operating system on iSCSI(via iSER) target, but it won’t boot from it.
Before I get into Flexboot shell and try to manual sanhook and sanboot, I tried to ping the IP address of my SAN, it times out and it won’t even ping it is own address! I tried to do ifopen and it shows me this:
ConnectX3 0x31ae4 command 0x11 failed with status 01:

ConnectX3 0x31ae4 could not write MTT at 200

ConnectX3 0x31ae4 port 1 could not create completion queue

Could not open net0: Not enough space

Please help!
VPI card: Connectx3-FCBT
FW version: 2.42.5000
Flexboot version: 3.4.752
I know it is pretty old, but that’s the only thing I can afford. Please help me! Let me know for any information you need!

2

Yes, CX-3 is indeed a pretty old NIC.

Anyway, did you try to increase the MTT size?
(Notes: increasing MTT would bring latency performance issues)

Here is how to increase it:
log_num_mtt
The default number of log_num_mtt is 20

Check the parameter value
cat /sys/module/mlx4_core/parameters/ log_num_mtt (parameters range – (0-22))

Change log_num_mtt value

  1. echo X > /sys/module/mlx4_core/parameters/ log_num_mtt
  2. These parameters are set on the mlx4_core module in /etc/modprobe.conf or (/etc/modprobe.d/mlx4_core.conf)
    You can set its value for example by adding this line à options mlx4_core log_num_mtt=20

log_mtts_per_seg
The default number of log_mtts_per_seg is 0

Check the parameters value
cat /sys/module/mlx4_core/parameters/log_mtts_per_seg (parameters range –(0-7) )

Change log_num_mtt value

  1. echo X > /sys/module/mlx4_core/parameters/log_mtts_per_seg
  2. These parameters are set on the mlx4_core module in /etc/modprobe.conf or (/etc/modprobe.d/mlx4_core.conf)
    You can set its value for example by adding this line à options mlx4_core log_mtts_per_seg =7
3

It is the only thing I can afford.

I compiled and burnt my own iPXE(I did that couple days ago, in latest version from Github) firmware into it(on my client server not on my SAN), this problem is gone. Seems like a bug. I lost the access to TUI interface.
But it still won’t boot over iSER. When I execute the command sanboot -d 0x80 after I successfully did sanhook, it just stucks forever and my san(iSER SCST) shows me “timeout after 30 seconds” and automatically kills the iSCSI connection.

4

@e.palmer I am having the same issue with you(Multiple issues with ConnectX-3 CX311A-XCAT firmware. - #3 by MvB) and I am using MCX354A-FCBT now. Compile new iPXE can mitigate the issue but won’t solve.

5

By the way, where is the source code of flexboot for CX4 to CX7?