Innova-2: MLNX_OFED > 5.2 has no mlx5_fpga_tools so innova2_flex_app fails

The latest MLNX_OFED release to include mlx5_fpga_tools.ko is 5.2-2.2.3.0. It is required by innova2_flex_app to burn FPGA configuration bitstream images, select the image to use, and enable/disable JTAG. I would like to use a newer version of MLNX_OFED.

I was able to trace innova2_flex_app as I made configuration changes and found that the program changes a ConnectX register:

cd /sys/kernel/tracing
echo 'p:myprobe mlx5_core_access_reg dev=$arg1:x64 data_in=$arg2:x64 di0=+0($arg2):x32 di1=+4($arg2):x32 di2=+8($arg2):x32 di3=+12($arg2):x32 size_in=$arg3:s32 data_out=$arg4:x64 size_out=$arg5:s32 reg_id=$arg6:x16 arg=$arg7:s32 write=$arg8:s32' > kprobe_events
echo 1 > events/kprobes/myprobe/enable
echo 1 > tracing_on
## Enable JTAG Access in innova2_flex_app ##
cat trace | grep myprobe

myprobe: (mlx5_core_access_reg+0x0/0x130 [mlx5_core]) dev=0xffff98bed2b40140 data_in=0xffffbc6c03283d08 di0=0x900di1=0x0 di2=0x0 di3=0x0 size_in=16 data_out=0xffffbc6c03283d18 size_out=16reg_id=0x4023arg=0write=1

After the register is changed, I can see the change using mlxreg:

sudo mst start
sudo mst status
sudo mlxreg -d /dev/mst/mt4119_pciconf0 --reg_id 0x4023 --reg_len 0x8 --get

mlxreg after Enable JTAG

Using the FPGA ConTRoL bit defines:

I should be able to set the operation[0x8] bits to disconnect the ConnectX-5 from the FPGA’s JTAG. The status[0x8] bits should then be 0x3. Same result as when using innova2_flex_app.

Unfortunately, I get a -E- Failed send access register: ME_ICMD_OPERATIONAL_ERROR when I try to set the register. I can set other registers like PAOS from the mlxreg examples.

sudo mlxreg -d /dev/mst/mt4119_pciconf0 --yes --reg_id 0x4023 --reg_len 0x4 --get
sudo mlxreg -d /dev/mst/mt4119_pciconf0 --yes --reg_id 0x4023 --reg_len 0x4 --set "0x0.16:4=0xA"
sudo mlxreg -d /dev/mst/mt4119_pciconf0 --yes --reg_id 0x4023 --reg_len 0x4 --set "0x0.16:4=0x9"
echo Try to set reserved bits:
sudo mlxreg -d /dev/mst/mt4119_pciconf0 --yes --reg_id 0x4023 --reg_len 0x4 --set "0x0.12:4=0x9"

Any ideas how to proceed with trying to set the register?

I recall innova had been EOL.

I recall innova had been EOL.

Unfortunately you are right. I found the EOL Notices:
Innova-2 Flex Open MNV303212A-ADLT and Innova-2 Flex Open VPI MNV303611A-EDLT
Innova-2 Flex EN MNV303212A-ADAT and Innova-2 IPsec EN MNV303212A-ADIT
Innova IPSec EN MNV101511A-BCIT and MNV101512A-BCIT

However, the boards are still very useful and I just need to be able to set a single register to regain all critical functionality. The original mlx5_fpga_tools did this in a complicated way with kernel modules to allow for additional features I do not need. Setting registers is why mlxreg exists.

mstreg from mstflint 4.26.0 is able to write the FPGA ConTRoL register.

Compile mstflint-4.26.0 and find the Innova-2 Ethernet device’s PCIe Address and the location of register_access_table.adb:

wget https://github.com/Mellanox/mstflint/releases/download/v4.26.0-1/mstflint-4.26.0-1.tar.gz
tar -xvf mstflint-4.26.0-1.tar.gz
cd mstflint-4.26.0
./configure --enable-adb-generic-tools --enable-rdmem
make
cd mlxreg/
lspci | grep Mellanox | grep Ethernet
sudo find / * -name register_access_table.adb

JTAG state returns to CONNECTed after a system (re)boot.

Disconnect to Enable JTAG by writing the 4-bit value 0x9 at the 16th bit position of word 0x0 of register 0x4023; --set "0x0.16:4=0x9":

sudo ./mstreg --yes --device 04:00.0 --adb_file /usr/share/mft/prm_dbs/hca/ext/register_access_table.adb --reg_id 0x4023 --reg_len 0x8 --get
sudo ./mstreg --yes --device 04:00.0 --adb_file /usr/share/mft/prm_dbs/hca/ext/register_access_table.adb --reg_id 0x4023 --reg_len 0x8 --set "0x0.16:4=0x9"
sudo ./mstreg --yes --device 04:00.0 --adb_file /usr/share/mft/prm_dbs/hca/ext/register_access_table.adb --reg_id 0x4023 --reg_len 0x8 --get

Connect to Disable JTAG with --set "0x0.16:4=0xA":

sudo ./mstreg --yes --device 04:00.0 --adb_file /usr/share/mft/prm_dbs/hca/ext/register_access_table.adb --reg_id 0x4023 --reg_len 0x8 --get
sudo ./mstreg --yes --device 04:00.0 --adb_file /usr/share/mft/prm_dbs/hca/ext/register_access_table.adb --reg_id 0x4023 --reg_len 0x8 --set "0x0.16:4=0xA"
sudo ./mstreg --yes --device 04:00.0 --adb_file /usr/share/mft/prm_dbs/hca/ext/register_access_table.adb --reg_id 0x4023 --reg_len 0x8 --get

Schedule the User Image to be active after the next (re)boot with --set "0x0.16:4=0x3,0x4.16:4=0x0":

sudo ./mstreg --yes --device 04:00.0 --adb_file /usr/share/mft/prm_dbs/hca/ext/register_access_table.adb --reg_id 0x4023 --reg_len 0x8 --get
sudo ./mstreg --yes --device 04:00.0 --adb_file /usr/share/mft/prm_dbs/hca/ext/register_access_table.adb --reg_id 0x4023 --reg_len 0x8 --set "0x0.16:4=0x3,0x4.16:4=0x0"
sudo ./mstreg --yes --device 04:00.0 --adb_file /usr/share/mft/prm_dbs/hca/ext/register_access_table.adb --reg_id 0x4023 --reg_len 0x8 --get

Schedule the Flex Image to be active after the next (re)boot with --set "0x0.16:4=0x3,0x4.16:4=0x1":

sudo ./mstreg --yes --device 04:00.0 --adb_file /usr/share/mft/prm_dbs/hca/ext/register_access_table.adb --reg_id 0x4023 --reg_len 0x8 --get
sudo ./mstreg --yes --device 04:00.0 --adb_file /usr/share/mft/prm_dbs/hca/ext/register_access_table.adb --reg_id 0x4023 --reg_len 0x8 --set "0x0.16:4=0x3,0x4.16:4=0x1"
sudo ./mstreg --yes --device 04:00.0 --adb_file /usr/share/mft/prm_dbs/hca/ext/register_access_table.adb --reg_id 0x4023 --reg_len 0x8 --get

I am exploring how the Factory+Flex Images work with innova2_flex_app. My current hypothesis is that the Factory Image (at 0x0) communicates with the ConnectX-5 to determine which configuration image, User or Flex, to load.

If you have a 1.8V Xilinx-compatible JTAG Adapter to recover from configuration failures or non-functional designs, you can ignore the Innova2 configuration system.

Add a Quad SPI IP Block to your designs and use XRT for configuration.

Include an M_AXI_LITE interface in your XDMA design:

Add a Quad SPI Block in Dual Quad Mode with 2 devices, a FIFO Depth of 256, and Use STARTUP Primitive Internal to IP:

Assign the block an address of 0x40000:

AXI_Lite Address

Connect usrcclkts to 0. The Block Diagram should look something like the following, plus the rest of your system.

Add the following to your project’s constraints .xdc file:

# Secondary Quad SPI Configuration Flash - Bank 65
# Primary Quad SPI Configuration Flash pins are single-purpose in STARTUPE3
set_property PACKAGE_PIN AM12     [get_ports spi_rtl_0_io0_io]
set_property IOSTANDARD  LVCMOS18 [get_ports spi_rtl_0_io0_io]
set_property PACKAGE_PIN AN12     [get_ports spi_rtl_0_io1_io]
set_property IOSTANDARD  LVCMOS18 [get_ports spi_rtl_0_io1_io]
set_property PACKAGE_PIN AR13     [get_ports spi_rtl_0_io2_io]
set_property IOSTANDARD  LVCMOS18 [get_ports spi_rtl_0_io2_io]
set_property PACKAGE_PIN AR12     [get_ports spi_rtl_0_io3_io]
set_property IOSTANDARD  LVCMOS18 [get_ports spi_rtl_0_io3_io]
set_property PACKAGE_PIN AV11     [get_ports spi_rtl_0_ss_io]
set_property IOSTANDARD  LVCMOS18 [get_ports spi_rtl_0_ss_io]

# Differential System Clock - 100MHz - Bank 65
set_property PACKAGE_PIN AR14        [get_ports {sys_clk_100MHz_clk_p[0]}]
set_property IOSTANDARD  DIFF_SSTL12 [get_ports {sys_clk_100MHz_clk_p[0]}]

After your project’s Synthesis+Implementation write your Memory Configuration File as an mcs file with a 0 Start Address. Use JTAG to initially write your configuration image to the Innova2.

Subsequently, the xbflash command can be used to program new bitstreams over existing ones.

lspci -d 10ee:
xbflash --card 3:00.0 --primary PROJECT_NAME_primary.mcs --secondary PROJECT_NAME_secondary.mcs

Write Memory Configuration File:

See also Programming a Configuration Memory Device.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.