Failed to burn an image onto Innova-2 Flex FPGA

Infrastructure & Networking Software And Drivers SoC And SmartNIC
1

I failed to burn an image onto Innova-2 Flex FPGA. Can anyone help?

$ sudo ./innova2_flex_app -v -b PCIe_DMA_wrapper.bin,0 
===============================================
 Verbosity:        1
 BOPE device:      None
 ConnectX device:  None
 Files to burn:
-------------------------------------------
 File name "PCIe_DMA_wrapper.bin"
 File length 12219172
 Aligned length 12219172
 Flash to burn 0
 Flash offset is default
 ConnectX device: /dev/0000:18:00.0_mlx5_fpga_tools
 BOPE device:     None
 Scheduled image:  User Image
 Running image:    Innova2 Flex Image(Success)
 Type of board: MorseQ

Burn-Diagnostics menu
------------------
[ 1 ] Query Innova2_Flex FPGA version
[ 3 ] PCI test
[ 4 ] Read FPGA thermal status
[ 5 ] Read Fan speed
[ 6 ] Burn of customer User image
[ 7 ] Set User image active (reboot required)
[ 8 ] Set Innova2_Flex image active
[ 9 ] Increase FPGA power consumption
[10 ] Enable JTAG Access - no thermal status
[99 ] Exit
Your choice: 6

	You are trying to burn an image using the Innova2_Flex image.
	Burning requires installing the factory driver.
	Please install the driver and try again```
2

You are missing the BOPE and ConnectX devices.

Try the following after a system reboot. Commands assume you have the Innova-2 Flex Open package extracted to ~ (home directory).

sudo mst start
sudo mst status
sudo mst status -v
sudo flint -d /dev/mst/mt4119_pciconf0 q
cd ~/Innova_2_Flex_Open_18_12/driver/
sudo ./make_device
sudo insmod /usr/lib/modules/`uname -r`/updates/dkms/mlx5_fpga_tools.ko
lsmod | grep mlx
cd ~
sudo ~/Innova_2_Flex_Open_18_12/app/innova2_flex_app -v

Check out my notes on using the Innova-2 for FPGA development.

3

The Innova-2 uses dual x4 FLASH ICs in parallel for faster loading/programming.

innova2_flex_app requires _primary.bin and _secondary.bin files. Note the ,0 and ,1.

sudo ~/Innova_2_Flex_Open_18_12/app/innova2_flex_app  -v  -b innova2_xcku15p_ddr4_bram_gpio_primary.bin,0  -b innova2_xcku15p_ddr4_bram_gpio_secondary.bin,1
4

So, I have to split my image img.bin into 2 separated images: img_primary.bin and img_secondary.bin to burn to the flash?

To do that, I splitted my image and got an error with this command:

$ sudo ~/innova2_flex_app -v -b img_primary.bin,0 img_secondary.bin,1
Error: wrong parameter "img_secondary.bin,1"

Then I added parameter -b to the command and burned the image:

$ sudo ./innova2_flex_app -v -b img_primary.bin,0 -b img_secondary.bin,1
===============================================
 Verbosity:        1
 BOPE device:      None
 ConnectX device:  None
 Files to burn:
-------------------------------------------
 File name "img_secondary.bin"
 File length 6109586
 Aligned length 6109588
 Flash to burn 1
 Flash offset is default
-------------------------------------------
 File name "img_primary.bin"
 File length 6109586
 Aligned length 6109588
 Flash to burn 0
 Flash offset is default
 ConnectX device: /dev/0000:18:00.0_mlx5_fpga_tools
 BOPE device:     /dev/0000:17:00.0_mlx_fpga_bope
 Scheduled image:  Innova2 Flex Image
 Running image:    Innova2 Flex Image(Success)
 Type of board: MorseQ
	***   FPGA image version: 0x8d   ***
	***   Mailbox Done counter   0   ***

Burn-Diagnostics menu
------------------
[ 1 ] Query Innova2_Flex FPGA version
[ 3 ] PCI test
[ 4 ] Read FPGA thermal status
[ 5 ] Read Fan speed
[ 6 ] Burn of customer User image
[ 7 ] Set User image active (reboot required)
[ 8 ] Set Innova2_Flex image active
[ 9 ] Increase FPGA power consumption
[10 ] Enable JTAG Access - no thermal status
[99 ] Exit
Your choice:

When I booted to the User image, the red light on the board turned on.
What did I do wrong?

5

Vivado generates the _primary.bin and _secondary.bin files during Memory Configuration File generation. I believe the data is interleaved.

Confirm your project’s constraints (.xdc) file includes the following which is from the official Innova-2 constraints package.

# Memory Configuration File Settings

set_property CONFIG_MODE SPIx8 [current_design]
set_property CONFIG_VOLTAGE 1.8 [current_design]
set_property CFGBVS GND [current_design]
set_property BITSTREAM.GENERAL.COMPRESS TRUE [current_design]
set_property BITSTREAM.CONFIG.CONFIGRATE 127.5 [current_design]
set_property BITSTREAM.CONFIG.SPI_32BIT_ADDR YES [current_design]
set_property BITSTREAM.CONFIG.SPI_BUSWIDTH 8 [current_design]
set_property BITSTREAM.CONFIG.SPI_FALL_EDGE YES [current_design]
set_property BITSTREAM.CONFIG.CONFIGFALLBACK DISABLE [current_design]
set_property BITSTREAM.CONFIG.UNUSEDPIN PULLUP [current_design]
set_property BITSTREAM.CONFIG.OVERTEMPSHUTDOWN ENABLE [current_design]
set_property BITSTREAM.CONFIG.EXTMASTERCCLK_EN DISABLE [current_design]
set_property BITSTREAM.GENERAL.CRC ENABLE [current_design]
set_property BITSTREAM.CONFIG.NEXT_CONFIG_REBOOT DISABLE [current_design]

Run Generate Bitstream in Vivado then Generate Memory Configuration File. Select bin, mt25qu512_x1_x2_x4_x8, SPIx8, Load bitstream files, and a location and name for the output binary files. The bitstream will end up, for example, in the DESIGN_NAME/DESIGN_NAME.runs/impl_1 subdirectory as SOMETHING.bit.

Vivado_Write_Memory_Configuration_File
Vivado_Write_Memory_Configuration_File600×567 69.7 KB

6

I do not see the file https://github.com/mwrnd/innova2_xcku15p_ddr4_bram_gpio/innova2_xcku15p_ddr4_bram_gpio.bin

Is it required to use specifically this file?
My board is MNV303611A-EDLT which does not have DDR4 onboard.

7

That was an image I took for illustrative purposes from these notes. You can name your .bin output file whatever you want. Vivado will then create _primary.bin and _secondary.bin files from your project’s Bitstream .bit file. It will add _primary.bin and _secondary.bin to NAME.bin.

You can use git to clone this demo project which contains a complete bitstream for the Innova-2.

sudo mst start
sudo mst status
sudo mst status -v
sudo flint -d /dev/mst/mt4119_pciconf0 q
cd ~/Innova_2_Flex_Open_18_12/driver/
sudo ./make_device
sudo insmod /usr/lib/modules/`uname -r`/updates/dkms/mlx5_fpga_tools.ko
cd ~

git clone https://github.com/mwrnd/innova2_xcku15p_ddr4_bram_gpio
cd innova2_xcku15p_ddr4_bram_gpio

sudo ~/Innova_2_Flex_Open_18_12/app/innova2_flex_app -v \
  -b innova2_xcku15p_ddr4_bram_gpio_primary.bin,0       \
  -b innova2_xcku15p_ddr4_bram_gpio_secondary.bin,1

After loading the User Image using innova2_flex_app, reboot your system and try testing the design. The AXI BRAM and AXI GPIO LED tests should work with the MNV303611A-EDLT as long as the DDR4 block gets out of reset. If the design fails, you can open the project in Vivado, delete the DDR4 block, and re-implement the design.

8

Here is the info after loading the User image. It looks like I cannot access the BRAM. The board is plugged into an enclosed chassis that I cannot see the leds on the board.

notooth@192:~$ lspci | grep -i Xilinx
17:00.0 RAM memory: Xilinx Corporation Device 9038

notooth@192:~$ sudo lspci  -s 17:00  -v
17:00.0 RAM memory: Xilinx Corporation Device 9038
	Subsystem: Xilinx Corporation Device 0007
	Flags: bus master, fast devsel, latency 0, IRQ 55
	Memory at fbff0000 (32-bit, non-prefetchable) [size=64K]
	Capabilities: [40] Power Management version 3
	Capabilities: [48] MSI: Enable+ Count=1/1 Maskable- 64bit+
	Capabilities: [70] Express Endpoint, MSI 00
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [1c0] Secondary PCI Express
	Kernel driver in use: xdma
	Kernel modules: xdma, qdma_pf

notooth@192:tools$ sudo lspci  -s 17:00  -vvv | grep "LnkCap\|LnkSta"
		LnkCap:	Port #0, Speed 8GT/s, Width x8, ASPM not supported
		LnkSta:	Speed 2.5GT/s (downgraded), Width x8 (ok)
		LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1-

notooth@192:~$ dmesg | grep -i xdma
[    7.454592] xdma:xdma_mod_init: Xilinx XDMA Reference Driver xdma v2020.2.2
[    7.454597] xdma:xdma_mod_init: desc_blen_max: 0xfffffff/268435455, timeout: h2c 10 c2h 10 sec.
[    7.455275] xdma:xdma_device_open: xdma device 0000:17:00.0, 0x000000004a5208ec.
[    7.455640] xdma:map_single_bar: BAR0 at 0xfbff0000 mapped at 0x000000004f9f69c7, length=65536(/65536)
[    7.455649] xdma:map_bars: config bar 0, pos 0.
[    7.455651] xdma:identify_bars: 1 BARs: config 0, user -1, bypass -1.
[    7.456084] xdma:pci_keep_intx_enabled: 0000:17:00.0: clear INTX_DISABLE, 0x447 -> 0x47.
[    7.456124] xdma:probe_one: 0000:17:00.0 xdma0, pdev 0x000000004a5208ec, xdev 0x00000000c8a3e3ef, 0x00000000115364bf, usr 16, ch 4,4.
[  260.694225] xdma:xdma_xfer_submit: xfer 0x000000005f873514,8192, s 0x1 timed out, ep 0x200102000.
[  260.694242] xdma:engine_reg_dump: 0-H2C0-MM: ioread32(0x000000004f9f69c7) = 0x1fc00006 (id).
[  260.694247] xdma:engine_reg_dump: 0-H2C0-MM: ioread32(0x00000000f3c06a94) = 0x00000001 (status).
[  260.694252] xdma:engine_reg_dump: 0-H2C0-MM: ioread32(0x00000000dc6fb349) = 0x00f83e1f (control)
[  260.694256] xdma:engine_reg_dump: 0-H2C0-MM: ioread32(0x00000000f3027d24) = 0x60a10000 (first_desc_lo)
[  260.694259] xdma:engine_reg_dump: 0-H2C0-MM: ioread32(0x0000000046487f88) = 0x00000012 (first_desc_hi)
[  260.694263] xdma:engine_reg_dump: 0-H2C0-MM: ioread32(0x000000003cbc2fba) = 0x00000001 (first_desc_adjacent).
[  260.694267] xdma:engine_reg_dump: 0-H2C0-MM: ioread32(0x00000000a3763493) = 0x00000000 (completed_desc_count).
[  260.694271] xdma:engine_reg_dump: 0-H2C0-MM: ioread32(0x000000006ce392af) = 0x00f83e1e (interrupt_enable_mask)
[  260.694277] xdma:engine_status_dump: SG engine 0-H2C0-MM status: 0x00000001: BUSY
[  260.694280] xdma:transfer_abort: abort transfer 0x000000005f873514, desc 2, engine desc queued 0.
[  278.102106] xdma:xdma_xfer_submit: xfer 0x000000002d9919b5,8192, s 0x1 timed out, ep 0x200102000.
[  278.102122] xdma:engine_reg_dump: 0-C2H0-MM: ioread32(0x00000000e10506c8) = 0x1fc10006 (id).
[  278.102126] xdma:engine_reg_dump: 0-C2H0-MM: ioread32(0x000000000ab5c974) = 0x00000001 (status).
[  278.102131] xdma:engine_reg_dump: 0-C2H0-MM: ioread32(0x00000000a3cec005) = 0x00f83e1f (control)
[  278.102135] xdma:engine_reg_dump: 0-C2H0-MM: ioread32(0x00000000bf4fe70b) = 0x50a10000 (first_desc_lo)
[  278.102139] xdma:engine_reg_dump: 0-C2H0-MM: ioread32(0x0000000071e8af2c) = 0x00000012 (first_desc_hi)
[  278.102142] xdma:engine_reg_dump: 0-C2H0-MM: ioread32(0x000000003f6ba3fe) = 0x00000001 (first_desc_adjacent).
[  278.102146] xdma:engine_reg_dump: 0-C2H0-MM: ioread32(0x000000009ee83017) = 0x00000000 (completed_desc_count).
[  278.102150] xdma:engine_reg_dump: 0-C2H0-MM: ioread32(0x00000000b47b6623) = 0x00f83e1e (interrupt_enable_mask)
[  278.102156] xdma:engine_status_dump: SG engine 0-C2H0-MM status: 0x00000001: BUSY
[  278.102158] xdma:transfer_abort: abort transfer 0x000000002d9919b5, desc 2, engine desc queued 0.

notooth@192:~$ cd '/mnt/Archive/Downloads/dma_ip_drivers/XDMA/linux-kernel/tools' 

notooth@192:tools$ dd if=/dev/urandom bs=1 count=8192 of=TEST
8192+0 records in
8192+0 records out
8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0286262 s, 286 kB/s

notooth@192:tools$ sudo ./dma_to_device   --verbose --device /dev/xdma0_h2c_0 --address 0x200100000 --size 8192  -f    TEST
dev /dev/xdma0_h2c_0, addr 0x200100000, aperture 0x0, size 0x2000, offset 0x0, count 1
host buffer 0x3000 = 0x1cc3000
/dev/xdma0_h2c_0, write 0x2000 @ 0x200100000 failed -1.
write file: Unknown error 512

notooth@192:tools$ sudo ./dma_from_device --verbose --device /dev/xdma0_c2h_0 --address 0x200100000 --size 8192 --file RECV
dev /dev/xdma0_c2h_0, addr 0x200100000, aperture 0x0, size 0x2000, offset 0x0, count 1
host buffer 0x3000, 0x185a000.
/dev/xdma0_c2h_0, read 0x2000 @ 0x200100000 failed -1.
read file: Unknown error 512

notooth@192:tools$ sha256sum TEST RECV
d1cc324764693498a3ff2e60ad5d217c2c89c4db2cc9d01f5001fced80cf0e92  TEST
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  RECV

notooth@192:tools$ echo -n -e "\xff" >ff.bin   ;   od -A x -t x1z -v  ff.bin
000000 ff                                               >.<
000001

notooth@192:tools$ echo -n -e "\x00" >00.bin   ;   od -A x -t x1z -v  00.bin
000000 00                                               >.<
000001

notooth@192:tools$ ls -l  ff.bin  00.bin
-rw-rw-r--. 1 notooth notooth 1 May 28 11:21 00.bin
-rw-rw-r--. 1 notooth notooth 1 May 28 11:21 ff.bin

notooth@192:tools$ sudo ./dma_to_device   --verbose --device /dev/xdma0_h2c_0 --address 0x200110000 --size 1 -f     ff.bin
dev /dev/xdma0_h2c_0, addr 0x200110000, aperture 0x0, size 0x1, offset 0x0, count 1
host buffer 0x1001 = 0x168b000
/dev/xdma0_h2c_0, write 0x1 @ 0x200110000 failed -1.
write file: Unknown error 512

notooth@192:tools$ sudo ./dma_to_device   --verbose --device /dev/xdma0_h2c_0 --address 0x200110000 --size 1 -f     00.bin
dev /dev/xdma0_h2c_0, addr 0x200110000, aperture 0x0, size 0x1, offset 0x0, count 1
host buffer 0x1001 = 0x25c9000
/dev/xdma0_h2c_0, write 0x1 @ 0x200110000 failed -1.
write file: Unknown error 512

notooth@192:tools$ sudo ./dma_from_device --verbose --device /dev/xdma0_c2h_0 --address 0x200110000 --size 8 --file RECV
dev /dev/xdma0_c2h_0, addr 0x200110000, aperture 0x0, size 0x8, offset 0x0, count 1
host buffer 0x1008, 0x235a000.
/dev/xdma0_c2h_0, read 0x8 @ 0x200110000 failed -1.
read file: Unknown error 512

notooth@192:tools$ od -A x -t x1z -v  RECV
000000

notooth@192:tools$ sudo ./performance --device /dev/xdma0_h2c_0
'/dev/xdma0_h2c_0'
device = /dev/xdma0_h2c_0, size = 0x00008000, count = 1
IOCTL_XDMA_PERF_START succesful.
IOCTL_XDMA_PERF_GET succesful.
perf.transfer_size = 32768
perf.iterations = 0
(data transferred = 0 bytes)
perf.clock_cycle_count = 500041280
perf.data_cycle_count = 496
(data duty cycle = 0%)
IOCTL_XDMA_PERF_STOP succesful.
perf.transfer_size = 32768 bytes
perf.iterations = 0
(data transferred = 0 bytes)
perf.clock_cycle_count = 500051900
perf.data_cycle_count = 496
(data duty cycle = 0%)
 data rate ***** bytes length = 32768, rate = 0.000001 
perf.pending_count = 0

notooth@192:tools$ sudo ./performance --device /dev/xdma0_c2h_0
'/dev/xdma0_c2h_0'
device = /dev/xdma0_c2h_0, size = 0x00008000, count = 1
IOCTL_XDMA_PERF_START succesful.
IOCTL_XDMA_PERF_GET succesful.
perf.transfer_size = 32768
perf.iterations = 0
(data transferred = 0 bytes)
perf.clock_cycle_count = 500040604
perf.data_cycle_count = 0
IOCTL_XDMA_PERF_STOP succesful.
perf.transfer_size = 32768 bytes
perf.iterations = 0
(data transferred = 0 bytes)
perf.clock_cycle_count = 500050420
perf.data_cycle_count = 0
perf.pending_count = 0
9

Great debug logs. Very useful and well done.

The good news is that you have all the drivers successfully installed and the card and XDMA block within the FPGA are recognized.

The innova2_xcku15p_ddr4_bram_gpio demo project unfortunately uses the DDR4 clock for the AXI blocks. Source the design in Vivado then Open Block Design and delete the ddr4_0 and rst_ddr4_0_300M blocks. Then connect the xdma_0 block’s axi_aclk and axi_aresetn signals to the s_axi_aclk and s_axi_resetn signals of the BRAM and GPIO blocks. Also, delete any unused signals. Generate Bitstream and an MNV303611A-EDLT compatible bitstream should be generated.

You can also check out this tutorial for a step-by-step guide to creating an XDMA demo in the Vivado Block Designer.

10

Here is a simple XDMA demo for the MNV303611A-EDLT. Just AXI BRAM and one LED under GPIO control. Please test it and let me know if it works.

11

I think it works now. Thanks for your help.

notooth@192:~$ lspci | grep -i Xilinx
17:00.0 Memory controller: Xilinx Corporation Device 9038

notooth@192:~$ sudo lspci  -s 17:00  -v
17:00.0 Memory controller: Xilinx Corporation Device 9038
	Subsystem: Xilinx Corporation Device 0007
	Flags: bus master, fast devsel, latency 0, IRQ 55
	Memory at fbff0000 (32-bit, non-prefetchable) [size=64K]
	Capabilities: [40] Power Management version 3
	Capabilities: [48] MSI: Enable+ Count=1/1 Maskable- 64bit+
	Capabilities: [70] Express Endpoint, MSI 00
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [1c0] Secondary PCI Express
	Kernel driver in use: xdma
	Kernel modules: xdma, qdma_pf

notooth@192:~$ dmesg | grep -i xdma
[    7.366839] xdma:xdma_mod_init: Xilinx XDMA Reference Driver xdma v2020.2.2
[    7.366848] xdma:xdma_mod_init: desc_blen_max: 0xfffffff/268435455, timeout: h2c 10 c2h 10 sec.
[    7.367834] xdma:xdma_device_open: xdma device 0000:17:00.0, 0x000000006a9788b1.
[    7.368520] xdma:map_single_bar: BAR0 at 0xfbff0000 mapped at 0x00000000266bf17c, length=65536(/65536)
[    7.368532] xdma:map_bars: config bar 0, pos 0.
[    7.368535] xdma:identify_bars: 1 BARs: config 0, user -1, bypass -1.
[    7.368845] xdma:pci_keep_intx_enabled: 0000:17:00.0: clear INTX_DISABLE, 0x447 -> 0x47.
[    7.368875] xdma:probe_one: 0000:17:00.0 xdma0, pdev 0x000000006a9788b1, xdev 0x0000000059af4a5c, 0x000000004b3dfd43, usr 16, ch 4,4.

notooth@192:~$ cd ./dma_ip_drivers/XDMA/linux-kernel/tools/

notooth@192:tools$ echo -n -e "\xff" >ff.bin   ;   od -A x -t x1z -v  ff.bin
000000 ff                                               >.<
000001

notooth@192:tools$ echo -n -e "\x00" >00.bin   ;   od -A x -t x1z -v  00.bin
000000 00                                               >.<
000001

notooth@192:tools$ sudo ./dma_from_device --verbose --device /dev/xdma0_c2h_0 --address 0x40000000 --size 4 --file RECV ; xxd -b RECV
dev /dev/xdma0_c2h_0, addr 0x40000000, aperture 0x0, size 0x4, offset 0x0, count 1
host buffer 0x1004, 0x174c000.
#0: CLOCK_MONOTONIC 0.000147425 sec. read 4/4 bytes
** Avg time device /dev/xdma0_c2h_0, total time 147425 nsec, avg_time = 147425.000000, size = 4, BW = 0.027132 
/dev/xdma0_c2h_0 ** Average BW = 4, 0.027132
00000000: 11101001 00000000 00000000 00000000                    ....

notooth@192:tools$ sudo ./dma_from_device --verbose --device /dev/xdma0_c2h_0 --address 0x40000000 --size 4 --file RECV ; xxd -b RECV
dev /dev/xdma0_c2h_0, addr 0x40000000, aperture 0x0, size 0x4, offset 0x0, count 1
host buffer 0x1004, 0x196c000.
#0: CLOCK_MONOTONIC 0.000163964 sec. read 4/4 bytes
** Avg time device /dev/xdma0_c2h_0, total time 163964 nsec, avg_time = 163964.000000, size = 4, BW = 0.024396 
/dev/xdma0_c2h_0 ** Average BW = 4, 0.024396
00000000: 00101001 00000000 00000000 00000000                    )...

notooth@192:tools$ sudo ./dma_from_device --verbose --device /dev/xdma0_c2h_0 --address 0x40000000 --size 4 --file RECV ; xxd -b RECV
dev /dev/xdma0_c2h_0, addr 0x40000000, aperture 0x0, size 0x4, offset 0x0, count 1
host buffer 0x1004, 0x2541000.
#0: CLOCK_MONOTONIC 0.000107604 sec. read 4/4 bytes
** Avg time device /dev/xdma0_c2h_0, total time 107604 nsec, avg_time = 107604.000000, size = 4, BW = 0.037173 
/dev/xdma0_c2h_0 ** Average BW = 4, 0.037173
00000000: 00110001 00000000 00000000 00000000                    1...

notooth@192:tools$ sudo ./dma_from_device --verbose --device /dev/xdma0_c2h_0 --address 0x00000000 --size 4 --file RECV ; xxd -b RECV
dev /dev/xdma0_c2h_0, addr 0x0, aperture 0x0, size 0x4, offset 0x0, count 1
host buffer 0x1004, 0xaea000.
#0: CLOCK_MONOTONIC 0.000155515 sec. read 4/4 bytes
** Avg time device /dev/xdma0_c2h_0, total time 155515 nsec, avg_time = 155515.000000, size = 4, BW = 0.025721 
/dev/xdma0_c2h_0 ** Average BW = 4, 0.025721
00000000: 00010000 00010001 00010010 00010011                    ....

notooth@192:tools$ sudo ./dma_to_device   --verbose --device /dev/xdma0_h2c_0 --address 0x00000000 --size 1  -f    ff.bin
dev /dev/xdma0_h2c_0, addr 0x0, aperture 0x0, size 0x1, offset 0x0, count 1
host buffer 0x1001 = 0x2599000
#0: CLOCK_MONOTONIC 0.000118067 sec. write 1 bytes
** Avg time device /dev/xdma0_h2c_0, total time 118067 nsec, avg_time = 118067.000000, size = 1, BW = 0.008470 
/dev/xdma0_h2c_0 ** Average BW = 1, 0.008470

notooth@192:tools$ sudo ./dma_from_device --verbose --device /dev/xdma0_c2h_0 --address 0x00000000 --size 4 --file RECV ; xxd -b RECV
dev /dev/xdma0_c2h_0, addr 0x0, aperture 0x0, size 0x4, offset 0x0, count 1
host buffer 0x1004, 0x6dd000.
#0: CLOCK_MONOTONIC 0.000177620 sec. read 4/4 bytes
** Avg time device /dev/xdma0_c2h_0, total time 177620 nsec, avg_time = 177620.000000, size = 4, BW = 0.022520 
/dev/xdma0_c2h_0 ** Average BW = 4, 0.022520
00000000: 11111111 00010001 00010010 00010011

notooth@192:tools$ cd ../tests

notooth@192:tests$ sudo ./run_test.sh
Info: Number of enabled h2c channels = 4
Info: Number of enabled c2h channels = 4
Info: The PCIe DMA core is memory mapped.
Info: Running PCIe DMA memory mapped write read test
	transfer size:  1024, count: 1
Info: Writing to h2c channel 0 at address offset 0.
Info: Writing to h2c channel 1 at address offset 1024.
Info: Writing to h2c channel 2 at address offset 2048.
Info: Writing to h2c channel 3 at address offset 3072.
Info: Wait for current transactions to complete.
/dev/xdma0_h2c_2 ** Average BW = 1024, 19.637548
/dev/xdma0_h2c_0 ** Average BW = 1024, 34.552570
/dev/xdma0_h2c_1 ** Average BW = 1024, 24.760614
/dev/xdma0_h2c_3 ** Average BW = 1024, 8.072272
Info: Reading from c2h channel 0 at  address offset 0.
Info: Reading from c2h channel 1 at  address offset 1024.
Info: Reading from c2h channel 2 at  address offset 2048.
Info: Reading from c2h channel 3 at  address offset 3072.
Info: Wait for current transactions to complete.
/dev/xdma0_c2h_2 ** Average BW = 1024, 12.910058
/dev/xdma0_c2h_3 ** Average BW = 1024, 8.150465
/dev/xdma0_c2h_0 ** Average BW = 1024, 5.995702
/dev/xdma0_c2h_1 ** Average BW = 1024, 8.220542
Info: Checking data integrity.
Info: Data check passed for address range  0 - 1024
Info: Data check passed for address range  1024 - 2048
Info: Data check passed for address range  2048 - 3072
Info: Data check passed for address range  3072 - 4096
Info: All PCIe DMA memory mapped tests passed.
Info: All tests in run_tests.sh passed.
1 Like
13

The XDMA Driver (Xilinx’s dma_ip_drivers) creates read-only and write-only character device files, /dev/xdma0_h2c_0 and /dev/xdma0_c2h_0, that allow direct access to the FPGA design’s AXI Bus. To read from an AXI Device at address 0x40010000 you would read from address 0x40010000 of the /dev/xdma0_c2h_0 (Card-to-Host) file. To write you would write to the appropriate address of /dev/xdma0_h2c_0 (Host-to-Card).

For example, you can read or write to the AXI BRAM Memory using dd. Note seek and skip are the Base10 address of the BRAM Memory on the FPGA design’s AXI Bus, which is 0 on the innova2_mnv303611a_xcku15p_xdma project. Try various read and write Block Sizes, bs, up to the maximum Range of the BRAM Memory as defined in the Vivado Address Editor. 128kb is the current maximum.

dd if=/dev/urandom bs=1 count=8192 of=TEST
sudo dd if=TEST of=/dev/xdma0_h2c_0 count=1 bs=8192 seek=0 oflag=seek_bytes
sudo dd if=/dev/xdma0_c2h_0 of=RECV count=1 bs=8192 skip=0 iflag=skip_bytes
md5sum TEST RECV

dd_AXI_BRAM
dd_AXI_BRAM800×339 55.9 KB

Here are more detailed notes including a simple C program that communicates with an XDMA design.

15

I changed the C code to write {6,7,8,9} to the BRAM,
but it returned a number different from 0x06070809 :
…
uint8_t read_data[DATA_SIZE];
…
printf("\nread_data: 0x%x\n", read_data);
…
Output:
read_data: 0x18b24058

read_data is the address of the array. There would have been an implicit type cast and gcc should have complained if run with -Wall. Add an index value for each byte you want to print.

printf("read_data = 0x%02X, wrte_data = 0x%02X\n", read_data[indx], wrte_data[indx]);
16

Is there a demo of XDMA Stream?

17

Is there a demo of XDMA Stream?

None that I am aware of. Search for xdma stream and look through the XMDA Guide. I would get something working with the basic XDMA driver before adding complexity.

18

Here is a tutorial for XDMA Stream Software and Hardware.

Check out Xilinx’s PCIe Design Hub.