Xenserver 6.2 LVMoISER

Hi,

Good News

This is my current status :

Done: load all required modules for iser transport

Done: Creating storages type : LVM over iSCSI , but with transport iser , in XenCenter

Only the transport was changed from tcp to iser .

All XenServer features like:

  • Live Storage Migration

  • XenMotion ® live migration

  • VM Copy / Move

are still available

A test with a VM :

win2008r2 - 1024MB Ram , 1CPU , Disk C, E , F ( XENSRC PVDisk SCSI Disk Device)

Atto Disk Benchmark v2.47 default prefs , but Queue Depth : 10

max iops read> 45000

max iops write > 45000

RMB / s> 1300

WMB / s> 1300

This is f : in Dom0

[root @ xenserver ~ ] # iostat -x - m 1 sdc | grep sdc

sdc0.00 0.00 0.00 45123.00 0.00 22.07 1.00 1.63 0.04 0.01 64.00

sdc 0.00 0.00 45918.00 0.00 22.42 0.00 1.00 1.61 0.04 0.01 65.00

see attached files

# # #Bad news :

The server is not stable. At high I / O load , the server crashed .

Here is the error message:

<3 > [ 434.554641 ] iser : iser_start_rdma_unaligned_sg : Failed to allocate mem size 66 266240 for copying SGList

<3 > [ 434.592026 ] iser : iser_prepare_write_cmd : Failed to register write cmd RDMA mem

<3 > [ 434.629054 ] iser : iser_send_command : conn e8c3b0c8 failed task- > itt 83 err -12

<3 > [ 434.648441 ] PCI - DMA: Out of SW- IOMMU space for 266240 bytes at device 0000:03:00.0

<0 > [ 434.686043 ] ------------ [ cut here] ------------

<2 > [ 434.704686 ] kernel BUG at / tmp/mlnx_iso.24095/OFED_topdir/BUILD/ofa_kernel-1.5.3/drivers/infiniband/ulp/iser/iser_memory.c : 98 !

<0 > [ 434.743223 ] invalid opcode : 0000 [# 1 ] SMP<0 > [ 434.743224 ] load sysfs file: / sys/class/iscsi_session/session2/targetname

Maybe someone an idea how I can solve the problem?

more info in the file error.txt

error.txt.zip (1.66 KB)

lvm-lun-256.txt.zip (3.43 KB)

I’m at home and could not login from here.

By the way my english is bad… sorry.

Openiscsi - iscsid, iscsiadm - base Xenserver install.

!!! xenserverkernel6.2 added by script - save as tgz - extract to usb-disk !!!

Mount my usb disk and /mnt/mlnx…install - you know - i see errors and install bla and fasel with yum :-)

Another /mnt/mlnx…install - No errors - good

Rpm -i /mnt/RPM/kernel…xen.rpm

Vi /etc/infiniband/openibd.conf - yes/no - you know

Chkfg --level 2345 on opensm

Reboot

Lsmod - mlnx*, ib_* - ok

Ibstat - ok

Ifconfig -a - i see ib0

Vi /etc/sysconfig/net…-scripts/chk-ib0 - static ip …

Ifup ib0

Storage is omnios - insert connectx card - ! opensm on linux/switch is running! config crossbow/comstar …

Xenserver console:

iscsiadm -m discover … -I iser

iscsiadm -m node -L all

iscsiadm m session:

iser: …

I’ m local in Berlin/Germany - so i say good night now…

Open-iscsi version from xenserver6.2 install iso. The Iser interface option is part of it.

[ XX] Loading iSCSI transport class v2.0-870.

[ XX] iscsi: registered transport (iser)

[ XX] iscsi: registered transport (tcp)

dmesg looks good.

<3>[ XX] iser: iser_start_rdma_unaligned_sg:Failed to allocate mem size 66 266240 for copying sglist

<3>[ XX] iser: iser_prepare_write_cmd:Failed to register write cmd RDMA mem

<3>[ XX] iser: iser_send_command:conn e8c3b0c8 failed task->itt 83 err -12

<3>[ XX] PCI-DMA: Out of SW-IOMMU space for 266240 bytes at device 0000:03:00.0

google “PCI-DMA: Out of SW-IOMMU space”

  • high I/O

so i set … dom0 cmdline swiotlb=256

[ XX] Software IO TLB enabled:

[ XX] Aperture: 256 megabytes

[ XX] Address size: 29 bits

[ XX] Kernel range: c3976000 - d3976000

[ XX] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)

[ XX] PCI-DMA: mask is set to 38 bits

and again a crash

<0>[ 434.686043] ------------[ cut here ]------------

<2>[ 434.704686] kernel BUG at /tmp/mlnx_iso.24095/OFED_topdir/BUILD/ofa_kernel-1.5.3/drivers/infiniband/ulp/iser/iser_memory.c:98!

<0>[ 434.743223] invalid opcode: 0000 [#1] SMP

<0>[ 434.743224] last sysfs file: /sys/class/iscsi_session/session2/targetname

/**

  • iser_start_rdma_unaligned_sg

*/

static int iser_start_rdma_unaligned_sg(struct iscsi_iser_task *iser_task,

enum iser_data_dir cmd_dir)

{

int dma_nents;

struct ib_device *dev;

char *mem = NULL;

struct iser_data_buf *data = &iser_task->data[cmd_dir];

unsigned long cmd_data_len = data->data_len;

if (cmd_data_len > ISER_KMALLOC_THRESHOLD)

mem = (void *)__get_free_pages(GFP_ATOMIC,

ilog2(roundup_pow_of_two(cmd_data_len)) - PAGE_SHIFT);

else

mem = kmalloc(cmd_data_len, GFP_ATOMIC);

if (mem == NULL) {

iser_err(“Failed to allocate mem size %d %d for copying sglist\n”,

data->size,(int)cmd_data_len);

return -ENOMEM;

}

if (cmd_dir == ISER_DIR_OUT) {

/* copy the unaligned sg the buffer which is used for RDMA */

struct scatterlist *sgl = (struct scatterlist *)data->buf;

struct scatterlist *sg;

int i;

char *p, *from;

p = mem;

for_each_sg(sgl, sg, data->size, i) {

from = kmap_atomic(sg_page(sg), KM_USER0);

memcpy(p,

from + sg->offset,

sg->length);

kunmap_atomic(from, KM_USER0);

p += sg->length;

}

}

sg_init_one(&iser_task->data_copy[cmd_dir].sg_single, mem, cmd_data_len);

iser_task->data_copy[cmd_dir].buf =

&iser_task->data_copy[cmd_dir].sg_single;

iser_task->data_copy[cmd_dir].size = 1;

iser_task->data_copy[cmd_dir].copy_buf = mem;

dev = iser_task->iser_conn->ib_conn->device->ib_device;

dma_nents = ib_dma_map_sg(dev,

&iser_task->data_copy[cmd_dir].sg_single,

1,

(cmd_dir == ISER_DIR_OUT) ?

DMA_TO_DEVICE : DMA_FROM_DEVICE);

98 is here >> BUG_ON(dma_nents == 0);

iser_task->data_copy[cmd_dir].dma_nents = dma_nents;

return 0;

}

I see

if (mem == NULL) {

iser_err(“Failed to allocate mem size %d %d for copying sglist\n”,

data->size,(int)cmd_data_len);

return -ENOMEM;

}

result <3>[ XX] iser: iser_start_rdma_unaligned_sg:Failed to allocate mem size 66 266240 for copying sglist

what can you do if the memory allocate failed?

Perhabs wait and try again later?

Preview Private Cloud meets ZFS on Infiniband

Which version of open-iscsi are you using?

You would have had to graft in iser code into XS version of open-iscsi as this was stripped out by Citrix.

You need to carefully match the open-iscsi version with the iser code base else you will get compatibility problems.

Are you sure iser transport is created on target login? There is no iser modules within XS open-iscsi, so whilst the kernel modules loads, open-iscsi has no means to make a connection over iser.

Can you try logging out of iscsi targets, and try from the command link to log into target like this.

iscsiadm -m discovery -t st -p 10.0.0.1 -I iser

iscsiadm -m node -T iqn.2010-10.com.example:storage-1000 -l

Let me know how it connects.