iSER for ESXi 6.5 No target detected, no traffic sent.


I have seen other reports of no targets being detected. I have esxi 6.5 host running the new iser driver, the adapter firmware is up to date (dual port connectx-4 vpi card in eth mode).

I can ping from the mellanox card to the esxi host. When i configure a target no devices are detected.

I loaded the newest mellanox on my linux target and followed the following guide: How-To Dump RDMA traffic Using the Inbox tcpdump tool (ConnectX-4)

How-To Dump RDMA traffic Using the Inbox tcpdump tool (ConnectX-4)

When i run a packet capture and rescan the iser adapter i see no traffic generated and sent to the linux target server. I logged just 2 cdp packets unrelated to the adapter refresh.

I have the esxi server directly connected into the linux server, so there is no switch currently and thus not alot of other traffic on this interface, but surely i should see the esxi server attempt to discover the targets?

the only thing i noticed was that the path is listed as not used, but i believe this is because there is no target detected:

should iser target port be something different than 3260?

How did you bind the iscsi adapter for this to work? Is it associated to the same vmnic as the iser hba? did you need to associate a vmkernel adapter with the iscsi initiator at all? If so did you assign an ip address to the iscsi vmkernel? I just dont want to give it a true iscsi path to the target if i do not have to.

RDMA vs TCP/IP = latency war …

You must check latency in test with another tool with multi-user configurations.

I’m also test it.

By increase client count, there are huge deffrence in latency factor on same throughput usage.


Jae-Hoon Choi

It’s a ZFS memory cache ARC range.

You must increase test sample size to 100GB or above.

Best Practice is using iometer with multi-user configurations…:)


Jae-Hoon Choi

i do not have a switch, they are working back to back though. there are also a few examples on mellanox’s site of direct connect servers as part of their demo. You can configure global pause and pfc in firmware, outside of OS. not sure why this would not work back to back if both adapters are sending pfc/global pause info?

once i enabled iscsi software adapter i was able to connect to my scst target. It is working now. Not really performing any better though. I was able to get better performance in iscsi by configuring 4 vmkernel adapters to my 1 vnic and then configuring round robin policy to 1 iops. It was the same adapter but it seemed to trick esxi into dedicating more hardware resources/scheduler to the adapter. At the moment the iser connection is on par with iscsi before i did this round robin policy. It doesnt look like i can configure round robin policy for iser adapters, but i am still looking into that.

I’m Sorry!

iSER initiator only works with Global Pause.

It doesn’t works with PFC on VMware environment.

If you mean that you just using PFC based RDMA network, you must have a Enterprise class ethernet switch that support PFC.

I have 2 of Mellanox SX6036G Gateway switches.

What’s your switch model?


Jae-Hoon Choi


RDMA must need a switched fabric like FCoE.

If you using direct connection between CX-4, you can’t do it!

Yes! CX-4 has a advanced unique logic on firmware over CX-3, but that logic must co-works with switch’s firmware and configuration.

RoCE RDMA is a kernel bypass protocol.

If your any HCA is down, you meet a kernel dump like ESXi PSOD or Linux kernel dump.

If there isn’t switch between host and storage, who control congestion or etc?

This link HowTo Configure PFC on ConnectX-4 that you mentioned previous message include basic system requirements.


Jae-Hoon Choi

Just add a VMware iSCSI adapter.

vmknic must bind iSER initiator, not VMware iSCSI initiator.

IP address also set to vmknic,too.

This test was completed with StarWind vSAN iSER target and RoCE v1 with Global Pause (access port in Arista switch)

This iSER driver for Global Pause(RoCE v1).

Mellanox said this stuff for ESXi 6.5, but QS shows a old ESXi 6.0 C# client picture and ethernet switch must set every port up to Global Pause mode.

This manual is very useless…:(

PFC on ESXi need some configuration with pfcrx, pfctx. But setup the default priority 3 to pfcrx, pfctx to 0x08, you will meet a Host Profile creation error message that general system error. Mellanox never provide a fix since driver


Jae-Hoon Choi

Unfortunately, iSER isn’t beta level.

It’s a alpha level quality.

Only 1st port can connect to iSER target.

And Software iSCSI initiator must be add to ESXi 6.5U1 host.

If you don’t do it, iSER initiator can’t connect to iSER target.

At last, whenever host reboot, iSER initiator disappeared.

You must excute esxcli rdma iser add command to host then rescan HBA adapter everytime.

We are not a beta tester.

We are not a QC employee on Mellanox or VMware,too.

If iSER will be a dominate RDMA storage protocol, It must be add a iSER eanble option to VMware software iSCSI initiator.

Not a funky CLI command.

PFC options and iSER initiator add to ESXi then Host Profile creation work failed everytime since IB iSER beta driver

Various bugs were exist on iSER initiator and PFC option on drivers.

Don’t use Mellanox HCA - especially iSER initiator alpha driver - on VMware production environment!

Best Regard.

Jae-Hoon Choi

can you provide any guidance for esxi pfc setup? i followed this on my linux target but all of the commands seem to not apply to esxi: HowTo Configure PFC on ConnectX-4

Here is a result from a vm, it has 2 cpus, 8gb and running 2016. It has a vmxnet3 adapter and vmware paravirtual iscsi hard drive.

These results are slightly better than what i was able to achieve with iscsi, so there is some improvement. I understand a switch is recommended but it clearly does work without one. I am monitoring for any packet loss/data issues but as explained before its point to point. The target is 3 nvme pciessds in what amounts to a raid 0 in a zfs pool/zvol btw.

Understood, i am testing the interconnect though, where its coming from is a bit irrelevant. Just saying that iser is performing somewhat better than iscsi so it is worth going through this hassle for those sitting on the sidelines wondering =). I was never able to get above 8800 or so on iscsi. The non sequential are very close, not a huge difference.