Question about ESXi 6.5 iSER PFC direct connection

1

I have connectx-4 cards currently directly connected to each other. My one host is running a scst storage target and the other is running esxi 6.5 u1. I see the iser driver has been released and i noted the pfc config mentioned. Since there is no switch between the cards, they are direct connect, is pfc needed? If so is there anything i could configure on the nics themselves to meet this requirement? The connectx-4 direct connections are used for storage traffic only. The linux target is running scst and has a dual port card. The two esxi hosts each plug into one of these dual ports. this has worked well for 100gb ethernet (and originally 56gb srp with connectx-3 cards). I would like to try iser. If anyone has a way to mimic DCB on the linux side (most likely solution) any insight would be appreciated.

2

Here is a basic question about your environment.

If you want to use ethernet iSER you should be need PFC enabled ethernet configuration.

Then you must configure ethernet port on your Linux target with PFC.

It also need PFC configuration on ethernet switch, too!

Flow control for loseless ethernet traffic is a basic component on iSER fabric or RoCE.

Best Regard,

Jae-Hoon Choi

3

Hmm i wonder if there is any software i can run on the linux target to emulate this functionality. Is there a true need for PFC in a nic to nic connection though?

4

At first, I’m not a Mellanox employee.

But some information on this site show me a hintbellows.

01.ESXi 6.5 native driver

ESXi 6.5 native driver can’t support vKernel driver like SRP, IPoIB and infiniband based iSER.

Therefore officially Ethernet driver supported in future.

02.Infiniband based SRP vs iSER

SRP just need SM on Target or Initiator.

Therefore you can build SRP fabtic with direct connection between HCAs.

Also Infiniband based iSER, too!

03.Ethernet based (or RoCE) iSER

RoCE must need a loseless ethernet fabric.

For example like DCB(X), ETS.

  • It’s a extention of ethernet.

All Converged Ethernet protocol like FCoE, RoCE relay on switched fabric only.

Ethernet iSER need a RoCE network.

That’s a basic mandatory.

Best Regard,

Jae-Hoon Choi

5

iSER protocol based on RoCE.

RoCE need a looseless network that based on PFC or GP.

Unfortunately you can’t do it.

I think you should purchase a PFC based ethernet switch.

Best Regard,

Jae-Hoon Choi

6

are you sure this wont work at all? since they are directly connected there is no real way for outside influences to affect the throughput. I would assume with 2 cards direct connected one cannot outsend the other etc. I did the same thing with srp without issue. I am currently doing it was iscsi as well. I will try anyways but i am holding off trying for now since everyone is reporting they cannot get iser to see their scst or lio targets anyways…

7

there is no switch in the storage path in this case, just on the data network. lldpad daemon may work, i havent started testing because i see users with normal roce setups are having problems connecting to scst, i didnt see a need to try my setup until the normal setups work =)