GIDs for connectx-6 devices are not showing

Hello, GIDs for mellanox card (connectx-6) are not showing.
Any idea what could be wrong, or a direction to look into to resolve this issue?

Hi,
What tool are you use?
Could you provide the result of ibstatus? ibv_devinfo -v

Thanks,
Suo

Sure
ibstatus
Infiniband device ‘mlx5_0’ port 1 status:
default gid: unknown
base lid: 0x0
sm lid: 0x0
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 100 Gb/sec (4X EDR)
link_layer: Ethernet

Infiniband device ‘mlx5_1’ port 1 status:
default gid: unknown
base lid: 0x0
sm lid: 0x0
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 100 Gb/sec (4X EDR)
link_layer: Ethernet

Infiniband device ‘mlx5_2’ port 1 status:
default gid: fe80:0000:0000:0000:ba59:9fff:fee3:c07e
base lid: 0x0
sm lid: 0x0
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 25 Gb/sec (1X EDR)
link_layer: Ethernet

Infiniband device ‘mlx5_3’ port 1 status:
default gid: fe80:0000:0000:0000:ba59:9fff:fee3:c07f
base lid: 0x0
sm lid: 0x0
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 2.5 Gb/sec (1X SDR)
link_layer: Ethernet
ibv_devinfo -v
hca_id: mlx5_2
transport: InfiniBand (0)
fw_ver: 16.35.3502
node_guid: b859:9f03:00e3:c07e
sys_image_guid: b859:9f03:00e3:c07e
vendor_id: 0x02c9
vendor_part_id: 4119
hw_ver: 0x0
board_id: MT_0000000080
phys_port_cnt: 1
max_mr_size: 0xffffffffffffffff
page_size_cap: 0xfffffffffffff000
max_qp: 131072
max_qp_wr: 32768
device_cap_flags: 0x25321c36
BAD_PKEY_CNTR
BAD_QKEY_CNTR
AUTO_PATH_MIG
CHANGE_PHY_PORT
PORT_ACTIVE_EVENT
SYS_IMAGE_GUID
RC_RNR_NAK_GEN
MEM_WINDOW
XRC
MEM_MGT_EXTENSIONS
MEM_WINDOW_TYPE_2B
RAW_IP_CSUM
MANAGED_FLOW_STEERING
max_sge: 30
max_sge_rd: 30
max_cq: 16777216
max_cqe: 4194303
max_mr: 16777216
max_pd: 8388608
max_qp_rd_atom: 16
max_ee_rd_atom: 0
max_res_rd_atom: 2097152
max_qp_init_rd_atom: 16
max_ee_init_rd_atom: 0
atomic_cap: ATOMIC_HCA (1)
max_ee: 0
max_rdd: 0
max_mw: 16777216
max_raw_ipv6_qp: 0
max_raw_ethy_qp: 0
max_mcast_grp: 2097152
max_mcast_qp_attach: 240
max_total_mcast_qp_attach: 503316480
max_ah: 2147483647
max_fmr: 0
max_srq: 8388608
max_srq_wr: 32767
max_srq_sge: 31
max_pkeys: 128
local_ca_ack_delay: 16
general_odp_caps:
ODP_SUPPORT
ODP_SUPPORT_IMPLICIT
rc_odp_caps:
SUPPORT_SEND
SUPPORT_RECV
SUPPORT_WRITE
SUPPORT_READ
SUPPORT_SRQ
uc_odp_caps:
NO SUPPORT
ud_odp_caps:
SUPPORT_SEND
xrc_odp_caps:
SUPPORT_SEND
SUPPORT_WRITE
SUPPORT_READ
SUPPORT_SRQ
completion timestamp_mask: 0x7fffffffffffffff
hca_core_clock: 156250kHZ
raw packet caps:
C-VLAN stripping offload
Scatter FCS offload
IP csum offload
Delay drop
device_cap_flags_ex: 0x3000005425321C36
RAW_SCATTER_FCS
PCI_WRITE_END_PADDING
Unknown flags: 0x3000004000000000
tso_caps:
max_tso: 262144
supported_qp:
SUPPORT_RAW_PACKET
rss_caps:
max_rwq_indirection_tables: 524288
max_rwq_indirection_table_size: 2048
rx_hash_function: 0x1
rx_hash_fields_mask: 0x800000FF
supported_qp:
SUPPORT_RAW_PACKET
max_wq_type_rq: 8388608
packet_pacing_caps:
qp_rate_limit_min: 1kbps
qp_rate_limit_max: 25000000kbps
supported_qp:
SUPPORT_RAW_PACKET
tag matching not supported

cq moderation caps:
	max_cq_count:	65535
	max_cq_period:	4095 us

maximum available device memory:	131072Bytes

num_comp_vectors:		48
	port:	1
		state:			PORT_ACTIVE (4)
		max_mtu:		4096 (5)
		active_mtu:		1024 (3)
		sm_lid:			0
		port_lid:		0
		port_lmc:		0x00
		link_layer:		Ethernet
		max_msg_sz:		0x40000000
		port_cap_flags:		0x04010000
		port_cap_flags2:	0x0000
		max_vl_num:		invalid value (0)
		bad_pkey_cntr:		0x0
		qkey_viol_cntr:		0x0
		sm_sl:			0
		pkey_tbl_len:		1
		gid_tbl_len:		255
		subnet_timeout:		0
		init_type_reply:	0
		active_width:		1X (1)
		active_speed:		25.0 Gbps (32)
		phys_state:		LINK_UP (5)
		GID[  0]:		fe80:0000:0000:0000:ba59:9fff:fee3:c07e, RoCE v1
		GID[  1]:		fe80::ba59:9fff:fee3:c07e, RoCE v2
		GID[  2]:		fe80:0000:0000:0000:9ec7:62b5:8160:f1df, RoCE v1
		GID[  3]:		fe80::9ec7:62b5:8160:f1df, RoCE v2

hca_id: mlx5_3
transport: InfiniBand (0)
fw_ver: 16.35.3502
node_guid: b859:9f03:00e3:c07f
sys_image_guid: b859:9f03:00e3:c07e
vendor_id: 0x02c9
vendor_part_id: 4119
hw_ver: 0x0
board_id: MT_0000000080
phys_port_cnt: 1
max_mr_size: 0xffffffffffffffff
page_size_cap: 0xfffffffffffff000
max_qp: 131072
max_qp_wr: 32768
device_cap_flags: 0x25321c36
BAD_PKEY_CNTR
BAD_QKEY_CNTR
AUTO_PATH_MIG
CHANGE_PHY_PORT
PORT_ACTIVE_EVENT
SYS_IMAGE_GUID
RC_RNR_NAK_GEN
MEM_WINDOW
XRC
MEM_MGT_EXTENSIONS
MEM_WINDOW_TYPE_2B
RAW_IP_CSUM
MANAGED_FLOW_STEERING
max_sge: 30
max_sge_rd: 30
max_cq: 16777216
max_cqe: 4194303
max_mr: 16777216
max_pd: 8388608
max_qp_rd_atom: 16
max_ee_rd_atom: 0
max_res_rd_atom: 2097152
max_qp_init_rd_atom: 16
max_ee_init_rd_atom: 0
atomic_cap: ATOMIC_HCA (1)
max_ee: 0
max_rdd: 0
max_mw: 16777216
max_raw_ipv6_qp: 0
max_raw_ethy_qp: 0
max_mcast_grp: 2097152
max_mcast_qp_attach: 240
max_total_mcast_qp_attach: 503316480
max_ah: 2147483647
max_fmr: 0
max_srq: 8388608
max_srq_wr: 32767
max_srq_sge: 31
max_pkeys: 128
local_ca_ack_delay: 16
general_odp_caps:
ODP_SUPPORT
ODP_SUPPORT_IMPLICIT
rc_odp_caps:
SUPPORT_SEND
SUPPORT_RECV
SUPPORT_WRITE
SUPPORT_READ
SUPPORT_SRQ
uc_odp_caps:
NO SUPPORT
ud_odp_caps:
SUPPORT_SEND
xrc_odp_caps:
SUPPORT_SEND
SUPPORT_WRITE
SUPPORT_READ
SUPPORT_SRQ
completion timestamp_mask: 0x7fffffffffffffff
hca_core_clock: 156250kHZ
raw packet caps:
C-VLAN stripping offload
Scatter FCS offload
IP csum offload
Delay drop
device_cap_flags_ex: 0x3000005425321C36
RAW_SCATTER_FCS
PCI_WRITE_END_PADDING
Unknown flags: 0x3000004000000000
tso_caps:
max_tso: 262144
supported_qp:
SUPPORT_RAW_PACKET
rss_caps:
max_rwq_indirection_tables: 524288
max_rwq_indirection_table_size: 2048
rx_hash_function: 0x1
rx_hash_fields_mask: 0x800000FF
supported_qp:
SUPPORT_RAW_PACKET
max_wq_type_rq: 8388608
packet_pacing_caps:
qp_rate_limit_min: 1kbps
qp_rate_limit_max: 25000000kbps
supported_qp:
SUPPORT_RAW_PACKET
tag matching not supported

cq moderation caps:
	max_cq_count:	65535
	max_cq_period:	4095 us

maximum available device memory:	131072Bytes

num_comp_vectors:		48
	port:	1
		state:			PORT_ACTIVE (4)
		max_mtu:		4096 (5)
		active_mtu:		1024 (3)
		sm_lid:			0
		port_lid:		0
		port_lmc:		0x00
		link_layer:		Ethernet
		max_msg_sz:		0x40000000
		port_cap_flags:		0x04010000
		port_cap_flags2:	0x0000
		max_vl_num:		invalid value (0)
		bad_pkey_cntr:		0x0
		qkey_viol_cntr:		0x0
		sm_sl:			0
		pkey_tbl_len:		1
		gid_tbl_len:		255
		subnet_timeout:		0
		init_type_reply:	0
		active_width:		1X (1)
		active_speed:		2.5 Gbps (1)
		phys_state:		LINK_UP (5)
		GID[  0]:		fe80:0000:0000:0000:ba59:9fff:fee3:c07f, RoCE v1
		GID[  1]:		fe80::ba59:9fff:fee3:c07f, RoCE v2
		GID[  2]:		fe80:0000:0000:0000:12b2:f761:211b:c4ad, RoCE v1
		GID[  3]:		fe80::12b2:f761:211b:c4ad, RoCE v2

hca_id: mlx5_0
transport: InfiniBand (0)
fw_ver: 22.41.1000
node_guid: b8ce:f603:00d2:cd22
sys_image_guid: b8ce:f603:00d2:cd22
vendor_id: 0x02c9
vendor_part_id: 4125
hw_ver: 0x0
board_id: MT_0000000436
phys_port_cnt: 1
max_mr_size: 0xffffffffffffffff
page_size_cap: 0xfffffffffffff000
max_qp: 4096
max_qp_wr: 32768
device_cap_flags: 0x25321c36
BAD_PKEY_CNTR
BAD_QKEY_CNTR
AUTO_PATH_MIG
CHANGE_PHY_PORT
PORT_ACTIVE_EVENT
SYS_IMAGE_GUID
RC_RNR_NAK_GEN
MEM_WINDOW
XRC
MEM_MGT_EXTENSIONS
MEM_WINDOW_TYPE_2B
RAW_IP_CSUM
MANAGED_FLOW_STEERING
max_sge: 30
max_sge_rd: 30
max_cq: 16777216
max_cqe: 4194303
max_mr: 16777216
max_pd: 8388608
max_qp_rd_atom: 16
max_ee_rd_atom: 0
max_res_rd_atom: 65536
max_qp_init_rd_atom: 16
max_ee_init_rd_atom: 0
atomic_cap: ATOMIC_HCA (1)
max_ee: 0
max_rdd: 0
max_mw: 16777216
max_raw_ipv6_qp: 0
max_raw_ethy_qp: 0
max_mcast_grp: 2097152
max_mcast_qp_attach: 240
max_total_mcast_qp_attach: 503316480
max_ah: 2147483647
max_fmr: 0
max_srq: 8388608
max_srq_wr: 32767
max_srq_sge: 31
max_pkeys: 128
local_ca_ack_delay: 16
general_odp_caps:
rc_odp_caps:
NO SUPPORT
uc_odp_caps:
NO SUPPORT
ud_odp_caps:
NO SUPPORT
xrc_odp_caps:
NO SUPPORT
completion timestamp_mask: 0x7fffffffffffffff
hca_core_clock: 1000000kHZ
raw packet caps:
C-VLAN stripping offload
Scatter FCS offload
IP csum offload
Delay drop
device_cap_flags_ex: 0x3000005425321C36
RAW_SCATTER_FCS
PCI_WRITE_END_PADDING
Unknown flags: 0x3000004000000000
tso_caps:
max_tso: 262144
supported_qp:
SUPPORT_RAW_PACKET
rss_caps:
max_rwq_indirection_tables: 8388608
max_rwq_indirection_table_size: 256
rx_hash_function: 0x1
rx_hash_fields_mask: 0x800000FF
supported_qp:
SUPPORT_RAW_PACKET
max_wq_type_rq: 8388608
packet_pacing_caps:
qp_rate_limit_min: 1kbps
qp_rate_limit_max: 100000000kbps
supported_qp:
SUPPORT_RAW_PACKET
tag matching not supported

cq moderation caps:
	max_cq_count:	65535
	max_cq_period:	4095 us

maximum available device memory:	131072Bytes

num_comp_vectors:		48
	port:	1
		state:			PORT_ACTIVE (4)
		max_mtu:		4096 (5)
		active_mtu:		1024 (3)
		sm_lid:			0
		port_lid:		0
		port_lmc:		0x00
		link_layer:		Ethernet
		max_msg_sz:		0x40000000
		port_cap_flags:		0x00000000
		port_cap_flags2:	0x0000
		max_vl_num:		invalid value (0)
		bad_pkey_cntr:		0x0
		qkey_viol_cntr:		0x0
		sm_sl:			0
		pkey_tbl_len:		1
		gid_tbl_len:		0
		subnet_timeout:		0
		init_type_reply:	0
		active_width:		4X (2)
		active_speed:		25.0 Gbps (32)
		phys_state:		LINK_UP (5)

hca_id: mlx5_1
transport: InfiniBand (0)
fw_ver: 22.41.1000
node_guid: b8ce:f603:00d2:cd23
sys_image_guid: b8ce:f603:00d2:cd22
vendor_id: 0x02c9
vendor_part_id: 4125
hw_ver: 0x0
board_id: MT_0000000436
phys_port_cnt: 1
max_mr_size: 0xffffffffffffffff
page_size_cap: 0xfffffffffffff000
max_qp: 4096
max_qp_wr: 32768
device_cap_flags: 0x25321c36
BAD_PKEY_CNTR
BAD_QKEY_CNTR
AUTO_PATH_MIG
CHANGE_PHY_PORT
PORT_ACTIVE_EVENT
SYS_IMAGE_GUID
RC_RNR_NAK_GEN
MEM_WINDOW
XRC
MEM_MGT_EXTENSIONS
MEM_WINDOW_TYPE_2B
RAW_IP_CSUM
MANAGED_FLOW_STEERING
max_sge: 30
max_sge_rd: 30
max_cq: 16777216
max_cqe: 4194303
max_mr: 16777216
max_pd: 8388608
max_qp_rd_atom: 16
max_ee_rd_atom: 0
max_res_rd_atom: 65536
max_qp_init_rd_atom: 16
max_ee_init_rd_atom: 0
atomic_cap: ATOMIC_HCA (1)
max_ee: 0
max_rdd: 0
max_mw: 16777216
max_raw_ipv6_qp: 0
max_raw_ethy_qp: 0
max_mcast_grp: 2097152
max_mcast_qp_attach: 240
max_total_mcast_qp_attach: 503316480
max_ah: 2147483647
max_fmr: 0
max_srq: 8388608
max_srq_wr: 32767
max_srq_sge: 31
max_pkeys: 128
local_ca_ack_delay: 16
general_odp_caps:
rc_odp_caps:
NO SUPPORT
uc_odp_caps:
NO SUPPORT
ud_odp_caps:
NO SUPPORT
xrc_odp_caps:
NO SUPPORT
completion timestamp_mask: 0x7fffffffffffffff
hca_core_clock: 1000000kHZ
raw packet caps:
C-VLAN stripping offload
Scatter FCS offload
IP csum offload
Delay drop
device_cap_flags_ex: 0x3000005425321C36
RAW_SCATTER_FCS
PCI_WRITE_END_PADDING
Unknown flags: 0x3000004000000000
tso_caps:
max_tso: 262144
supported_qp:
SUPPORT_RAW_PACKET
rss_caps:
max_rwq_indirection_tables: 8388608
max_rwq_indirection_table_size: 256
rx_hash_function: 0x1
rx_hash_fields_mask: 0x800000FF
supported_qp:
SUPPORT_RAW_PACKET
max_wq_type_rq: 8388608
packet_pacing_caps:
qp_rate_limit_min: 1kbps
qp_rate_limit_max: 100000000kbps
supported_qp:
SUPPORT_RAW_PACKET
tag matching not supported

cq moderation caps:
	max_cq_count:	65535
	max_cq_period:	4095 us

maximum available device memory:	131072Bytes

num_comp_vectors:		48
	port:	1
		state:			PORT_ACTIVE (4)
		max_mtu:		4096 (5)
		active_mtu:		1024 (3)
		sm_lid:			0
		port_lid:		0
		port_lmc:		0x00
		link_layer:		Ethernet
		max_msg_sz:		0x40000000
		port_cap_flags:		0x00000000
		port_cap_flags2:	0x0000
		max_vl_num:		invalid value (0)
		bad_pkey_cntr:		0x0
		qkey_viol_cntr:		0x0
		sm_sl:			0
		pkey_tbl_len:		1
		gid_tbl_len:		0
		subnet_timeout:		0
		init_type_reply:	0
		active_width:		4X (2)
		active_speed:		25.0 Gbps (32)
		phys_state:		LINK_UP (5)

gid is not appearing for mlx5_0 and mlx5_1

Thank you, it seems like a tool issue.
Please check with #show_gids

Thanks,
Suo

show_gids
DEV PORT INDEX GID IPv4 VER DEV


mlx5_2 1 0 fe80:0000:0000:0000:ba59:9fff:fee3:c07e v1 enp129s0f0np0
mlx5_2 1 1 fe80:0000:0000:0000:ba59:9fff:fee3:c07e v2 enp129s0f0np0
mlx5_3 1 0 fe80:0000:0000:0000:ba59:9fff:fee3:c07f v1 enp129s0f1np1
mlx5_3 1 1 fe80:0000:0000:0000:ba59:9fff:fee3:c07f v2 enp129s0f1np1
n_gids_found=4
Still no gid for mlx5_0 and mlx5_1

Can you try to reload driver or just mlx5_ib ?
It might impact your job.
rmmod mlx5_ib
modprobe mlx5_ib

I have tried it multiple times before, and also tried that now. But the issue is still here. I am unable to see gids for my connectX-6 device.

Hi,
Can you collect a sysinfo-snapshot, and then open a customer support ticket?

Thanks,
Suo

sure, thanks