Hi NVIDIA Forum,
we’re having a custom board design with the T5000 - XFI-C0 routed to a SFP28 Port. We successfully verified the 25GBit/s connection following this: Jetson Thor Adaptation and Bring-Up — NVIDIA Jetson Linux Developer Guide
We’ve used a QSFP-100G to a 4SFP28 DAC Cable - this worked.
Then we reverted to 10GBit/s:
uphy {
status = "okay";
//uphy0-config = <0x07>; //original
uphy1-config = <0x07>; //original
//25Gbit/s
uphy0-config = <0x06>;
//uphy1-config = <0x08>;
mgbe0-speed = <0x02>;
mgbe1-speed = <0x02>;
mgbe2-speed = <0x02>;
mgbe3-speed = <0x02>;
/*
mgbe0-speed = <0x03>;
mgbe1-speed = <0x03>;
mgbe2-speed = <0x03>;
mgbe3-speed = <0x03>;
*/
};
Using the same board, but flashed to 10Gbit/s we got issues, during using a 10Gbit/s SFP+ DAC Cable. Now the Link mgbe0_0 is displayed all the time as “down”.
I mean it’s quite easy connected - and the QSFP or the SFP+ should not have physically any difference. This is the schematic:
And here the output of mgbe0_0, we’re not getting the Link detected to “yes”.
root@t5000:/boot/overlays# ethtool mgbe0_0
Settings for mgbe0_0:
Supported ports: [ TP MII ]
Supported link modes: Not reported
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: Not reported
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Auto-negotiation: on
Port: MII
PHYAD: 0
Transceiver: external
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000000 (0)
Link detected: no
Also plese find the running .dtb for the mgbe0_0:
ethernet@a808a10000 {
mac-address = [4c bb 47 dc c5 3c];
compatible = "nvidia,tegra264-mgbe";
status = "okay";
reg = <0xa8 0x8a10000 0x00 0x10000 0xa8 0x8aa0000 0x00 0x10000 0xa8 0x8ad0000 0x00 0x10000 0xa8 0x8a00000 0x00 0x10000>;
reg-names = "mac\0xpcs\0macsec-base\0hypervisor";
interrupts = <0x00 0x366 0x04 0x00 0x369 0x04 0x00 0x36a 0x04 0x00 0x36b 0x04 0x00 0x36c 0x04 0x00 0x36d 0x04 0x00 0x368 0x04 0x00 0x367 0x04>;
interrupt-names = "common\0vm0\0vm1\0vm2\0vm3\0vm4\0macsec-ns-irq\0macsec-s-irq";
resets = <0x02 0x3f 0x02 0x3e 0x02 0x40>;
reset-names = "mac\0pcs\0macsec_ns_rst";
clocks = <0x02 0xd8 0x02 0x164 0x02 0xda 0x02 0xd9 0x02 0xd7 0x02 0xd5 0x02 0x168 0x02 0x166 0x02 0x165 0x02 0x169 0x02 0x167 0x02 0x130 0x02 0x12c 0x02 0x1cb 0x02 0x1cf>;
clock-names = "mac\0tx\0mgbe\0macsec\0tx-pcs\0ptp-ref\0rx-input\0rx-pcs-input\0tx-m\0rx-input-m\0rx-pcs-m\0utmi_pll1_clk\0pll_bpmpcam\0tx_ser\0rx_ser";
power-domains = <0x02 0x0d>;
nvidia,num-dma-chans = <0x04>;
nvidia,dma-chans = <0x00 0x01 0x02 0x03>;
iommus = <0x139 0x1300>;
nvidia,num-mtl-queues = <0x0a>;
nvidia,mtl-queues = <0x00 0x01 0x02 0x03 0x04 0x05 0x06 0x07 0x08 0x09>;
nvidia,tc-mapping = <0x00 0x01 0x02 0x03 0x04 0x05 0x06 0x07 0x00 0x01>;
nvidia,residual-queue = <0x01>;
nvidia,rxq_enable_ctrl = <0x02 0x02 0x02 0x02 0x02 0x02 0x02 0x02 0x02 0x02>;
nvidia,vm-irq-config = <0x151>;
nvidia,vm-vdma-config = <0x152>;
nvidia,tx-queue-prio = <0x00 0x01 0x02 0x03 0x04 0x05 0x06 0x07 0x00 0x00>;
nvidia,rx-queue-prio = <0x01 0x02 0x04 0x08 0x10 0x20 0x40 0x80 0x00 0x00>;
nvidia,dcs-enable = <0x01>;
nvidia,macsec-enable = <0x01>;
nvidia,mgbe-riit-config = <0x153>;
nvidia,rx_riwt = <0x200>;
nvidia,rx_frames = <0x40>;
nvidia,tx_usecs = <0x100>;
nvidia,tx_frames = <0x10>;
//10G XFI
nvidia,phy-iface-mode = <0x00>;
nvidia,promisc_mode = <0x01>;
nvidia,slot_num_check = <0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00>;
nvidia,slot_intvl_vals = <0x00 0x7d 0x7d 0x7d 0x7d 0x7d 0x7d 0x7d 0x7d>;
nvidia,ptp_ref_clock_speed = <0x12a05f20>;
nvidia,instance_id = <0x00>;
nvidia,ptp-rx-queue = <0x03>;
nvidia,dma_rx_ring_sz = <0x1000>;
nvidia,dma_tx_ring_sz = <0x1000>;
dma-coherent;
nvidia,mac-addr-idx = <0x01>;
//10G Mode 0x01
nvidia,uphy-gbe-mode = <0x01>;
nvidia,max-platform-mtu = <0x2328>;
nvidia,pps_op_ctrl = <0x08>;
nvidia,if-name = "mgbe0_0";
nvidia,force-restart-lane-bringup = <0x01>;
phandle = <0x15f>;
fixed-link {
full-duplex;
speed = <0x2710>;
};
};
We also verified on the Thor DevKit that the same cable like mentioned above (QSFP28 to SFP28) works with just one SFP28 connected to another switch.
What is interesting, when we disconnect the SFP, inserted in the custom board then we got following messages in the dmesg:
[ 2144.875109] nvethernet a808a10000.ethernet: [xpcs_lane_bring_up][827][type:0x4][loga-0x0] PCS block lock SUCCESS
[ 2145.387147] nvethernet a808a10000.ethernet: [xpcs_lane_bring_up][827][type:0x4][loga-0x0] PCS block lock SUCCESS
[ 2145.483121] nvethernet a808a10000.ethernet: [xpcs_lane_bring_up][827][type:0x4][loga-0x0] PCS block lock SUCCESS
[ 2145.931120] nvethernet a808a10000.ethernet: [xpcs_lane_bring_up][827][type:0x4][loga-0x0] PCS block lock SUCCESS
[ 2146.443172] nvethernet a808a10000.ethernet: [xpcs_lane_bring_up][827][type:0x4][loga-0x0] PCS block lock SUCCESS
[ 2146.541141] nvethernet a808a10000.ethernet: [xpcs_lane_bring_up][827][type:0x4][loga-0x0] PCS block lock SUCCESS
[ 2146.987005] nvethernet a808a10000.ethernet: [xpcs_lane_bring_up][827][type:0x4][loga-0x0] PCS block lock SUCCESS
[ 2147.498841] nvethernet a808a10000.ethernet: [xpcs_lane_bring_up][827][type:0x4][loga-0x0] PCS block lock SUCCESS
[ 2147.595003] nvethernet a808a10000.ethernet: [xpcs_lane_bring_up][827][type:0x4][loga-0x0] PCS block lock SUCCESS
[ 2148.043029] nvethernet a808a10000.ethernet: [xpcs_lane_bring_up][827][type:0x4][loga-0x0] PCS block lock SUCCESS
So how to enable 10Gbit/s XFI for the mgbe0_0 for the T5000?