MGBE to ethernet RJ45

[jmm1]

Hey, so I’m looking into getting both a 5GbE and a 1GbE Ethernet port (RJ45) on the T5000 module. I’ve looked into using the MGBE controller for the 5GbE port, but I haven’t been able to find any information confirming whether this is actually possible.

So, I found this information stating that XFI + MDIO + PHY is not supported.
https://forums.developer.nvidia.com/t/thor-carrier-board-xfi-phy/355049

Or at least not with the AQR113C.

So, is it true that there’s no way to connect the MGBE module to an RJ45 port like on the Orin?
You can only use QSFP?

Or am I misunderstanding something?

Would the only way to do this then be by using two NICs like this?

One for 1GbE and one for 5GbE. I’d like to use as few PCIe lanes as possible.

I’ve seen a board design where they implemented two 10GbE ports from the MGBE using the AQR113C, but based on what was said in the other forum post, I don’t understand how that is possible.

I can see here that it might be supported, but that you would have to provide your own software. Is this true? And what software do you mean by that ?

How to design four MGBE used AQR113C

I star a thor hardware design with four MGBE , used the AQR113C , and the T5000 module seems only provide four MDIO signal and RST/INT sinal for XFI0 and XFI3 , but can not found the XFI1 and XFI2 RST/INT sinal. so how to assign the XFI1 and XFI2 RST/INT signal on T5000 module?

Here you say that there is no support for XFI.

Network design of Thor T5000

[Question1] From page 43.Table 7-4 of “jetson-thor-series-modules-designguide-dg12084001-v1.3.pdf”, can we use 4 XFI ? If you have the T5000 module, there are 4x XFI available. [Question2] There are 4 XFI(XFI0 to XFI3) in “P4071_A02_Concept_schematics.pdf”, we can find the reset and interrupt pins of XFI0 and XFI3, but can’t find these two pins of XFI1 and XFI2. Which pins are suitable for XFI1 and XFI2 ? The reference design shows a QSFP connector so it only has one set of QSFP_RESET_N…

But in the design guide, you say that you can communicate with the MGBE using XFI under 7.3 MGBE.
I hope someone can clarify some of my confusion.

[ChrisB_NV]

You could use the interface you mentioned as below from the Design Guide for the 5GbE and add a 1GbE controller+PHY on an available PCIe x1.

@jmm1

But in the design guide, you say that you can communicate with the MGBE using XFI under 7.3 MGBE.

The Design Guide covers the hardware side and the Jetson Linux Developer Guide covers software support. The hardware is capable of XFI as per the Design Guide but as you found in other forum posts, there is no software support.

[jmm1]

Okay, so if you want to use MGBE as standard Ethernet, you will need to write your own software. The hardware support is there, but there is currently no ready-made software support.

And just to be clear: when I say this, I mean MGBE to Ethernet.

I have a second question, if that’s okay.

So, when I read this, I understand it as meaning that you can only have all 12 PCIe lanes available if you use C1, C2, C3, and C4.

Can C4 do PCIe bifurcation, so that C4 can support two M.2 slots with x4 lanes each? Or is that not possible, meaning one of the M.2 slots would have to run at x2 or x1?

And thank you for your reply.

[jmm1]

I can see that this is not currently supported, but would it be possible if a PCIe switch were used instead? PCIe C4 Bifurcation on Jetson Thor (T5000)

like they talk about here.

https://community.cadence.com/cadence_blogs_8/b/ip/posts/rethinking-ai-infrastructure-the-rise-of-pcie-switches

[ChrisB_NV]

@jmm1

Okay, so if you want to use MGBE as standard Ethernet, you will need to write your own software. The hardware support is there, but there is currently no ready-made software support.

Yes.

@jmm1

So, when I read this, I understand it as meaning that you can only have all 12 PCIe lanes available if you use C1, C2, C3, and C4.

Yes.

@jmm1

Can C4 do PCIe bifurcation, so that C4 can support two M.2 slots with x4 lanes each? Or is that not possible, meaning one of the M.2 slots would have to run at x2 or x1?

Thor does not support PCIe bifurcation so one M.2 could be x4 on C4 and the second M.2 would have to be x2 on C3 or x1 on C1 or C2.

@jmm1

I can see that this is not currently supported, but would it be possible if a PCIe switch were used instead?

C4 x8 as the root port to a switch and then two x4 M.2s on the switch could work hardware-wise though I’m not sure about booting from one of the drives in that configuration. Checking with the software team.

[ChrisB_NV]

@ChrisB_NV

C4 x8 as the root port to a switch and then two x4 M.2s on the switch could work hardware-wise though I’m not sure about booting from one of the drives in that configuration. Checking with the software team.

I received feedback that boot failures have been seen with this setup if bootloader doesn’t have the driver for the PCIe switch. So please take this into consideration if one of the drives on the switch is the boot drive.

[jmm1]

okay thank you i think that was the questions i had for now.

[ChrisB_NV]

An imminent Jetson Thor Supported Components List update includes a PHY part validated by NVIDIA, including software support. I will reply here with a link when the new SCL is available on the download center.

v1.5 of the Jetson Thor Series Modules Interface Design Guide released yesterday adds this PHY connection example:

[ChrisB_NV]

v1.4 of the Jetson Thor Series Supported Component List released today includes the Marvell AQR113C-B0-C PHY.