We need 5 Ethernet ports in design. TX2 already has one Ethernet port. For the remained 4 ports, I am going to use PCIe X1 port to a fan-out PCIe switch, The Fanout switch will have enough ports to connect 4 NIC, is this feasible? can this design reach more than 3gbps traffic?
With PCIe single lane, under rev. 1 speed, no. Under rev. 2 speed, maybe. Rev. 2 runs at 5GT/s per lane, and is 8b/10b encoded. So basically 8/10 of 5Gbit/s if all goes perfectly. In terms of practical performance I have doubts even at rev. 2 speeds if using a single lane. You’d be fairly safe with 2-lane under rev. 2. A single rev. 2 lane might have a few hiccups with four gigabit ports, though it would depend on whether all ports get traffic at the same time (and video would be constant traffic).
Thanks, The bandwidth looks like an issue. But I have to live with it.
From software architecture point of view, can One PCIe host port work with 4 NIC devices? Those 4 NIC devices are behind a PCIe Switch.
Imagine you have a gigabit switch with five ports (one going to the internet, four going to your devices). Imagine the line going to the internet is just like all of the other ports, and supports gigabit. If a single device is using bandwidth, then then internet line should provide full bandwidth. If you have two or more ports both trying to use a gigabit, then the main line is going to have to split bandwidth and each individual port will be at about half. Use four ports all at once, and the single gigabit feeding it will still have gigabit throughput, and so the other four ports would only have a quarter of gigabit available.
A single PCIe lane isn’t much different than a gigabit port, but overhead and basic throughput will be different. Four ports implies it might use four gigabit of raw throughput. A perfect PCIe lane (having its own dedicated CPU core) can put out 5GT/s * 8b/10b * 1byte/transfer == 4Gbit/s. Keep in mind that if the CPU itself, or some component in the network stack can’t process that much data without slowing, then those 4Gbit/s are going to respond slower some times versus other times. If there isn’t enough buffer and the packets can’t be thrown out, then it also depends on being able to consume and process the data in time. You have the possibility of four ports working as you want, but if you want it real time and behaved well for video I suspect it’ll have some issues from time to time. It won’t all go smoothly.
If you have two lanes at PCIe gen. 2, then there probably would never be an issue. If you have one lane at PCIe gen. 1, then it’ll never work well with four ports. One lane at gen. 2 might be ok at times, but is unlikely to run completely smooth at all times.
Keep in mind that CPU0 is doing most hardware I/O for other parts of the system. You might find USB use, or the integrated NIC, or any other number of uses which have to go through CPU0 (and have no alternative) are out of your control. If you’re using this to drive a car down the highway, then I wouldn’t even try. If this is for a drone going slowly through a forest, then it probably won’t care. If it is used for a human interaction with some sort of media display, then it may be a bit like a video game which stutters under heavier loads and is noticed by the people participating.
Some data transfer might be treated as blocking (reliable communications, e.g., TCP). Some data might be treated as best case, and dropped when it can’t be processed in time (e.g., UDP). Stuttering versus frame drop might be a choice you have to make. You won’t know how bad it is (or how well it can be ignored) until you try.