How to make a transparent network bridge between usb and ethernet ports

Robotics & Edge Computing Jetson Systems Jetson Nano
,
1

Hi,

I want to connect multiple (up to 4) Jetson Nano (or other Jetson models) in a chain so, the first Jetson is connected to my LAN switch through ethernet connector. The second is connected to the first with a USB 2.0 cable between the micro-usb connector of second and USB 3.0 type A connector in first. The rest of the chain is the same as this step: usb cable between micro-usb (Jetson N) and USB 3.0 type A (Jetson N+1).
My product is a an arrange of Jetson separated like 50cm and I want to save components, space, money, power and point of failure replacing a Ethernet switch with chain of ‘Ethernet over USB’ connections.
In my application I need max of 10Mbps of each Jetson uplink throughput to internet, so with 4 Jetsons, the heaviest USB link will support aprox 30Mbps. What I guess should be fine.
I have been testing the default (working quite weel) USB gadget that provide internet connection to downstream Jetson. But the first jetson act as NAT and hide the rest of the chain in a subnet. And subsequently.
I want that each Jetson can be seen as in my LAN, getting IP from DHCP server (or static ip). So the Jetson act as a transparent network bridge instead of a NAT.

My cuestions are:

  • That it makes sense this type of solution for such use case?
  • Any potential risk? instability? overload? or any other drawback?
  • How to implement?
  • Same solution for all Jetson (NX format) SoMs?

I am using latest Jetpack.

2

I haven’t worked on a bridge in decades, and I couldn’t answer this. Some observations though say this won’t work.

First, a bridge is not something the computer providing it can use for its own traffic unless you can dual home the NIC. So far as I know the network connections and hardware involved do not support dual homing. Without that it is nothing more than a wire (a very useful wire if you want to use an application such as snort to detect malware traffic). A bridge in and of itself does not provide any way for the host PC (normally; without added software) to actually use that connection for itself. You’re speaking of a system with two NICs (the USB part is just software, but it is treated as a NIC), and it really takes three NICs to be useful the way you are thinking.

The overhead of doing this would also be far higher than you are probably thinking of, and the throughput you could possibly get the 10 Mb/s you are wanting, but latency (and perhaps reliability) would tend to get worse.

A Jetson can have its own DHCP server, and so a Jetson could become a router. Still, you’d need an Ethernet switch due to connectors. This latter would not require a router since one Jetson would become a router, but I think this does defeat the purpose of reducing hardware.

FYI, when running as a bridge, the connection is in “promiscuous mode”. This tends to mean traffic goes through the CPU. On a desktop PC each NIC and each CPU core can handle any hardware device independently, but on a Jetson, much of the hardware can only go through the first CPU core (CPU0). I think you’d end up with a lot of traffic on CPU0 and start running into serious issues under any kind of significant load. A desktop PC tends to not have that problem because they have either an I/O APIC (asynchronous programmable interrupt controller) or some equivalent which can take a hardware IRQ (triggered in a physical wire and not just software) and route it to any CPU core.

By the time you add another NIC to each Jetson you don’t save on hardware/expense/power consumption. A single Ethernet switch could once again handle that less expensively.

3

Hi Linuxdev,

Thanks for extensive explanation.
Some questios/comments:

  • The ‘promiscious’ problem (overhead) also apply to current supported NAT based opt/nvidia/nv-usb-device-mode?
  • Not sure if I undetstood you completely. Imagine one of my product in a factory. 4 Jetson Nanos in a 2 meters long capsule. Each one with a camera producing frames and uploading to a cloud server. If I connect only one Jetson to closest network switch ( may me 50 meters apart), and the rest in a chain. 1<->2<->3<->4 I dont need a extra switch or 3 extra 50 meters cable to the factory switch.
4

I don’t even know if the USB device mode can go into promiscuous mode, but probably it can. However, what you’ll find is that the promiscuous mode traffic likely must all go through CPU0, which also processes most of the hardware interrupts/drivers. There will be a lot of computation on a single core where all of that hardware is competing for that single core near 100% capacity (that’s my guess). Now add the fact that the most upstream Jetson, next to the outside world, is going to see its own traffic, the 2nd Jetson’s traffic, the 3rd Jetson’s traffic, and the 4th Jetson’s traffic. That’s a lot of data. That same core is going to have all of your disk or eMMC or SD card traffic.

You still need to either multihome the wired NIC (I doubt the USB gadget wired Ethernet can dual home), which I don’t think works on Jetson NICs, or else it is only a bridge which the local Jetson itself will have no access to. It is the dual home function which gives the local system access to its own bridge. Not all Ethernet hardware and/or drivers support that. Most likely you will have to add another NIC if you want local access. Then that third NIC has to go to your router, so you are not really gaining anything.

Keep in mind that a bridge is also called a “transparent bridge” for a reason. It gets inserted in the middle of an Ethernet cable and nothing at either end really knows it is there. One side can be a different range of network addresses from the other to adapt as a WAN. The driver can add firewall capability and intrusion detection. The local system though cannot use this as an ordinary Ethernet and it must either get another NIC or dual home one of the bridge NICs to use it locally. Without dual home that bridge is invisible to the local system.

NAT works despite its complications, and it does so without extra hardware. Putting several NAT devices in a daisy-chain is far from ideal, but if you really want that, then it would avoid requiring a network switch. You’d have to be quite desperate for space and cost reduction to go the NAT route, but it would get the job done if bandwidth requirements are not too high. Overload and loss of data is a very high risk when you daisy-chain the Jetsons.

If you really want to do this, then the Jetson is no different from an ordinary desktop host PC. If your L4T release runs Ubuntu 18.04 (L4T R32.x is Ubuntu 18.04; see “head -n 1 /etc/nv_tegra_release”). The ability to dual home though has both software and hardware requirements, and I suspect the Realtek NIC is not set up to dual home. If it is, then the kernel version must also have some capability to dual home and bridge (which is probably just adding a kernel feature).

I can’t tell you the specifics of setting it up, I have not done so in a long long time. The parts about NIC hardware not supporting dual homing is probably (not definitely) true on all of the Jetsons. The need for kernel support is true on all Jetsons, and those from a given series most likely use the same kernel, but for example the Xavier can also use L4T R35.x, and Orin can be L4T R35.x or R36.x. The NX format doesn’t change much, but the driver does (and that might change during major release changes, e.g., L4T R36.x of the Orin is a mainline kernel).

5

Thenks Linuxdev for extense respond.
I was testing 2 Jetson nano connected in dasiy-chain with NAT. Both working at mak speed for my application (uploading camera images jpeg at 15 FPS). The summary is that it works but first Jetson increasy CPU load from 18% to 25% aprox. If longer chain I gess the CPU load will increase lineraly.
As you said may be a solution in case there are no other solution.