How many USB 3.0 interfaces can be extended through the PCIe interfaces on Jetson TX2?

Urgent!I would appreciate it very much if someone can help me answer my question?I want to link several USB3.0 cameras on the USB3.0 interfaces,but on the Jetson tx2,there’s only one USB3.0 interface.So I want to extend USB3.0 interfaces through the PCIe interface on the Jetson tx2.However,the question is that I don’t konw how many USB3.0 interfaces can be extended through the PCIe interface.So someone knows?Please…

Hi, it depends on the bridge chip you choose, you can check with vendor about that, e.g. this one from Broadcom:

Hello Trumany,it means that it has nothing to do with the Jetson tx2?

Not an answer, but this may help…

The big issue is that cameras use a lot of bandwidth. Some more than others. USB has a “root HUB”, and this can be split among multiple ports…but all of the ports share bandwidth with the root HUB. USB3 cameras typically are high resolution and require a good chunk of the bandwidth…adding two or more cameras to a single root HUB can exceed a camera’s required bandwidth. If not two cameras, then more than two…eventually you’ll exceed the USB3 root HUB’s abilities. So it is important to know what any individual camera bandwidth requirement is. Knowing resolution, color depth, and frame rate helps to answer that.

On the other side of the question, plug something in to USB and run the “tree view” of “lsusb”:

lsusb -t

The base of each branch is a “root_hub”. A camera might have a control function, but this doesn’t use much bandwidth; however, the actual video device channel can consume a lot of bandwidth. If you have two video channels converging on a single root_hub, then expect an inability to maintain frame rates.

The mentioned bridge chips would use a PCIe lane and add a root HUB (assumes you are building your own carrier). When you buy a PCIe USB expander add-on card, then this is essentially what you get. Not all add-on cards are equal…some will use a single root HUB and all exposed ports share a bandwidth (not good, but such cards are cheaper to purchase)…others will legitimately provide multiple ports where each port has its own root_hub using as many PCIe lanes as needed to prevent any port from interfering with another port’s bandwidth. If you don’t have enough root HUBs, then this latter PCIe expansion is what you will want.

Thank you so much, the way,do you know how much bandwidth the PCIe interface has on Jetson tx2?

PCIe is a standard. The TX2 I believe supports up to revision 2 only (the Xavier supports up to revision 4).

Revision 1 is 2.5GT/s (“giga transfers per second”…basically a bit/s…on each lane). With 8b/10b encoding, this is a theoretical max throughput of 2Gb/s per lane. The four lane PCIe slot would be 8Gb/s if using all lanes at revision 1 speeds.

Revision 2 is 5GT/s, and also uses 8b/10b encoding, implying 4Gb/s per lane…or 16Gb/s for a four lane PCIe bus.

Revision 3 is 8.5GT/s, but uses a more efficient 128b/130b encoding, implying about 8.4Gb/s per lane. Four lanes would be around 33.5Gb/s.

Revision 4 is 16GT/s, and I’m not positive, but I think it uses the same 128b/130b encoding. Although the Xavier supports this I do not know of any real world hardware which exists at rev. 4…Xavier is among the first of hosts rated to this spec and nothing exists “in the wild”.

If an add-on card supports a higher speed than the bus supports, then it simply throttles back to the earlier revision. A rev. 3 card works great on a rev. 2 bus (such as a TX2), but only at rev. 2 speeds. If signal quality is an issue, then a PCIe card may throttle back even more despite both add-on card and host being rated to a higher revision.

If an add-on card supports more lanes than the socket provides, then the card simply uses the number of lanes available.

Consider revision 2 (4Gb/s per lane) what you would get, using four lanes. The physical slot is 8-lane, but only four lanes are connected.

Hi, say Revision 2 is 5GT/s,why it implys 4GT/s per lane?

And another question.For Revision 2,it has four lanes,and per lane is 4GT/s,so if I use a PCIe USB expander add-on card which has 8 ports(each port for one USB3.0 camera),does the situation like the following:It use exact the all of the four lanes,and each lane is shared exactly by two cameras equally(i.e. each camera shares the bandwidth of 2GT/s).If not,can you show me some solutions to achieve this?

“5GT/s” is the number of transfers…but each transfer is partially made up of some encoding bits used to assure valid data. An 8b/10b scheme means that of every 10 bits sent, 2 bits are non-data, and 8 bits are data. Thus 8b/10b is only 80% of the traffic…5Gb/s times 0.8. Revision 3 is more efficient because it is a 128b/130b encoding (about 98.5% is data throughput, about 1.5% is overhead).

Keep in mind that GT/s is not data, it is total bits.

The design of the actual PCIe card differs by a lot depending on the card. Fooling buyers with cheap cards can be achieved without ever providing false information to the buyer. First you must consider that USB also has an encoding…if you get a USB3 card rated at 5Gb/s, then actual data throughput will not be this high even on a perfect card.

Next, you must understand the difference between a USB port and a USB root HUB. The root HUB is the part which actually talks to the host computer. The port is the connector which routes to the root HUB or a non-root HUB. A non-root HUB (just a “HUB”, e.g., like one you sit on the desktop) is just a way to add ports which share a root HUB.

If you have a root HUB controller which is rated to 5Gb/s, and there is only one port, then after you consider the encoding a single port will have all of that bandwidth available. When you add more ports which merge through that single root HUB by any means, then the root HUB will not magically go faster than 5Gb/s…which works well if only one device needs bandwidth at a time. However, if you have two cameras, then at most the sum total bandwidth of both cameras must be less than 5Gb/s after encoding inefficiencies. Like an extension cord in a wall socket, if you add one device the cord is rated at, it’ll work…but if you try to add two devices, the cord crashes and burns (perhaps literally for the example).

When you have a PCIe card the manufacturer can choose to provide one root HUB shared by four ports, or the manufacturer can choose four ports with their own root HUB. The former case will suffer greatly under simultaneous load, while the latter case means each port can run full bandwidth at the same time.

The PCIe card itself can share lanes for bandwidth as a pool, or it can essentially give each lane to a different component of the card. If you have a single controller, then all lanes would have to pool to that controller. If you have four lanes, then four controllers could have their own lane (which is the best situation). Four lanes might be enough for perfect throughput of four root HUBs of USB3.1 gen 1 (5Gb/s) because the encoding of the USB itself will drop the actual requirement on PCIe lanes to less than 5Gb/s…if USB actual throughput is 80% or less on 5Gb/s, then in theory a PCIe rev. 2 lane could meet that demand (I don’t know what the encoding is on USB3).

Once all of this is considered, then it is only a question of whether the host computer can service the drivers fast enough. If you were to add too many root HUBs, then the CPU or whatever consumes the data would become the bottleneck.

NOTE: 8 ports of fully utilized USB3 5Gb/s would require 8 revision 2 PCIe lanes. If you achieve this, then there is a reasonable possibility that any inefficiency at all in drivers or programs consuming the data would result in dropped data.

One more comment useful to your case: More expensive cameras use hardware compression and encoding. The actual bandwidth of compressed video is far less than that of uncompressed video. Cheaper cameras tend to not have hardware compressed video. In your case a camera with hardware compression could provide a lot of room to do what you want.

linuxdev,I’m greatly appreciate you for your patience and detailed exposition.But I still have something I don’t understand.For example,you say ‘When you have a PCIe card the manufacturer can choose to provide one root HUB shared by four ports, or the manufacturer can choose four ports with their own root HUB’,does the ‘PCIe card’ means a converter which convert PCIe interface to several USB3.0 interfaces? Moreover,does ‘the manufacturer’ means TX2 or someone else? So,considering the TX2,which way does it provide:one root HUB shared by four ports or four ports with their own root HUB?

Whoever manufactures the PCIe card controls the design of the root HUB and port layout. So far as the host of the PCIe slot goes (the Jetson) the host will see one or more root HUBs on the PCIe card…and whether that card contains one root HUB and many ports sharing a HUB, or whether that card contains many root HUBs and each with its own independent port is up to the card designer. The host itself cannot change that.

I’ll give you some “rules of thumb”…meaning that there all kinds of ways this might be wrong, but it is a useful way of looking at it and deciding what to research.

Two cards with the same number of ports and specifications, but one being much cheaper than the other, is often from the cheaper card having only a single root HUB to share many ports. Both could be 4-port, but only the one with one root HUB per port would be able to maintain USB bandwidth on all ports simultaneously. This is similar to an external USB HUB where a single cable cannot provide all ports full bandwidth at the same time (when you are sharing low bandwidth items like keyboards and mice, then there is no problem with single root HUB sharing…but with cameras this is bad).

Two cards with the same specs of multiple ports, but a single root HUB would be compatible with a PCIe x1 lane design…the bottleneck in this example is the PCIe end. A card with multiple root HUBs would probably require one lane per root HUB so as to be able to take advantage of this. Two cards with multiple ports, where one card uses more lanes, is probably both more expensive and actually capable of doing what you want.

At 5Gb/s USB3 speeds a single revision 2 (PCIe v2 or rev. 2) is probably sufficient to maintain bandwidth. A 10Gb/s spec probably requires two lanes with PCIe v2, and a single PCIe v3 lane is almost enough, but not quite…and the TX2 is only rev. 2, so getting a rev. 3 card would imply it throttles speed back to rev. 2. Consider only PCIe rev. 2 or higher, but treat the card as if it is rev. 2 in cases where it is actually rev. 3 (rev. 4 is not out yet, but Xavier can handle that if you ever find such a card).

There may be limitations on how much throughput the host itself is able to achieve. There will be cases where your CPU core handling I/O works great with your USB ports…until perhaps you also need low latency simultaneous gigabit network (where the two compete). So there is actually a chain of things to consider, you can’t be absolutely certain about whether it will work, but certain conditions (such as a single shared root HUB instead of independent HUBs) will guarantee failure.

1 Like

Oh,linuxdev,under your enthusiastic help,I have a much deeper understanding about this.Could you help me judge whether what the following states are right or not:

First of all,it has nothing to do with Jetson tx2 about whether there are noly one root HUB or many root HUB but depends on the PCIe card we purchase(the PCIe card is just a external card which is not a part of original Jetson tx2 and the PCIe card is just a converter which convert PCIe interface to USB3.0 interfaces).

Second,I looked into some info on the Internet and I found that the Jetson tx2 has four-way serial channel(Is that right?).So can it be understood as the following: Since the Jetson has four lanes,I can buy a PCIe card which has four root HUB,so there are four USB3.0 ports,and each of the ports can consumes the full of the bandwidth at the same time(dont’t consider the throughput or something like this).

Third,if the above states are right,when I connect a PCIe card which has four root HUB on the Jetson tx2,then there are four USB3.0 ports with each of the ports can consume the full bandwidth at the same time.Now,I add four USB expanders which can expand each USB port to two ports(now totally 8 ports because each USB port is expanded to two and there are four USB ports).Then the situation whether goes the following or not:each of the two ports which are connected to the same root HUB can share the bandwidth together and every two ports which dont’t belong to the same root HUB are still independent.

Above the three questions are urgent for me.I really appreciate it if you could help me to solve these and I hope to recieve you reply as soon as possible.Sincerely!

1 Like

For PCIe expansion it is correct that root HUB count depends on the PCIe card and not on the Jetson. A good design has one root HUB per port. Less expensive cards tend to have multiple ports for a single controller (but prices will have come down over the years so it doesn’t mean a card has to be expensive). “Controller” could be considered an alternate term for “root HUB” (“root HUB” is the topology, “controller” is the name of the chip).

The PCIe slot on the dev carrier board has available 4 lanes rated to revision 2. Thus a four port PCIe card using four lanes is likely a card which has four independent root HUBs (you would still have to verify this). In this case all four ports would be able to run at full bandwidth. More ports without more root HUBs would reduce bandwidth and not all ports (when there are more than four ports) would be able to operate simultaneously at full speed speed.

Any external HUB you connect to expand port count will cut down available bandwidth. A root HUB with an external four port HUB would imply each port can consume only one quarter of the available bandwidth of the rated speed. There is no chance of adding an external HUB and maintaining speed…only a root HUB can do what you want. A four port PCIe expansion might do what you want, an external four port HUB will always fail for your purposes.

USB has a “tree topology”. This means that there is one root location which all communications must pass through. Any HUB connected within the tree, other than that single root, will always compete with other HUBs or ports. All combined ports which go through the root will be limited by the shared root.

Hi,linuxdev.Since the PCIe slot Jetson tx2 has available 4 lanes,does it mean that any PCIe card(has four ports) which converts PCIe interface into USB3.0 interfaces can have each port consumed the full bandwidth no matter whether it has many root HUBs or not?

If not,you say that ‘For PCIe expansion it is correct that root HUB count depends on the PCIe card and not on the Jetson’, and so if I can think that I can own an arbitrarily number of ports with each port has one root HUB as long as I have such a PCIe card with so many root HUBs. Is that reasonable? For example,can I have 12 port with each port has one root HUB?

By the way,you say the ‘PCIe expansion’ and ‘PCIe card’,do they all mean the card which convert PCIe interface into USB3.0 interfaces or the card which convert PCIe interface into more PCIe interfaces?

FYI, a PCIe “expansion” is just a card plugging in to the PCIe slot. It isn’t hard wired to the host. PCIe “card” and “expansion” are synonymous for this context.

NOTE: There is old naming “USB3” for 5Gb/s before overhead. The newer naming is that it is all USB3.1, but USB3.1 gen. 1 is 5Gb/s, and USB3.1 gen. 2 is 10Gb/s. Everything in this context is either “USB3” or “USB3.1 gen. 1” and means the same thing.

A single USB3 generation 1 (5Gb/s before counting overhead…this is what the TX2 has) root HUB on a PCIe expansion can consume the full bandwidth of a single PCIe rev. 2 lane. If the PCIe lane were to revert to rev. 1, then it would require two lanes. A PCIe rev. 2 card hosting a single USB3 controller is incapable of using more than a single PCIe rev. 2 lane even if four lanes are used. Some counterfeit cards put a single controller on a 4-lan rev. 2 card to make people think that four lanes are used. If it is a real manufacturer not pulling tricks on you, then it is very likely a 4-lane PCIe card with four USB3 ports will have four controllers and thus use all four lanes.

Each controller has a single pool of bandwidth. Put one port on it, and that port can use the full bandwidth. Double the number of ports on a single controller to two, and each port only gets half. Double that to four ports, and your 5Gb/s ports can now only provide 1.25Gb/s (5/4). You can keep adding ports, but if they share a controller, then you won’t get something for free.

There are four port cards available with a single root HUB. Those won’t work for you even if it fits a four lane PCIe slot. There are four port cards with four root HUBs. These will work if they go to a four lane PCIe slot.

You can have a card with 12 root HUBs. That works on the USB side of things. However, this won’t work on any PCIe slot with revision 2 speed if that slot isn’t at least 12 lanes (and the closest size available which can handle 12 is PCIe x16…nobody makes x12). At revision one a PCIe x16 will handle only 8 root HUBs. The Jetson has PCIe x4 at rev. 2, so that means you cannot use more than a four port (with for root HUB controllers) card without losing bandwidth. I do not believe there is any way to get 12 USB3 root HUBs running at 5Gb/s on each port for a TX2 (the only possibility would be a custom carrier board, and I doubt even that would do it).

The Xavier handles PCIe rev. 3 cards with 8 lanes. 12 USB3 at 5Gb/s implies a 60Gb/s pool is required. Dividing in 8.5Gb/s (and this leaves out overhead…so it isn’t actually this good) would mean 7 lanes could almost handle it. Eight lanes of PCIe rev. 3 could handle 12 port USB3 at 5Gb/s per port. The TX2 can’t even come close to that, the Xavier could do it.

There are internal ports, but how they are used or connected may mean not all of them are useful. There are all kinds of other issues when you run that much bandwidth, e.g., latency might matter, and the more collisions a driver has with other drivers trying to run at the same time the more your standard deviation of what actually gets utilized goes up. The Xavier might not even handle some situations when latency is important, but it is probably the most capable of any embedded system available, and is certainly more capable of lowering latency than is the TX2. If you switched to Xavier, using an 8 lane rev. 3 card of 12 ports, it might work.

Oh,linuxdev.With your help,I almost understand this.I cannot appreciate you much more for your detailed explanation.

Now,another question comes to me which is stated in the following:

Nowadays,are there such a PCIe card which converts PCIe interface into several USB3.0 interfaces and it has four USB3.0 controllers(i.e.,four root HUBs) at the markets? If does,what’s the key words I can search for purchasing?

What follows is more a methodology on finding the right card. I don’t have any of the cards listed so I can’t claim I know for sure one works, but I the ones which I say won’t work definitely won’t work for your purposes.

There are actually many PCIe cards with 4 USB3 ports and 4 USB3 controllers, but most of what you start with will be trying to get away with deceptive marketing (the individual facts they list are true, but will only be a partial truth and not the whole truth). Simply start by looking for PCIe cards advertising four USB3 ports. Then check the specs to see that they use at least four PCIe lanes (PCIe x4), and that the lanes are PCIe rev. 2 or higher (a.k.a., v2 or higher). The Jetson PCIe will only use up to revision 2, but a revision 3 card will work fine at the rev. 2 speed; a rev. 1 card will not be fast enough when there are only four lanes. You could have more lanes, e.g., PCIe x8…which would actually work if the card is rev. 1, but the TX2 does not have 8 lanes. Summary: PCIe rev. 2 or faster, using 4 lanes, i.e., “PCIe x4 using revision 2 or faster”.

The question as to the controllers is a tough one since this is often not advertised. If the above is met it is highly likely there are four separate controllers…but some counterfeit cards will occupy four lanes and not actually use all four. My assumption is that you are not shopping somewhere that sells counterfeits.

I don’t know if you can see items where you are at, but this is the start of some listings:

Just for reference (I don’t know if you get wikipedia where you are) here is a URL with a picture of the slots for each number of lanes x1, x4, and x16:
(the bottom of that URL also shows the picture of a PCIe x4 card)

Look closely at the x4 slot. The portion of the edge card closest to the outside of the PC, up through the slot, is the same on all lane counts. The length after the notch (towards the inside if just a desktop PC) is how you can spot the lane count if no lane count is given. The x1 is really short for that length past the notch, the x4 doesn’t look long, but it is definitely longer than x1.

If you see a picture you’ll know which lane count the card is even if the lanes are not specifically mentioned. If you see an advertisement stating the PCIe lane can achieve the full 5Gb/s, then you do not know if this is what you want (although it might be)…because all revision 2 or 3 lanes can do this…but you need to know for sure that there are four lanes, which goes beyond bragging about how fast one lane can run.

Look closely at this particular card:
…which is very inexpensive. The advertisement says:

Up to 5.0 Gbps Data Transfer Rate

This is quite misleading…if you ask if a port can run the full 5Gb/s, then yes every port can. If you ask if the PCIe lanes are revision 2, then it must be since it is capable of 5Gb/s. However, it is only a single lane (x1) slot…which means they are trying to avoid emphasizing that no two ports can handle 5Gb/s at the same time. This card is a bit of a joke unless you are using a mouse and keyboard for low bandwidth (in which case it reverts to USB1.1 anyway). Never buy a four port USB3 card like this with a single lane. This card uses only a single PCIe lane (x1), and thus it isn’t possible for all four ports to have independent root HUBs.

This is the most expensive card in that list, but this too is only PCIe x1 and is not capable of having four independent controllers even if it ran at revision 3 (and the TX2 cannot run revision 3):
…this card is also misleading and won’t work.

In fact most of the cards I see are only PCIe x1. Here is one which is actually x4:±sata&oq=%224-Port%22+%22USB+3.0%22+%22PCI+Express%22+%22x4%22±sata&prds=paur:ClkAsKraX5Z7jUz1Z4KpAmAxC42snAypDRRLG6-FUJ39BxCNRTIMMQC1uovjuOdWyD3F4DGf6K8rd8f5y63t9lb45nrRtWLt7-wCLDC0qhnJC0hZ8HbCWWtJyBIZAFPVH73cfMtLLfc1CQatB8-tP9Anffc3lQ&sa=X&ved=0ahUKEwimyu_Ej4zgAhUl6IMKHb4kBe8Q8wIItAI
…listed as “HighPoint RocketU 1344A USB Adapter - PCIe 3.0”. Notice the price is much higher than the others which are only PCIe x1. Incidentally, this advertisement uses the acronym “HBA”…which is Host Bus Adapter…and is what I’ve been calling a “PCIe card”.

This latter Highpoint RocketU card has something interesting…it actually supports “USB3 gen. 2”…which is a USB connector capable of 10Gb/s. The PCIe would have to be rev. 3 before it could stand a chance of actually using 10Gb/s per port, but if your USB camera is USB gen. 1 (5Gb/s), then this would revert to 5Gb/s without any error. The four ports on this card can handle 5Gb/s on each port simultaneously if the PCIe x4 slot runs at least at revision 2 speeds…and this is what the TX2 runs at. This card would work…if the drivers are available (and I don’t know if they are…more on that topic in the next paragraph).

I searched for this card on, and here it is:

When I went into the specifications the card mentions it is compatible with Linux. So you know it will work, at least on a desktop PC. It is highly likely it will work on the Jetson as well, but you might have to compile a kernel module. This is a serious product and not a joke if you need all four ports to work simultaneously.

I do find something odd about that card…most cards which advertise USB3.1 gen. 2 10Gb/s USB ports do so with a type-C connector, but this uses the typical type-A (I’m going by the picture…I don’t see the connector type explicitly listed). Normally I associate type-A with only 5Gb/s, but that’s ok…it doesn’t hurt unless your camera wants a type-C connector. It is unlikely any of the cameras you look at require type-C, so you are actually better off with the type-A connectors. I’m going to guess that their marketing people purposely left out that it doesn’t use type-C connectors…but for your case that is likely an advantage.

Thank you very much,linuxdev.

Hello,linuxdev. I’m sorry to bother you again.But I have one more question to be solved.

I’ll buy some USB3.0 expander devices.The question is about USB expanders devices(i.e. extend one USB3.0 interface to several USB3.0 interfaces).
Nowadays,in the market,can all the USB expanders devices actually extend one USB3.0 interface to several USB3.0 interfaces?Put it another way,can I say that in the market,it doesn’t exit such USB expanders devices,some of whose extended USB3.0 interfaces don’t work at all,i.e. if I connect one USB camera to the extended USB3.0 interface,actually tx2 doesn’t recognised the camera just becuase the USB expanders device.

Or probably the following situation:I bought a USB expanders device which extend one USB3.0 interface to two USB3.0 interfaces.And the two extended USB3.0 interfaces are intended to connect one USB camera respectively.However,one camara can only share just very few bandwidth because of the USB expander device,so it can’t satisfy the transfer rate of the camera.Does this situation exit?

I’m not sure what it is that you are calling an expander. Are you talking about a USB HUB?

FYI, USB is a tree topology where the computer itself talks to the root HUB. Several ports can be wired to share that HUB, and in that case total bandwidth must be shared. Power deliver would also be shared and can be a serious problem, and so there are also some HUBs sold which are externally powered (external power is highly recommended).

The port of one HUB can have another HUB connected. A port of that other HUB can have yet another HUB, and so on. All combined HUBs and ports share the bandwidth of the single root HUB.

Some motherboards provide header pins for attaching extra ports…those ports share some root HUB on the motherboard. A port is a physical connection, a HUB is what routes and controls that connection. The root HUB is in charge of everything downstream. Extra logic is added when extra ports are used with a single controller.

Regarding power requirements there are times when too much power being drawn will cause two otherwise functioning USB devices to not work together. At least one of the devices would basically be powered down.

So to give a better answer you would need to give a more detailed description of what it is you mean by an expander.