Hi everyone,
I just created hardware acceleration support for libfreenect2 on Jetson. I am able to obtain 70Hz depth processing and 60Hz RGB processing rate. It is slower with visualization at ~56Hz for both.
Here is a demo video: GPU acccelerated libfreenect2 for Kinect 2 on Jetson TK1 full performance - YouTube.
The code is available at https://github.com/xlz/libfreenect2
Instructions: https://github.com/xlz/libfreenect2/wiki/Jetson-TK1-HOWTO
This is pretty exciting!
I tried it, and ran into an issue.
When I compile and run Protonect, I get an error:
terminate called after throwing an instance of 'std::runtime_error' what(): JPEG parameters struct mismatch: library thinks size is 560, caller expects 536
Any idea what I did wrong?
It is using jpeglib.h from somewhere else, not the Tegra JPEG headers. Check if you have headers in depends/nv_headers/. And maybe touch tegra_*.cpp; make VERBOSE=1 and check the specific build command to see if jpeglib.h is found in depends/nv_headers/ or /usr/include.
(Edit) To clarify: nv_headers/jpeglib.h defines different structs than the standard one. The error message here is talking about the struct size.
A lot of people have had interest in this, so I’ve added the above URLs to the Jetson links sticky thread.
I am using this for a school project, quite excited to get SLAM working with Kinect 2 on the Jetson. But, when I run Protonect I get a bunch of “[DepthPacketStreamParser::handleNewData] subpacket incomplete (needed 298496 received 84784)”. Looking at the code, it looks like this is not something i’d want to be seeing. Is this a known error/any hints?
I’ve answered the above question here.
I had similar issues running xlz’s great code. I think there have been changes to libusb or libfreenect that cause this, the one that seems to be causing this is MAX_ISO_BUFFER_LENGTH.
In my case, the built in USB 3.0 port didn’t receive depth images, but an external miniPCIE USB 3.0 card with a Renesas chipset worked correctly after changing the MAX_ISO_BUFFER_LENGTH in libusb. I wrote up an install script and a blog post: http://jetsonhacks.com/2015/02/26/install-kinect-v2-using-libfreenect2-on-nvidia-jetson-tk1/
I’m kind of looking for some information related to this regarding both the mPCIe slot and USB. If you run “lspci -vv” for the USB mPCIe card, one of the information blocks for “Capabilities” will contain several “LnkCap/LnkSta/LnkCtl” blocks naming “Speed”. Some are for capabilities of the mPCIe card, others are for actual speed being used. For your working USB3 full-length mPCIe card, could you tell me what those sections show?
Also, I do not have a kinect, but can I verify if the device is designed to run best on USB3 (some devices say USB3 capable, but they are just USB2 telling customers that they work on USB3…I’m hoping your kinect model takes advatange of USB3 speeds when available)? With that connected to the mPCIe USB port, could I see the output of both “lsusb” and “lsusb -t”? The latter is a tree view which shows each device, it’s parent, and child if it has child nodes (speed shows up as something like “480M” for USB2 or “1.5M” for USB1.1).
Hi Linuxdev,
The Kinect V2 is a true USB 3.0 device, little blue USB connector and all. It spits out three video streams simultaneously: One is RGB color 1920x1080 @ 30hz, the other two are grayscale 512x424 @ 60 hz (IR and depth images). It also has a microphone array for which it streams data. @xlz was kind enough to write some great CUDA code to speed up the decoding of the color stream and depth image streams (The RGB stream is JPEG encoded). Thus the Kinect requires the USB ‘SuperSpeed’ isochronous mode, which is the ~5Gb/s rating that the chip manufacturers like to spew. To it’s credit, the Jetson comes close to keeping up and displaying the images in real time at full frame rates with xlz’s modifications.
Here’s the other information you requested:
$ lspci -vv
USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02) (prog-if 30 [XHCI])
<snipped bit>
Capabilities: [a0] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <4us, L1 unlimited
ClockPM+ Surprise- LLActRep- BwNot-
LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Kernel driver in use: xhci_hcd
$ lsusb
Bus 004 Device 003: ID 2109:0812
Bus 004 Device 002: ID 2109:0812
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 006: ID 05ac:021d Apple, Inc. Aluminum Mini Keyboard (ANSI)
Bus 003 Device 005: ID 05ac:1005 Apple, Inc.
Bus 003 Device 004: ID 046d:c018 Logitech, Inc. Optical Wheel Mouse
Bus 003 Device 003: ID 2109:2812
Bus 003 Device 002: ID 2109:2812
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 003: ID 045e:02d8 Microsoft Corp.
Bus 002 Device 002: ID 045e:02d9 Microsoft Corp.
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 045e:02d9 Microsoft Corp.
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
$ lsusb -t
/: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=tegra-xhci/2p, 5000M
|__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 5000M
|__ Port 4: Dev 3, If 0, Class=Hub, Driver=hub/4p, 5000M
/: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=tegra-xhci/6p, 480M
|__ Port 3: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 4: Dev 3, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 3: Dev 4, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
|__ Port 4: Dev 5, If 0, Class=Hub, Driver=hub/3p, 480M
|__ Port 2: Dev 6, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
|__ Port 2: Dev 6, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 5000M
|__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/1p, 5000M
|__ Port 1: Dev 3, If 0, Class=Vendor Specific Class, Driver=, 5000M
|__ Port 1: Dev 3, If 1, Class=Vendor Specific Class, Driver=, 5000M
|__ Port 1: Dev 3, If 2, Class=Audio, Driver=snd-usb-audio, 5000M
|__ Port 1: Dev 3, If 3, Class=Audio, Driver=snd-usb-audio, 5000M
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 480M
|__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/1p, 480M
Hope this helps
Excellent information. The mPCIe is performing at maximum speed on a device which actually uses the speed.
For USB you’ll notice there are four root HUBs, two of which are designated as USB3. The manufacturer ID is listed on all of the root HUBs as 1d6b, which I believe implies the USB ASIC controller chip is the same in both the mPCIe card and in the tegra124…the mPCIe card does not require a different driver from the built-in USB controllers (although drivers could be different if desired). The tegra124 itself has two root HUBs, which means the add-on card also contains two root HUBs, for four total HUBS from lsusb…two native, two mPCIe. It appears this controller chip has dual controllers for separate use on the two ports rather than a single controller with “TT” transaction translators.
For performance checks it’s necessary to know which port the kinect2 is plugged into…the tree of lsusb shows which device is a parent/child of another device. Since the kinect2 has three video streams it makes sense that up to three devices of the same manufacturer (Microsoft) are compound video devices in a single physical package and cable. Bus 1 and 4 HUBs have nothing but bus ports, eliminate those. Bus 3 contains only HUB or HID devices (this class includes keyboard/mouse) running at 1.5Mbit/s. The HUB being used as parent for kinect2 must be bus 2 device 1, the mPCIe port for kinect2.
Has this particular Jetson been configured for USB3 on the native full-sized USB connector (the second USB3 listed could certainly be the other port of the mPCIe or it could be the native port)? If so, what is lsusb and lsusb -t when all remains the same except for plugging the kinect2 into the native USB3 port?
@linuxdev, thanks for your analysis. The mPCIe card has a pigtail with two USB 3.0 connectors on it, as you surmised the Kinect was plugged into the mPCIe card. I apologize for leaving that information out.
The Jetson is configured for USB 3.0 on the native full size USB connector. There is a 3.0 hub plugged into the connector, with a mouse and keyboard plugged into the hub. The mouse and keyboard of course are USB 2.0. When the Kinect is also plugged into the hub:
ubuntu@tegra-ubuntu:~$ lsusb
Bus 004 Device 003: ID 2109:0812
Bus 004 Device 005: ID 045e:02d8 Microsoft Corp.
Bus 004 Device 004: ID 045e:02d9 Microsoft Corp.
Bus 004 Device 002: ID 2109:0812
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 006: ID 05ac:021d Apple, Inc. Aluminum Mini Keyboard (ANSI)
Bus 003 Device 005: ID 05ac:1005 Apple, Inc.
Bus 003 Device 004: ID 046d:c018 Logitech, Inc. Optical Wheel Mouse
Bus 003 Device 003: ID 2109:2812
Bus 003 Device 007: ID 045e:02d9 Microsoft Corp.
Bus 003 Device 002: ID 2109:2812
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
and:
ubuntu@tegra-ubuntu:~$ lsusb -t
/: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=tegra-xhci/2p, 5000M
|__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 5000M
|__ Port 1: Dev 4, If 0, Class=Hub, Driver=hub/1p, 5000M
|__ Port 1: Dev 5, If 0, Class=Vendor Specific Class, Driver=, 5000M
|__ Port 1: Dev 5, If 1, Class=Vendor Specific Class, Driver=, 5000M
|__ Port 1: Dev 5, If 2, Class=Audio, Driver=snd-usb-audio, 5000M
|__ Port 1: Dev 5, If 3, Class=Audio, Driver=snd-usb-audio, 5000M
|__ Port 4: Dev 3, If 0, Class=Hub, Driver=hub/4p, 5000M
/: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=tegra-xhci/6p, 480M
|__ Port 3: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 1: Dev 7, If 0, Class=Hub, Driver=hub/1p, 480M
|__ Port 4: Dev 3, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 3: Dev 4, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
|__ Port 4: Dev 5, If 0, Class=Hub, Driver=hub/3p, 480M
|__ Port 2: Dev 6, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
|__ Port 2: Dev 6, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 5000M
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 480M
Right now I’m trying to sort out the USB tree related to the kinect2 to make a comparison between the error-free add-on mPCIe adapter versus the error of dropped depth images on the native port. Your situation is unusually good for this because you have a perfectly working reference via the add-on card which uses the same driver and probably same chip/die as in the tegra124…plus actual use truly requires full USB3 bandwidth (this is a compound USB3 device and very likely had the mPCIe run only at 2.5GT/s speeds bottlenecks could have been from mPCIe and not USB…it all runs at 5GT/s so any bottlenecks are via USB and not via internal data lanes).
While looking at what is connected where in this latter information, can I confirm nothing is connected to the micro USB connector?
I see some audio class devices, is this known to be part of kinect2 or some other attached device, e.g., something with speakers (I’ve never seen kinect devices in person, so I have no idea if they have audio)?
Although I’m fairly certain the micro connector is one of the two unused USB ports, I see one other unused port…probably one of the pigtail cables of mPCIe previously occupied by kinect2. However, if all items were removed from mPCIe the only root HUB with non-HUB children would be the full-sized A connector. Is anything at all still on the mPCIe cables after moving the kinect2 to the HUB of the full-size A connector? I kind of expected to see 4 root HUBs with only one in use, that one being the full-sized connector…instead there are two root HUBs in use and two not in use, so I must have an incorrect idea of how things are connected.
A little background: To interface with the Kinect, I’m using libfreenect2. This library uses libusb-1.0 to interface against the kernel driver. libusb-1.0 did not work properly against the LT4 19.3 release, but did against 21.2 back in January. I was able to get libfreenect2 to work off of either the built in port or the mPCIe card. @xlz reported that he was able to get the on board port working just fine when he published his work. Unfortunately this changed at some point, my guess would be that either the libfreenect2 to libusb interface or libusb changed enough that there is some issue that causes the on board port to stop working with libfreenect2. Either way, I’m pretty sure it’s upstream from the kernel driver itself.
Early adaptors on Linux were reporting that only NEC/Renasas USB chipsets were working with the Kinect V2, that was the reason I bought the mPCIe card. The current libusb-1.0 required changes to get the mPCIe card to work. Here’s the blog entry: Install Kinect V2 using libfreenect2 on NVIDIA Jetson TK1 - JetsonHacks , but the summary is that MAX_ISO_BUFFER_LENGTH in usbfs.h caused issues. In the published version, the size is 49152 * 128, the mPCIe card/Jetson didn’t like that, but works when the size is 49152. A little bit like building on quicksand, but not unusual when the project is not the mainstream solution.
To answer some of your questions:
Previously, the micro usb port was occupied, I forgot to take it out after flashing. The Kinect V2 has a microphone array which accounts for the audio streams you noticed.
With that said, I find it interesting that when only a USB 3.0 hub is connected with keyboard and mouse; No mPCIe, No microusb:
ubuntu@tegra-ubuntu:~$ lsusb
Bus 002 Device 003: ID 2109:0812
Bus 002 Device 002: ID 2109:0812
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 006: ID 05ac:021d Apple, Inc. Aluminum Mini Keyboard (ANSI)
Bus 001 Device 005: ID 05ac:1005 Apple, Inc.
Bus 001 Device 004: ID 046d:c018 Logitech, Inc. Optical Wheel Mouse
Bus 001 Device 003: ID 2109:2812
Bus 001 Device 002: ID 2109:2812
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
ubuntu@tegra-ubuntu:~$ lsusb -t
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=tegra-xhci/2p, 5000M
|__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 5000M
|__ Port 4: Dev 3, If 0, Class=Hub, Driver=hub/4p, 5000M
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=tegra-xhci/6p, 480M
|__ Port 3: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 4: Dev 3, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 3: Dev 4, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
|__ Port 4: Dev 5, If 0, Class=Hub, Driver=hub/3p, 480M
|__ Port 2: Dev 6, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
|__ Port 2: Dev 6, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M
Then plug in the Kinect:
ubuntu@tegra-ubuntu:~$ lsusb
Bus 002 Device 003: ID 2109:0812
Bus 002 Device 005: ID 045e:02d8 Microsoft Corp.
Bus 002 Device 004: ID 045e:02d9 Microsoft Corp.
Bus 002 Device 002: ID 2109:0812
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 006: ID 05ac:021d Apple, Inc. Aluminum Mini Keyboard (ANSI)
Bus 001 Device 005: ID 05ac:1005 Apple, Inc.
Bus 001 Device 004: ID 046d:c018 Logitech, Inc. Optical Wheel Mouse
Bus 001 Device 003: ID 2109:2812
Bus 001 Device 007: ID 045e:02d9 Microsoft Corp.
Bus 001 Device 002: ID 2109:2812
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
ubuntu@tegra-ubuntu:~$ lsusb -t
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=tegra-xhci/2p, 5000M
|__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 5000M
|__ Port 1: Dev 4, If 0, Class=Hub, Driver=hub/1p, 5000M
|__ Port 1: Dev 5, If 0, Class=Vendor Specific Class, Driver=, 5000M
|__ Port 1: Dev 5, If 1, Class=Vendor Specific Class, Driver=, 5000M
|__ Port 1: Dev 5, If 2, Class=Audio, Driver=snd-usb-audio, 5000M
|__ Port 1: Dev 5, If 3, Class=Audio, Driver=snd-usb-audio, 5000M
|__ Port 4: Dev 3, If 0, Class=Hub, Driver=hub/4p, 5000M
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=tegra-xhci/6p, 480M
|__ Port 3: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 1: Dev 7, If 0, Class=Hub, Driver=hub/1p, 480M
|__ Port 4: Dev 3, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 3: Dev 4, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
|__ Port 4: Dev 5, If 0, Class=Hub, Driver=hub/3p, 480M
|__ Port 2: Dev 6, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
|__ Port 2: Dev 6, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M
Then when you add the mPCIe back (USB 3.0 hub with mouse and keyboard connected, no Kinect):
ubuntu@tegra-ubuntu:~$ lsusb
Bus 004 Device 003: ID 2109:0812
Bus 004 Device 002: ID 2109:0812
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 006: ID 05ac:021d Apple, Inc. Aluminum Mini Keyboard (ANSI)
Bus 003 Device 005: ID 05ac:1005 Apple, Inc.
Bus 003 Device 004: ID 046d:c018 Logitech, Inc. Optical Wheel Mouse
Bus 003 Device 003: ID 2109:2812
Bus 003 Device 002: ID 2109:2812
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
ubuntu@tegra-ubuntu:~$ lsusb -t
/: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=tegra-xhci/2p, 5000M
|__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 5000M
|__ Port 4: Dev 3, If 0, Class=Hub, Driver=hub/4p, 5000M
/: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=tegra-xhci/6p, 480M
|__ Port 3: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 4: Dev 3, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 3: Dev 4, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
|__ Port 4: Dev 5, If 0, Class=Hub, Driver=hub/3p, 480M
|__ Port 2: Dev 6, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
|__ Port 2: Dev 6, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 5000M
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 480M
My guess would be that the mPCIe card is capable of running both of it’s hubs at 3.0 speeds, but the kernel driver configures it similarly to the built in ports.
A complication with figuring out USB is the whole hot plug scheme causing numbering and enumeration of devices in general to sometimes change, depending on order of install (including order of driver load as well as physical devices). An additional possible explanation of the mPCIe not showing at all is that perhaps the driver is loaded “on demand” and has never shown up without having been required at least once…in which case plugging something into each port of the mPCIe card and then removing it could change whether it shows up or not.
One simplification of the issues above is that the cable used for flash is a micro type “B”, and will normally be the same as no cable connected…the only micro connector confusion would come from a micro-A connector which is not shipped with Jetson. The included flash cable will always be ignored during regular non-flash operation, it simply isn’t wired the same as a type “A” connector and cannot function in a way which shows up on lsusb.
Trying to sort out what is connected where using hot plug enumeration more or less requires knowing which connector is physically used and comparing it to the tree view. Since I can’t see what is physically connected where I tend to “over-verify” connections by asking a lot of questions.
Looking at a sorted diff of the non-tree list from prior to the “Then plug in the Kinect:”, along with the actual “Then plug in the Kinect:” listing shows the kinect2 to be the three devices with the description “Microsoft Corp.”. Oddly, this compound single physical device with three internal functions shows up partly on bus 1, and partly on bus 2…this I didn’t expect because bus 1 and bus 2 are tied to separate USB root HUBs (I would think that all 3 functions were connected to the same connector and by implication the same root HUB…this might to be related to an error I’ve been chasing if my concept of which connector has what plugged in is correct).
I really need something like this kinect2 to debug, but let me just verify one more time the middle lsusb with label “Then plug in the Kinect:”. In this case nothing was connected to the mPCIe? The kinect2 was plugged into the same USB3 HUB that the keyboard and mouse are plugged into? This HUB is in turn plugged into the full-size native USB connector?
Right, it seems a little counter intuitive to me also.
To clarify from the last post:
First lsusb set:
Second lsusb
Third lsusb set:
This last post clarifies enough to show the issue I’ve been chasing…basically one of where USB performance differs: Indicated versus actual performance. Reference is to the most recent USB listings, as clarified by the above post.
For the first listing with no mPCIe card, and nothing connected to the micro A/B, the only root HUB listed at USB3 speed is the micro A/B. Proof is in the tree view showing 5000M USB3 speed for the branch with only HUB connections, and no device connections; the other branch shows 480M USB2, and contains all connected devices.
The difficulty has been in figuring out whether this is a listing issue (tools such as lsusb), a reporting issue (tegra-xhci driver providing incorrect information to tools and sysfs), or if the information is correct and USB3 has been set on the wrong port.
If performance can be definitively shown to be USB3 on the full-sized connector, then we know it is either a tool or reporting issue, and that USB3 itself functions as intended. Actually reaching USB3 bandwidth is far more difficult than it sounds though, as this requires a more or less continuous data stream capable of constantly saturating slower USB2 functions. There is no such thing as an external USB3 hard drive enclosure with drives fast enough to do this without resorting to something like RAID0 of many drives…burst speeds don’t count. Even gigabit NICs are difficult to consider, especially on R21.x.
This kinect2 system is an unusual great match to testing requirements since it uses multiple hidef video streams which truly require the full bandwidth almost all of the time. The external mPCIe USB3 HUB running at gen. 2 PCIe data speeds provides a best case comparison and shows the same tegra-xhci driver is quite capable of handling the full bandwidth without dropping data. So this stage shows the driver is delivering what it should when we can ignore any specific connector.
As far as physical connectors go, is it correct to state that the kinect2 functions without frame drops on the native full-size USB connector once the libraries have any corrections needed? This seems to be the case, though I’ll still ask since I don’t have the same hardware. If this is correct, it means that the full-size USB connector is also functioning as intended at USB3 speeds…the question then divides into whether lsusb and sysfs are in error, or if tegra-xhci is working correctly except for reporting the wrong information.
Since this one physical device with three functions must all be on one bus, but since it seems at times some of the functions can appear in the tree view as spread across two buses, that display is at fault. I do not believe lsusb and sysfs source code are this bad, as they’ve worked quite well on many distributions and kernels without issue, and this code is shared. The evidence implies that if the performance is USB3 on the native connector then tegra-xhci is reporting incorrectly which is propagating out to sysfs and lsusb -t and indirectly makes these appear to be incorrect.
So, to repeat the key to all of this, is it correct that the dropped depth images does not occur on the full size A connector when related libraries are corrected?
The statement is correct. I have seen the Kinect V2 work with full size A connector previously.
Thank you for your great blog! I should clarify my demo was run with the builtin USB 3 port without extra plugin.
libfreenect2 does have issues with USB isochronous buffer size here and there. We have been tracking it here: New libusb version not working on AMD · Issue #164 · OpenKinect/libfreenect2 · GitHub
I will address the issues in your blog in my repo. Thanks!
You have posted a lot which I can’t read it all. Just speaking from my experience, I have also purchased this mPCIe USB3 controller from NEC:
“Syba 19-Pin USB 3.0 Header Mini PCI-Express Card with Female USB 3.0 Cable SD-MPE20142”
And it didn’t work at all even before connecting Kinect 2. But Kinect 2 worked with the builtin USB 3 controller the first time I tried it (first when L4T 19.3), so I didn’t bother more with this controller.
I don’t expect Jetson works with Kinect 2 all the time without a lot of fiddling. The drivers in L4T are not bullet proof and lag behind Linux upstream. When it doesn’t work, please try reconnecting a few times. If it still doesn’t work, please look at the console output of Protonect, dmesg, and USB autosuspend settings in sysfs, and post them. Reconnecting it might revert its autosuspend settings so verify it. Besides driver stability issues, isochronous transfer seems very delicate, so a few warnings are to be expected. My branch of libfreenect2 uses a new depth stream parser, and its warnings have specific causes.
About issues in your blog.
If you need more help, you are welcome to post here, or create issues in my repo.
Thanks for the tips xlz. Also, thank you for your CUDA code, it is very useful.
I’ll flash a Jetson and test out your code.
I do have another question. It strikes me that there is one variable in the equation I had not thought about previously.
Are you plugging the Kinect V2 straight into the Jetson, or are you using a USB 3.0 hub? If your are using a hub, which one?
Seeing that USB 3.0 out in the wild is still relatively rare, it could be that there are differences or issues with the hub that I am using when it has to run at speed. The hub could be causing issues in reporting also, but that seems a stretch.