Stereolabs ZED and Jetson

I don’t have a double buffer, but I have multiple buffers so when one is locked by user-space (my main loop), the timer callback function can use a buffer which is not locked and set a variable to direct the main loop that a "new image has been stored, and it is located in buffer #'. Thus the main loop can read which buffer has the latest image, lock that buffer, copy the image, and unlock that buffer and move on with processing of the image.

Maybe my approach is ‘sound’ but I am not executing it correctly. If you are interested in seeing some code, I’d be more than happy to send it to you.

Mostly I’m interested in knowing the source any data reliability issues…kernel space, versus user space, versus the camera’s ability to throw out data in high volume. Which kernel drivers are involved (other than USB)?

I’m not sure how to answer the question for you ;0) Do you have an example? I’m not sure what drivers are currently ‘active’ if that’s what you meant.

I have an ethernet cable plugged in, with no use on it that I am aware of. The HDMI display is active… I have usb mouse and usb keyboard plugged into the same usb hub which the camera is plugged into. I have tried multiple types of 3.0 HUBs, all with the same outcome.

Did you have to recompile the kernel or kernel modules with a configuration addition? Or was there a kernel driver requirement listed somewhere for this camera? I’m wondering what kernel feature is providing the data to your application (USB is part of that, but is already known).

No I did not have to recompile the kernel. I did flash the Grinch kernel, so that may have included drivers that I am not aware of, since that is a rather loaded kernel distribution.

Looking at the web site for the camera, I can’t see any listed kernel requirements other than USB. You might want to test with a non-Grinch kernel just to see what changes (make sure it still gets USB3 enabled and disable autosuspend). If everything is sent directly from the USB pipeline to user space, it’s possible for an interrupt to briefly break data transfer at a bad moment. In the case of a kernel driver it could be arranged to avoid data loss from interrupts at critical moments. You may want to ask the manufacturer if it is possible this is what’s causing those brief frame failures even with your current fixes.

New information on testing:

I wanted to see what would happen when I change the video display resolution, as I have been using 1920x1080 (16:9) at 2 different locations; same size monitor.

I dropped the resolution to 1280x800 (16:10), and so far no glitch. I can change back to 1920x1080 and it takes about 2 minutes, but the glitch returns. Change it back to 1280x800 and the glitch returns to un-noticeable, if it even exists at this resolution.

I’m wondering if the display driver was having trouble keeping up with the higher resolution, and my USB transfer issue could be a side effect of this.

To know for sure you’d have to be able to see the actual data when there is a glitch. Then you’d know if it is a producer of data or consumer of data which is falling short. Unfortunately, this is so much data that you can’t just save it to the hard drive without a lot of buffering (perhaps if you had a ramdisk which wasn’t a real drive it would work). Even so, if you have a means to save a brief clip of video in which you’ve seen the glitch, playback might tell you if data was bad/missing versus just unable to display it fast enough. Somewhere right at the 1920x1080 resolution a bottleneck is at its limit.

I have confirmed (to my knowledge) that the data corruption is happening not in the display but in the acquisition. I did this by displaying the original image and displaying the image rotated 45 degrees. Since a common corruption was a grouping of horizontal rows being garbled up then a display issue would have garbled rows in both the original and rotated image. However, the garbled data in the new image was also rotated (so this tells me that the data was corrupt prior to the image rotation function).

So the data acquisition is likely like key symptom, and changing display resolution may only be a band-aid but is helping.

A USB analyzer could tell you what data is going through, letting you know if it is the communications versus the receiving driver…even a USB2 analyzer is very expensive though, a USB3 analyzer is far more expensive. I still believe the USB pipe itself is reasonably well tested, and lower on the list of dropped data than would be failing to copy the data after USB transfer into memory. In the latter case if this is done in kernel space you would have some control over drivers not being interrupted via IRQ, but I don’t have a clue as to what driver might be used or how the data transitions from kernel space to user space. A query to the manufacturer of the specific question of what kernel code is used after USB has transferred data would make it easier to narrow.

On a related note, do you happen to know if this camera uses compression? If the data is compressed, the the loss almost has to be at decompression or copy of the decompressed data…USB3 would easily be able to keep up.

I did ask, and they said there is no compression done to the images. I also asked if they knew which kernel ‘code’ might be used after USB transfers and I’m still waiting for their dev team to get back to me on that. They also said they are trying to work with NVidia on this issue (also they had the issue where they were not able to utilize the full BW of USB3.0, so I know they were already working to get this solved as well).

With no compression all of the bandwidth really is needed, which sort of puts USB back in the possibility of dropping data…but until knowing what part of the kernel is consuming or copying the USB data it’s impossible to say.

Just wanted to also point out, I have been running on the lower resuolution all day maybe 1 noticeable frame corruption (I’m sure there are more when I’m not looking). But this is the most stable I’ve ever had this.

If you were to look at the Tegra K1 TRM and check out the block diagram near the bottom of page 12, you can see that the USB3.0 controller and the Display controllers interface through the same memory interface.

https://developer.nvidia.com/gameworksdownload#?search=Tegra%20K1%20Technical

I wonder if a display resolution of 1920x1080 @ 60 FPS combined with the 30FPS 1280x480 YUV uncompressed frame grabs from USB builds up enough bus contention (where the USB transfers are the ‘lower priority’ so end up suffering data loss).

I’m hoping someone from NVidia with more knowledge about how this is coded in the current kernel release can comment.

Anyway, Stereolabs say they already have NVidia working on this so we may just have to wait and see if NVidia comes through with a fix in a future L4T release.

Hi,

I am experiencing exactly the same problem.

I am using the ZED Camera on a Jetson TK1. I have ZED SDK 0.9.2, CUDA 6.5, opencv4tegra (2.4.10.1) installed. Have a usb 3.0 hub with Mouse, Keyboard and Camera plugged in. A screen is connected vie HDMI and an ethernet cable.

The distorted frames are only visible when setting the resolution to HD720 (also in ZED Explorer), when setting VGA, I have never seen bad frames.

Has the cause been detected in the meantime, or are there workaround suggestions?

Thanks!

I have flashed Linux for Tegra R21.4 by the way.

Hi 123er,

I’m afraid the support for Jetson TK1 from Stereolabs is pretty much gone. Whether or not the root cause is on NVidia’s end or not, we cannot be sure. Unfortunately, I was not able to get their SDK working on the Jetson at all without frame errors.

I was able to use OpenCV4Tegra to do stereo matching algorithms, however it was nowhere as convenient as using Stereolab’s SDK. Right now, there is not really a good solution using the ZED camera with the Jetson TK1, (even though it has been advertised as such!)

We are working with the Jetson TX2 and the ZED camera. We are having difficulties with the ZED because although we have all the necessary connections and codes, we can’t get any visual output. We saw on your video about the ZED that there should be a file called ZED explorer, however, we cannot find this file and we are a little confused as to what we should do about it. Do you know if there have been any changes in how the ZED runs with the TX2?

Thank you for your help, it is greatly appreciated!

Sincerely,

Spike

If you have installed the ZED software, then see if you have:

/usr/local/zed<possible version info>/tools/
/usr/local/zed<possible version info>/tools/ZED Explorer

FYI, you will of course need to be logged in locally to the Jetson to see the output. You will also need the ZED to have the latest firmware and be the sole user of one of the USB3 ports.