I was wondering how many of you have actually used the Stereolabs ZED stereo USB3.0 camera with the Jetson?
They have claimed support on the NVidia Jetson architecture, however I have been experiencing corrupted frame grabs, where usually the frames grabbed look good but every so often either:
The image grabbed will have 20 or so rows of checkered green or pink color (corrupted rows)
The image grabbed will be all green or all pink (no data?)
The image grabbed will not be aligned properly
I have used both OpenCV and stereolab’s SDK, and both have the same issue. I am getting 30FPS @ 1280x480 (they stream both left/right images as one stitched image). I have also tried maxing and not-maxing the GPU/CPU clocks etc.
I have also tried Point Grey’s “Change usbfs_memory_mb = 1000” from 2, but no change. Also have tried bare L4T 21.4 and with Grinch kernel update, no difference…
The full-size USB port is by default USB2. Have you enabled USB3? Kernel parameter “usb_port_owner_info=0” would be changed to “usb_port_owner_info=2” in extlinux.conf.
USB and CPU can be set to always run at full speed…if they try to go to any kind of power conserve and back performance will sometimes choke. Have you set for performance? If not, see:
[url]http://elinux.org/Jetson/Performance[/url]
I haven’t used the usbfs_memory_mb, but looking at a kernel parameter list, 16 is the default (16MB). If using 1000, this means a full gigabyte of buffer on a system that has only 2GB to start with. Since there are two cameras, I’d test it with “usbcore.usbfs_memory_mb=32” (which is double the default and sticks to a power of 2). Are you adding this to the “APPEND” line of extlinux.conf?
I’m not surprised that a small amount of USB buffer increase would help (after all, you’re using 2 cameras instead of 1). The question remains whether the source/camera/USB is a bottleneck, or the consumer/app. At this stage you might want to give a detailed description of the build or other setup of applications or libraries using the data.
I’m not sure what exactly you mean, but I’ll give it a shot:
I am using a small c++ program using OpenCV (opencv4tegra) to loop doing frame-grabs and displaying the full image (left and right as one image, as this is how the camera streams both images) using imshow and a waitKey(1) (wait for 1 ms). Just displaying this image, I can observe the randomness of the frame corruptions.
Stereolabs provides an alpha-release of their SDK with some sample programs. Using their sample programs, I can observe the issue.
GPU/CPU set to max performance.
I have tried requesting help from Stereolabs, but no luck from them either.
Is there something else that may be more useful for you to know?
One thing I wondered is if the camera is producing too much data and something was being dropped, or if the data actually reached Jetson and parts of a frame were not consumed fast enough (USB is using isochronous mode and packetizing data like a network with the possibility of dropping packets which might be only part of an image rather than an entire frame). However, I’ve been assuming that not all frames were bad, that some frames are good…let me know if this is incorrect.
Do you have a way of testing with slowed down data output from the camera? E.G., same picture size at half the frame rate, smaller pictures at same frame rate, or same picture size/rate but one camera only? If so, how does the image quality change?
Well, I could try using a timer event to initiate frame grabs. I’ll do this today and try 30fps and 35fps to see if there is a difference. I can’t lower the resolution, as I am using the lowest already.
The camera sends one image, it is the left-right stitched image (1240 x 480) so I cannot choose only left or only right.
It ‘seems’ I may have solved the data corruption issue, and it does not seem it was related to USB transfers. It may be due to a few reasons:
Doing openCV grabs “as fast as the camera can” introduced some instability. Instead, I have the openCV grab function working off of a fixed 33.34 ms timer. This seemed to make things happier, but alone did not fix the issue.
Probably the main culprit: I was using openCV to copy image pointers instead of image data. I believe there was some data corruption occurring when the image data was used in calculations while the memory was not ‘locked’ from anything else from altering the data. I’m not 100% convinced, as my functions should have behaved ‘sequentially’ as far as I know, but changing this has seemed to work so far. I created my own form of image buffering and mutex to protect the data when being copied to my main loop. Will post if anything changes.
ALSO: I noticed in Stereolab’s newer release of the SDK (0.9), they may also be using buffer mutexes as well so maybe they are also onto the same fix since this problem exists in their 0.6 SDK release.
The kernel drivers for USB should already be capable of passing data without loss, they’re well-tested. What I’m wondering is if there is another kernel space driver involved which needs changes in locking behavior…not likely if the driver has been around a while. Can you specify what kernel drivers are being used (other than USB)?
I’m thinking there may be brief moments during user space locking in which the kernel space moves on to a new frame…not often, but enough to account for your more rare bad frames when your locking has helped eliminate most issues. In your user space code, have you considered double buffering where lock-during-copy still allows the kernel to send data to the other buffer? The idea being that the kernel would always have a buffer to write to, while user space would have lock on a buffer so it wouldn’t get overwritten before used.