I should have mentioned that I do use DirectShow for my video filter graph implementation that results in frame capture and event notification. In retrospect, I only use the Logitech API for camera control. I have two OrbitCams
with controllable tilt, pan and focus. I run both into the PC and can capture simultaneously from both, but most of the time I only used one at a time.
This may not be the best way to do things, as I usually resort to brute force, straight forward approaches, but I capture each frame of video into a texture, copy the bitmap to CUDA device, do as many kernel convolutions as needed to process that frame using a bitmap buffer output, and then I copy the bitmap output buffer back to the host, into the the DirectX texture and then display the texture on a planar surface in 3D space. I am able to to these at full resolution and full frame rate (well at least in the 60FPS range). I am using an NVIDIA Quadro FX 3700 display card and even though I have 3 Tesla C1060 cards in my machine, I so far have only used one of those at a time. So, I do have quite a bit of horsepower, but don’t rule out shuttling big bitmaps back and forth from host to device. It works very well. I do not use DirectX nor OpenGL interoperative capabilities, I just brute force everything with my own bitmap arrays.
One fun thing I have done recently is implement a true 3D version of Conway’s Life cellular automata simulation, which I run with a texture size of 256x256 and 256 levels of depth, so I display a tomographic representation of 3D objects as a cube of 16 million pixels. It’s pretty fun to watch. I have to admit I really start to bog down in frame rate on this, but still, I am shuttling 256 bitmaps per frame between host and device and back. It’s pretty fun to fly the camera through the 3D cube while it is running, or pause it and fly around in the 3D space. That is why I was saying that what you want to do with interactive, realtime 2D or 3D imagery is very possible.
That is the nice thing about programming, say in C++/C with few APIs; if you can imagine it, you can usually do it. But I’m sure you know that. It does take a lot of time to get these things to work though.
The CUDA parallel kernel architecture lends itself excellently to buffered convolution kernel video image processing. With alpha channel blending of images and depth buffered 3D (all via DirectX) , overlaying images in many ways is very easy, on the host side.
Now finding an open source framework to encapsulate all of this will be fun.
By the way, I do all of this as a hobby, not as a profession. Just for the fun and challenge.
Ken Chaffin