Maybe if you were crazy enough you could use a PBO with glReadPixels to load the frame buffer into CUDA and then use the CUDA accelerated Dirac codec to make a movie out of it. I’m not sure how much of the dirac processing is done on the GPU though.
Below I attach some Python code (an extended translation of one of the CUDA SDK examples). This uses OpenGL (via my own python-ogl bindings - you can substitute PyOpenGL for that) and Qt with the QtOpenGL widget. The “screen capture” is a single of code image = self.grabFrameBuffer() within this code and the image saved as PNG. You can probably substitute your own favorite programming language for that (C/C++/Fortran). You may have to reproduce what grabFrameBuffer does, presumably, as others have pointed out glReadPixels. But the short summary answer is: yes, you can do screen capture when using CUDA, although not in CUDA itself. The example below using vertex buffer objects, to store the result of the CUDA computation. You may read this yourself with a cudaMemCopy, I guess - minus all color and lighting info. Note that if you use Qt, life is very simple and if you are not doing this commercially, you can use the GPL versions of Qt and PyQt - plus my CUDA bindings for Python which you can find at ftp://graviscom.com/pub/code/python-cuda/ or via http (graviscom.com/sources/rpms/python-cuda. CUDA and OpenGL work together very well, there are several examples in the CUDA SDK. Don*t know about DirectX, as I am not using that.