Stereoscopic video chat a new project idea

So I’ve received my Kinect sensor, and I already own a 3DVision capable monitor - and I think my next project will be a 3D transmission and display of a video signal. When this works in one direction it could be extended into a video chat feature.

I am not sure if I should allow the viewer to rotate the received image or change perspective. Achieving a plain stereoscopic view from the perspective of the Kinect should be enough, IMHO. And it could be done without any OpenGL hacks. Simply adding a horizontal parallax based on Z distance should work (which could be done entirely in a CUDA kernel)