For buffers of type RTP_BUFFER_TYPE_CUDA_LINEAR, how am I supposed to allocate/fill the buffer? Do I have to do a cudaMalloc() and then fill it in device code?
Yes, I believe you must do that. You can use the Buffer class found in the simple prime example, and it will do it for you. You need CUDA linked though.
Oh, and a word of advice if you are using the Buffer class from the examples SDK: you cannot access the pointer on the host, just like with a regular CUDA pointer. So if you use the Buffer class, you should probably use some CUDA kernels to work with them, and/or use cudaMemcpy to move the pointers back to the host.