We are developing VR & AR applications to operate our robots. In our framework, we pre-process our real-time image streams in the GPU memory using Nvidia Cuda and some existing Nvidia SDKs. To augment our real-time video feeds together with UE objects in the scene, we developed a direct GPU copy of video feed on 2D textures rendered in UE in Windows using D3D11. (GitHub - leggedrobotics/cuda_ue4_linux
The feature/windows branch is the working version for Windows platforms. If you request, I can also share the sample raw video dataset with you as well.)
Similarly, I would like to access the captured game scene in the GPU through a camera actor. Then I could pass it to perform further processing using CUDA & Nvidia SDKs. Such as a scene capture application into GPU memory instead of file copy: GitHub - TimmHess/UnrealImageCapture: A small tutorial repository on capturing images with semantic annotation from UnrealEngine to disk.
However, we would like to achieve this direct GPU copy operation on Linux platforms using VulkanRHI as well. Unfortunately, I couldn’t find a working example for this type of operation.
It would be great if it is possible to solve this challenge together with the Nvidia experts and its community.