How to Efficiently Transfer DriveWorks Video Frames to NvMediaIEP?

Autonomous Vehicles DRIVE AGX Orin DRIVE AGX Orin General
1

Please provide the following info (tick the boxes after creating this topic):
Software Version
DRIVE OS 6.0.10.0
DRIVE OS 6.0.8.1
DRIVE OS 6.0.6
DRIVE OS 6.0.5
DRIVE OS 6.0.4 (rev. 1)
DRIVE OS 6.0.4 SDK
other

Target Operating System
Linux
QNX
other

Hardware Platform
DRIVE AGX Orin Developer Kit (940-63710-0010-300)
DRIVE AGX Orin Developer Kit (940-63710-0010-200)
DRIVE AGX Orin Developer Kit (940-63710-0010-100)
DRIVE AGX Orin Developer Kit (940-63710-0010-D00)
DRIVE AGX Orin Developer Kit (940-63710-0010-C00)
DRIVE AGX Orin Developer Kit (not sure its number)
other

SDK Manager Version
2.1.0
other

Host Machine Version
native Ubuntu Linux 20.04 Host installed with SDK Manager
native Ubuntu Linux 20.04 Host installed with DRIVE OS Docker Containers
native Ubuntu Linux 18.04 Host installed with DRIVE OS Docker Containers
other

Dear Nvidia Team,

My goal is to grab a video stream from GMSL cameras and encode them with the NvMedia IEP h265 encoder into a bitstream with the lowest possible latency. Using the DriveWorks encoder does not work for me, because I want to have more freedom in changing the encoder parameterization and also want to change the target bitrate on the fly.

I obtain the GMSL cameras frame by frame with the dwSensorCamera_getImage() function. Now, I am only able to transfer the frame-wise data into the NvSciBufObj buffers provided by the IEP application, by converting the data into CPU format and feeding the raw pointer by NvSciBufObjPutPixels() into the appBuffer->bufObj buffer that NvMediaIEPFeedFrame() is accepting. However, this introduces ~30 ms of latency due to the copy step, which seems to be avoidable.

I already tried to create a dwImage in DriveWorks by taking an appBuffer from the queue.

CHECK_DW_ERROR(dwImage_createAndBindNvSciBuf(
            &imageBlockMemory,
            &imageProperties,
            appBuffer->bufObj,
            *mContext));

But this results in the error Driveworks exception thrown: Error: ImageCuUDABlock() Failed to import external memory. Error cudaErrorInvalidValue: invalid argument. The image properties are set to

imageProperties.memoryLayout = DW_IMAGE_MEMORY_TYPE_BLOCK;
imageProperties.type = DW_IMAGE_NVMEDIA;
imageProperties.format = DW_IMAGE_FORMAT_YUV420_UINT8_SEMIPLANAR;

analogously to the properties of the IEP buffers.

Is it generally possible to share a NvSciBuf between the IEP and DriveWorks? Or are there other options to quickly parse the captured frames to the IEP encoder?

3

Dear @kristian.fischer ,
Could you list what are the things you feel missing in DW related to your case. Let me check and provide guidance accordingly. This helps us to improve the SW as well.

4

Sure!
Regarding the DW Image Encoder, the adaptation of bitrate on the fly is missing when using the DW encoder as described in the camera sample example. Besides, other features are missing that seem to be available in the NvMedia Image Encoder such as RoI coding, slicing, intra refreshing, etc. Those are also features that I like to use and why I take the effort to build the pipeline from DW to NvMedia IEP.

Regarding the pipeline from dwSensorCamera_getImage() to NvMediaIEPFeedFrame(), I would wish for a function that can transfer the received image from the camera to the format the NvMedia IEP accepts as fast as possible. If this requires a copying step, then there should be a faster way than doing this via the CPU memory. Maybe that would be possible to accelerate with CUDA?
One final, general remark regarding the usability of the nvscibuf API: In my opinion, the NvSciError_BadParameter is not enough information to see what the exact problem is, and why a certain function is not accepting the input parameters/buffers. Finding the actual problem is too cumbersome and therewith time consuming.

5

Dear @kristian.fischer,
Did you try using dwImage_getNvMedia() to get dwImageNvMedia object and try using NvSciBuf from dwImageNvMedia?
Also, note that NvMedia API uses NvEnc engine to perform encoding. So CUDA is not used.

6

Dear SivaRamaKrishnaNV,

I indeed tried to directly get the NvSciBuf from the dwImageNvMedia. The issue I got stuck with this approach was that the NvSciBuf has to be registered before the first call of NvMediaIEPFeedFrame according to the documentation. And for that, I did not find a solution on how to register the NvSciBuf buffer that is coming from the camera for NvMediaIEPFeedFrame during runtime. Besides, it might also be the case that the NvSciBuf would have to be created from a reconciled attribute list between IEP and DW, is that right?

The only solution I found so far is the one I described above to first get the CPU memory of the camera image and copy that memory into the registered buffers for IEP.

So for me, there would be too suitable options:
1.) If you can tell DriveWorks to write the incoming camera data into a dwImage that uses the registered NvSciBuf. This is what I tried with the dwImage_createAndBindNvSciBuf function mentioned above.
2.) If you can accelerate the copying between the dwImage buffer and the registered buffer (something faster than copying via CPU). To that end, I thought one might possibly use CUDA to accelerate the copying between the two NvSciBuf buffers.