I want to input a mp4 footage to the sample_drivenet.

Hello,
I want to input a mp4 footage to the sample_drivenet.
I think that it can be realized by changing the following code.

/usr/local/driveworks/sample/src/drivenet/common/common.cpp

bool getNextFrameImages(dwImageCUDA** rgbCudaImageOut, dwImageGL** rgbaGLImageOut, dwCameraFrameHandle_t frameHandle)
{
    dwStatus result = DW_SUCCESS;

    // raw image
    if (gCameraOutputType == DW_CAMERA_RAW_IMAGE) {

        const dwImageDataLines* dataLines;
        dwImageCPU  *rawImageCPU;
        dwImageCUDA *rawImageCUDA;

#ifdef VIBRANTE
        dwImageNvMedia *rawImageNvMedia = nullptr;

        if (gInputType.compare("camera") == 0) {
            result = dwSensorCamera_getImageNvMedia(&rawImageNvMedia, DW_CAMERA_RAW_IMAGE, frameHandle);
            rawImageNvMedia->prop.pxlFormat = DW_IMAGE_RAW;
        }else
#endif
        {
            result = dwSensorCamera_getImageCPU(&rawImageCPU, gCameraOutputType, frameHandle);
        }

        if (result != DW_SUCCESS) {
            std::cerr << "Cannot get raw image: " << dwGetStatusName(result) << std::endl;
            return false;
        }
        dwSensorCamera_getDataLines(&dataLines, frameHandle);

        // process
#ifdef VIBRANTE
        if (gInputType.compare("camera") == 0) {
            result = dwImageStreamer_postNvMedia(rawImageNvMedia, gInput2cuda);
        }else
#endif
        {
            result = dwImageStreamer_postCPU(rawImageCPU, gInput2cuda);
        }
        if (result != DW_SUCCESS) {
            std::cerr << "Cannot post raw image: " << dwGetStatusName(result) << std::endl;
            return false;
        }

        // input image was posted, get now CUDA image out of the streamer
        dwImageStreamer_receiveCUDA(&rawImageCUDA, 10000, gInput2cuda);
        {
            // run softISP, get RCB and RGB images
            {
                dwSoftISP_bindRawInput(rawImageCUDA, gSoftISP);
                CHECK_DW_ERROR(dwSoftISP_processDeviceAsync(DW_SOFT_ISP_PROCESS_TYPE_DEMOSAIC | DW_SOFT_ISP_PROCESS_TYPE_TONEMAP,
                                                            gSoftISP));
            }

            // return used RAW image, we do not need it anymore, as we now have a copy through the RawPipeline
            dwImageStreamer_returnReceivedCUDA(rawImageCUDA, gInput2cuda);
        }

        // wait
#ifdef VIBRANTE
        if (gInputType.compare("camera") == 0) {
            dwImageStreamer_waitPostedNvMedia(&rawImageNvMedia, 10000, gInput2cuda);
        }else
#endif
        {
            dwImageStreamer_waitPostedCPU(&rawImageCPU, 10000, gInput2cuda);
        }

    }
    // h264 image
    else {
        dwImageCUDA *yuvImageCUDA = nullptr;

#ifdef VIBRANTE
        dwImageNvMedia *imageNvMedia = nullptr;
        result = dwSensorCamera_getImageNvMedia(&imageNvMedia, DW_CAMERA_PROCESSED_IMAGE, frameHandle);
        result = dwImageStreamer_postNvMedia(imageNvMedia, gInput2cuda);
        result = dwImageStreamer_receiveCUDA(&yuvImageCUDA, 60000, gInput2cuda);
#else
        result = dwSensorCamera_getImageCUDA(&yuvImageCUDA, DW_CAMERA_PROCESSED_IMAGE, frameHandle);
#endif
        if (result != DW_SUCCESS) {
            std::cerr << "Cannot get h264 image: " << dwGetStatusName(result) << std::endl;
            return false;
        }
        result = dwImageFormatConverter_copyConvertCUDA(&gRGBImage, yuvImageCUDA, gConvertYUV2RGB, g_cudaStream);
        if (result != DW_SUCCESS) {
            std::cerr << "Cannot convert to rgb: " << dwGetStatusName(result) << std::endl;
            return false;
        }

#if VIBRANTE
        dwImageStreamer_returnReceivedCUDA(yuvImageCUDA, gInput2cuda);
        dwImageNvMedia *tempNvMedia;
        result = dwImageStreamer_waitPostedNvMedia(&tempNvMedia, 33000, gInput2cuda);
#endif
    }

    // RGB -> RGBA
    {
        dwImageFormatConverter_copyConvertCUDA(&gRGBAImage, &gRGBImage, gConvertRGB2RGBA, g_cudaStream);
    }

    // get GL image
    {
        dwImageStreamer_postCUDA(&gRGBAImage, gCuda2gl);
        dwImageStreamer_receiveGL(rgbaGLImageOut, 10000, gCuda2gl);
    }

    // RCB result
    *rgbCudaImageOut = &gRGBImage;

    return true;
}

If someone has already been done, I would like you to help me.

Thanks.

also looking for a way to do this

Dear kkazama & Matthew_vsi_labs,

Could you please see DriveNet Sample part in DriveWorks doc?
According to the doc, it is not possible to use mp4 file format for drivenet input format.
And the networks NV provide have been trained with images that are RCB (red, clear, blue … no green) DriveNet (and all the networks) consume those images. Thanks.

Dear SteveNV,

Thank you for your comment.
I understand that mp4 files are not adapted to drivenet.
I simply want to convert a image that I can extract from mp4 footage to rccb format and input it in drivenet.
I don’t pay attention to the performance now.

Rather, does there exist reasons why you can not input mp4 files?

Thanks.

Dear SteveNV,

I would also like to convert a mp4/h264 file to the raw format the DriveNET uses. In some other threads you posted that you can use ffmpeg to convert to the raw format that drivenet uses… However, that doesn’t seem to work, or could you please share additional details necessary to do the proper conversion.

Please and thank you.

Dear driveworks users,

Should I give up?

Thanks.

Hi,

At least in SDK 5.0.5.0 (I haven’t checked older versions), DriveNet does support h264 as input format, you don’t need to change a single line of code for this.
Just look at the code in DriveNetApp::initSensors (in DrivNetApp.cpp), “camera-virtual” with h264 is supported for VIBRANTE.
See also this thread:
https://devtalk.nvidia.com/default/topic/999642/

Dear Frank_cisco,

Thank you for your reply!
My question has been solved.

Best regards,
kkazama.