Serialization of Frames

Autonomous Vehicles DRIVE AGX Orin DRIVE AGX Orin General
1

Software Version
DRIVE OS 6.0.10.0
DRIVE OS 6.0.8.1
DRIVE OS 6.0.6
DRIVE OS 6.0.5
DRIVE OS 6.0.4 (rev. 1)
DRIVE OS 6.0.4 SDK
other

Target Operating System
Linux
QNX
other

Hardware Platform
DRIVE AGX Orin Developer Kit (940-63710-0010-300)
DRIVE AGX Orin Developer Kit (940-63710-0010-200)
DRIVE AGX Orin Developer Kit (940-63710-0010-100)
DRIVE AGX Orin Developer Kit (940-63710-0010-D00)
DRIVE AGX Orin Developer Kit (940-63710-0010-C00)
DRIVE AGX Orin Developer Kit (not sure its number)
other

Host Machine Version
native Ubuntu Linux 20.04 Host installed with SDK Manager
native Ubuntu Linux 20.04 Host installed with DRIVE OS Docker Containers
native Ubuntu Linux 18.04 Host installed with DRIVE OS Docker Containers
other

Issue Description
Hello guys,
I am trying to serialize Frames, that I captured with a Camera, in order to create a mp4 file. When I start the program, a mp4 file is created, but it remains empty. The code is attached down below. I hope you can help me with this problem

Thanks in advance!

Code

include “CCameraSample.h”

using namespace std;

void CCameraSample::initContext() {
// Initialize the DW Context
CHECK_DW_ERROR(dwGetVersion(&m_sdkVersion));
CHECK_DW_ERROR(dwInitialize(&m_context, m_sdkVersion, NULL));

// Print start message
cout << getCurrentTime().str() << “Initialize DriveWorks SDK v”
<< m_sdkVersion.major << “.” << m_sdkVersion.minor << “.”
<< m_sdkVersion.patch << endl << endl;
}

void CCameraSample::initCamera() {
// Initialize the Sensor Abstraction layer
CHECK_DW_ERROR(dwSAL_initialize(&m_sal, m_context));

// Specifiy the gmsl protocol for the fourth camera
m_senParams.parameters =
“camera-name=C2SIM623S2RU1195NB20,interface=csi-cd,CPHY-mode=1,”
“link=3,output-format=processed,skip-eeprom=1”;
m_senParams.protocol = “camera.gmsl”;

// Create the sensor (here: camera)
CHECK_DW_ERROR(dwSAL_createSensor(&m_camera, m_senParams, m_sal));

// Start the Sensor Abstraction Layer
CHECK_DW_ERROR(dwSAL_start(m_sal));

// Start the sensor (here: camera)
CHECK_DW_ERROR(dwSensor_start(m_camera));
}

void CCameraSample::initImageStreamer() {
// Get the CUDA Image Properties
dwImageProperties cudaProp { };
CHECK_DW_ERROR(
dwSensorCamera_getImageProperties(&cudaProp,
DW_CAMERA_OUTPUT_CUDA_RGBA_UINT8, m_camera));

// Initialize the ImageStreamer Module
CHECK_DW_ERROR(
dwImageStreamer_initialize(&m_streamerCUDA2CPU, &cudaProp,
DW_IMAGE_CPU, m_context));
}

void CCameraSample::initSerializer() {
// Specifiy the Video Parameters
dwSerializerParams serializerParams;
serializerParams.parameters =
“format=mp4,bitrate=800000,framerate=30,type=disk,”
“file-buffer-size=16777216,file=test.mp4”;

// Initialize the Serializer
CHECK_DW_ERROR(
dwSensorSerializer_initialize(&m_serializer, &serializerParams,
m_camera));

// Start the serializer
CHECK_DW_ERROR(dwSensorSerializer_start(m_serializer));
}

void CCameraSample::init() {
initContext();
initCamera();
initSerializer();
initImageStreamer();
}

CCameraSample::CCameraSample() {
m_context = DW_NULL_HANDLE;
m_camera = DW_NULL_HANDLE;
m_sal = DW_NULL_HANDLE;
m_serializer = DW_NULL_HANDLE;
m_streamerCUDA2CPU = DW_NULL_HANDLE;
m_choice = WAIT;
init();
displayMenu();
}

void CCameraSample::exportImage(dwImageHandle_t frameCUDA, int idx) {
dwImageHandle_t imageHandle;
dwImageCPU *imgCPU;
cout << getCurrentTime().str() << “ImageStreamer(CUDA → CPU)” << endl;

// stream the CUDA frame to the CPU domain
CHECK_DW_ERROR(dwImageStreamer_producerSend(frameCUDA, m_streamerCUDA2CPU));

// receive the streamed CPU image as a handle
CHECK_DW_ERROR(
dwImageStreamer_consumerReceive(&imageHandle, 0,
m_streamerCUDA2CPU));

// Converts the Image into the CPU format
CHECK_DW_ERROR(dwImage_getCPU(&imgCPU, imageHandle));

// write the image data to a png-file
char fname[128];
sprintf(fname, “GMSL_IDX_framegrab_%s.png”, std::to_string(idx).c_str());
lodepng_encode32_file(fname, imgCPU->data[0], imgCPU->prop.width,
imgCPU->prop.height);
cout << getCurrentTime().str() << "Image saved to: " << fname << “\n”;

// Return the consumed image
CHECK_DW_ERROR(
dwImageStreamer_consumerReturn(&imageHandle, m_streamerCUDA2CPU));

// Notify the producer that the work is done
CHECK_DW_ERROR(
dwImageStreamer_producerReturn(nullptr, 0,
m_streamerCUDA2CPU));
}

void CCameraSample::getFrame() {
// has to be called twice
dwCameraFrameHandle_t frameHandle = DW_NULL_HANDLE;
CHECK_DW_ERROR(dwSensorCamera_readFrame(&frameHandle, 0, m_camera));

// Create a image out of the taken Frame
dwImageHandle_t imageHandle = DW_NULL_HANDLE;
CHECK_DW_ERROR(
dwSensorCamera_getImage(&imageHandle,
DW_CAMERA_OUTPUT_CUDA_RGBA_UINT8, frameHandle));

// Passes the Image to the Image-Streamer
exportImage(imageHandle, 3);

// Releasing the handles
CHECK_DW_ERROR(dwSensorCamera_returnFrame(&frameHandle));

// needs to be called, otherwise there is always the same frame in the buffer
CHECK_DW_ERROR(dwSensor_reset(m_camera));
}

2

Dear @0b.d3,
Is your camera module listed at DRIVE AGX Orin Sensors & Accessories | NVIDIA Developer?
Is sample_camera able to record mp4 video with your camera?

4

Dear @SivaRamaKrishnaNV,
the Camera ist not officially listed, but nevertheless it has by default a plattform configuration in the nvsipl query library.

I am able to record videos with sample_camera, even the object detector sample worked fine.

5

Could you check the sample_camera sample code as reference and let us know if you have any doubts in code. You don’t want to use rig file?

6

I am using following rig-rile and it still doesn’t work, even though it works perfectly fine with sample_camera:

{
“rig”: {
“sensors”: [
{
“name”: “camera:sample0”,
“nominalSensor2Rig_FLU”: {
“roll-pitch-yaw”: [
0.0,
0.0,
0.0
],
“t”: [
1.8621,
-0.1939,
1.3165
]
},
“parameter”: “camera-name=C2SIM623S2RU1195NB20,interface=csi-cd,CPHY-mode=1,link=0,output-format=processed,async-record=1,file-buffer-size=16777216,skip-eeprom=1,fifo-size=1”,
“properties”: {
“Model”: “ftheta”,
“bw-poly”: “0.000000000000000 5.35356812179089e-4 4.99266072928606e-10 4.27370422037554e-12 -6.68245573791717e-16”,
“cx”: “1927.764404”,
“cy”: “1096.686646”,
“height”: “2168”,
“width”: “3848”
},
“protocol”: “camera.gmsl”
}
],
“vehicle”: {
“valid”: false
}
},
“version”: 2
}

7

Hi Ob.d3,

From the rig file, it looks like in the parameters section you are missing “type=file”. This function within sample_camera sets up the additional saving serialized results parameters:

    dwStatus initializeSerializer(dwSensorParams const& paramsClient, std::string fileName, uint32_t pipeLineIdx = 0)
    {
        dwSerializerParams serializerParams;
        std::string newParams(paramsClient.parameters);

        // sample_camera recorder does not support async-record
        std::string asyncRecordStr(",async-record=1");
        size_t pos = newParams.find(asyncRecordStr);

        if (pos != std::string::npos)
        {
            newParams.erase(newParams.find(asyncRecordStr), asyncRecordStr.size());
        }

        newParams += std::string(",type=disk,file=") + fileName;

        if (pipeLineIdx == dwCameraISPType::DW_CAMERA_ISP1)
        {
            newParams += ",isp-type=1";
        }

        serializerParams.parameters = newParams.c_str();
        serializerParams.onData     = nullptr;

        CHECK_DW_ERROR(dwSensorSerializer_initialize(&(m_serializer[pipeLineIdx]), &serializerParams, m_camera[FIRST_CAMERA_IDX]));
        CHECK_DW_ERROR(dwSensorSerializer_start(m_serializer[pipeLineIdx]));

        return DW_SUCCESS;
    }

However, I strongly recommend you to use “type=user” since you can customize what you want to do with the serialized results. Can you see if you can gain some inspiration from this thread? Hope this will help you with your development.

8

Hello @extern.ray.xie,
thank you for your help.

So I wrote the following method:

And as you can see, I handed over the parameters when I initialized the serializer, or should the parameters be handed over when initializing the camera?

If I use type=user, will I still be able to create a mp4. file?