ISensorPrivateMetadata is null when ISP stage is disabled in Argus API

[Star_sea]

Hello,

Our team is trying to use the Argus API to obtain the RAW data and embedded metadata from a camera. I have followed the Argus sample code to write a loop for capturing RAW images, and the capture process itself seems fine.

Based on the obtained frame object, I would like to further extract the metadata contained in the frame (coming from the sensor). My main configuration code is as follows:

UniqueObj<OutputStreamSettings> outSettings(iSession->createOutputStreamSettings(STREAM_TYPE_EGL));
IEGLOutputStreamSettings*      iOutSet = interface_cast<IEGLOutputStreamSettings>(outSettings);
iOutSet->setPixelFormat(Argus::PIXEL_FMT_RAW16);
iOutSet->setResolution(interface_cast<ISensorMode>(modes[Sensor_mode])->getResolution());
iOutSet->setMetadataEnable(true);
iOutSet->setMode(EGL_STREAM_MODE_FIFO);
UniqueObj<OutputStream>       stream(iSession->createOutputStream(outSettings.get()));

UniqueObj<FrameConsumer>      consumer(FrameConsumer::create(stream.get()));
IFrameConsumer*               iConsumer = interface_cast<IFrameConsumer>(consumer);

UniqueObj<Request>            request(iSession->createRequest(CAPTURE_INTENT_STILL_CAPTURE));
Ext::ISensorPrivateMetadataRequest* iMetaReq =
interface_cast<Ext::ISensorPrivateMetadataRequest>(request);
IRequest* iReq = interface_cast<IRequest>(request);    
iReq->setEnableIspStage(true);
iReq->enableOutputStream(stream.get());
ISourceSettings *iSourceSettings = interface_cast<ISourceSettings>(request);
iSourceSettings->setSensorMode(modes[Sensor_mode]);
iSession->repeat(request.get());    
       
UniqueObj<Frame> frame(iConsumer->acquireFrame(TIMEOUT_NS, &status));
const EGLStream::IArgusCaptureMetadata* iArgusCaptureMetadata =
interface_cast<const EGLStream::IArgusCaptureMetadata>(frame);
if(!iArgusCaptureMetadata)
  printf("No IArgusCaptureMetadata on frame\n");
const CaptureMetadata* capMeta = iArgusCaptureMetadata->getMetadata();
if(!capMeta)
  printf("No CaptureMetadata on frame\n");
const Ext::ISensorPrivateMetadata* iMeta =
interface_cast<const Ext::ISensorPrivateMetadata>(capMeta);
if(!iMeta)
  printf("No ISensorPrivateMetadata on frame\n");

In my tests, I found something strange: if I set the capture pixel format to RAW16, i.e.,

iOutSet->setPixelFormat(Argus::PIXEL_FMT_RAW16);

and disable the ISP stage with

iReq->setEnableIspStage(false);

then the ISensorPrivateMetadata* iMeta I get is null.

However, if I enable the ISP stage with

iReq->setEnableIspStage(true);

then ISensorPrivateMetadata* iMeta is not null, and the metadata retrieved matches the actual data.

What I don’t quite understand is: since camera metadata should come from the sensor, why is it that I can only get ISensorPrivateMetadata after enabling the ISP stage?

To obtain RAW data that is as unmodified as possible, I believe it is necessary to skip the ISP stage. So, in the Argus framework, if I want to get the sensor metadata from the frame object, do I have to enable the ISP stage? Or is there something wrong with my settings?

[Star_sea]

Could someone clarify this? Is there a way to access ISensorPrivateMetadata from an Frame while the ISP stage is disabled? We rely on that metadata to tag/identify frames.

[kayccc]

To answer your question, yes, you need to enable the ISP stage to get the sensor metadata from the Argus::CaptureMetadata object. The ISP stage is responsible for generating the metadata, and disabling it will not provide the correct metadata.

Alternative Approach

If you want to obtain RAW data that is as unmodified as possible, you can use the V4L2 API instead of the Argus API. The V4L2 API allows you to capture RAW data without enabling the ISP stage. However, you will need to handle the metadata retrieval separately, as the V4L2 API does not provide a built-in mechanism for retrieving metadata.

Comparison of Argus API and V4L2 API

Here’s a comparison of the Argus API and V4L2 API:

	Argus API	V4L2 API
ISP Stage	Required for metadata retrieval	Not required
RAW Data	Modified by ISP stage	Unmodified
Metadata Retrieval	Built-in mechanism	Separate handling required

In summary, to get the sensor metadata from the Argus::CaptureMetadata object, you need to enable the ISP stage. However, if you want to obtain RAW data that is as unmodified as possible, you can use the V4L2 API, but you will need to handle metadata retrieval separately.