Access embedded Metadata (EBD) for MIPI CSI2 using LibArgus on TX2

Hi,

I’m trying to get the (EBD) data that is produced by a sensor using Lib Argus , a couple of posts pointed that it should be possible:
*https://devtalk.nvidia.com/default/topic/1030163/embedded-metadata-ebd-for-mipi-csi2-on-tx1/#5242706
*https://devtalk.nvidia.com/default/topic/1026113/jetson-tx1/obtain-information-from-raw-buffer-when-using-nvcamerasrc/post/5219504/

I’m quite sure I got the configuration on the DTB right by setting the variable on the mode with the amount of bytes that that will be generated:

embedded_metadata_height = "1";

So my current block is accessing that data using LibArgus; I actually haven’t been able to find a explicit example on how to get to the specific interfaces. Using argus_camera I have been able to check that the extension to get that data SensorPrivateMetadata is supported (this information was provided by the stdout logs of the argus_camera application):

Argus extensions:
  BayerSharpnessMap: supported
  DebugCaptureSession: supported
  DeFog: supported
  FaceDetect: supported
  InternalFrameCount: supported
  SensorPrivateMetadata: supported

According to the documentation there are a couple of objects and interfaces available for either gathering information about the EBD support or accessing the actual data. According to my understanding :

a -ISensorPrivateMetadataCaps : Interface used to query the availability and size in bytes of sensor private metadata. Should be accessible from the CameraDevice.

b -ISensorPrivateMetadata: Interface used to access sensor private metadata. Should be accesible from the container for metadata generated by a single completed capture; the object CaptureMetadata generated after a IEventCaptureComplete event.

c -SensorPrivateMetadata: This data is metadata that the sensor embeds inside the frame, the type and formating of which depends on the sensor. It is up to the user to correctly parse the data based on the specifics of the sensor used. Should be accessible from the CameraDevice.

Given this information ; I tried the following;

1- I tried to get the ISensorPrivateMetadataCaps from cameraDevice:

std::vector<CameraDevice*> cameraDevices;
    EXIT_IF_NOT_OK(iCameraProvider->getCameraDevices(&cameraDevices),
        "Failed to get camera devices");
    EXIT_IF_NULL(cameraDevices.size(), "No camera devices available");
    if (cameraDevices.size() <= options.cameraIndex)
    {
        printf("Camera device specified on the command line is not available\n"); //I don't see this error, execution continues properly
        return EXIT_FAILURE;
    }
    
    /*Sensor metadata Caps*/
	const Ext::ISensorPrivateMetadataCaps *ISensorMetadataCap = 
	 interface_cast<const Ext::ISensorPrivateMetadataCaps>(cameraDevices[options.cameraIndex]);
     if(!ISensorMetadataCap)
     {
			printf( "Failed to get ISensorPrivateMetadata Caps Interface\n"); //I get this message , which means the returned object is NULL
	 }

2- I tried to get the ISensorPrivateMetadata from CaptureMetadata after a captured event

const Event* event = iQueue->getEvent(iQueue->getSize() - 1);
        const IEventCaptureComplete *iEventCaptureComplete
            = interface_cast<const IEventCaptureComplete>(event);
        EXIT_IF_NULL(iEventCaptureComplete, "Failed to get EventCaptureComplete Interface");  //I don't see this error, execution continues properly
        const CaptureMetadata *metaData = iEventCaptureComplete->getMetadata();
        const ICaptureMetadata* iMetadata = interface_cast<const ICaptureMetadata>(metaData);
        EXIT_IF_NULL(iMetadata, "Failed to get CaptureMetadata Interface"); //I don't see this error, execution continues properly

        /*Sensor metadata Interface*/
        const Ext::ISensorPrivateMetadata *ISensorMetadata = 
        interface_cast<const Ext::ISensorPrivateMetadata>(metaData);
        if(!ISensorMetadata)
        {
			printf( "Failed to get ISensorPrivateMetadata Interface\n");//I get this message , which means the returned object is NULL
        }

No luck so far , on both cases trying to access the Metadata interfaces return a NULL pointer. Any hints , recommendations or pointers to an example on how to get the actual metadata will be gladly appreciated.

Best Regards,
JJ

Hi,

I’m still unable to get the EBD metadata using LibArgus, any recomendations?

Regards,
JJ

Please check the argus API Ext::SensorPrivateMetadata from the l4t-document

Ext::SensorPrivateMetadata
Extensions
Detailed Description
Adds accessors for sensor embedded metadata.

This data is metadata that the sensor embeds inside the frame, the type and formating of which depends on the sensor. It is up to the user to correctly parse the data based on the specifics of the sensor used.

Ext::ISensorPrivateMetadataCaps: Determines whether a device is capable of private metadata output.
Ext::ISensorPrivateMetadataRequest: Enables private metadata output from a capture request.
Ext::ISensorPrivateMetadata: Accesses the sensor private metadata.
Data Structures
class Argus::Ext::ISensorPrivateMetadataCaps
Interface used to query the availability and size in bytes of sensor private metadata. More…

class Argus::Ext::ISensorPrivateMetadataRequest
Interface used enable the output of sensor private metadata for a request. More…

class Argus::Ext::ISensorPrivateMetadata
Interface used to access sensor private metadata. More…

Hi ShaneCCC,
Thanks for your response, I checked that documentation but according to my understanding from there:

  1. ISensorPrivateMetadataCaps should be a interface implemented on cameraDevice Objects
  2. ISensorPrivateMetadata should be a interface implemented on CaptureMetadata Objects

However, trying to get the interface from those objects results in a NULL pointers. Could you clarify how should be the process to get:

  • SensorPrivateMetadata object
  • ISensorPrivateMetadataCaps interface
  • ISensorPrivateMetadata interface

Thanks again!

Regards,
JJ

@josejich
Please enable the sensor metadata by setMetadataEnable(true) before capture.

Ext::SensorPrivateMetadata

Public Member Functions
virtual void 	setMetadataEnable (bool enable)=0
 	Enables the sensor private metadata, will only work if the sensor supports embedded metadata.

Hi ShaneCCC,

Thanks for the help, I’m still blocked since it’s not clear from the documentation were you should be able to get the Ext::SensorPrivateMetadata from, Could you clarify how should be the process to get:

  • SensorPrivateMetadata object
  • ISensorPrivateMetadataCaps interface
  • ISensorPrivateMetadata interface
  • You need to enable the sensor private metadata by setMetadataEnable() before the request. Like the bayerAveragemap enable it by setBayerAverageMapEnalbe(true) for the request.

    This patch is modify the userAutoWhiteBalance to get sensor private data that worked for me on 28.2 with imx274/imx185.

    public/samples/userAutoWhiteBalance/main.cpp | 29 ++++++++++++++++++++++++++++
     1 file changed, 29 insertions(+)
    
    diff --git a/public/samples/userAutoWhiteBalance/main.cpp b/public/samples/userAutoWhiteBalance/main.cpp
    index 76ac7c6..ba5df7e 100644
    --- a/public/samples/userAutoWhiteBalance/main.cpp
    +++ b/public/samples/userAutoWhiteBalance/main.cpp
    @@ -31,6 +31,7 @@
     #include <stdlib.h>
     #include <Argus/Argus.h>
     #include <Argus/Ext/BayerAverageMap.h>
    +#include <Argus/Ext/SensorPrivateMetadata.h>
     #include <EGLStream/EGLStream.h>
     #include "PreviewConsumer.h"
     #include "Options.h"
    @@ -98,6 +99,11 @@ static bool execute(const ExecuteOptions& options)
             return false;
         }
     
    +    /* Make sure sensor private metadata is supported by the device. */
    +    const Ext::ISensorPrivateMetadataCaps *iSensorMetadataCaps =
    +        interface_cast<const Ext::ISensorPrivateMetadataCaps>(cameraDevices[options.cameraIndex]);
    +    EXIT_IF_NULL(iSensorMetadataCaps, "Failed to get ISensorPrivateMetadataCaps Interface");
    +
         UniqueObj<CaptureSession> captureSession(
             iCameraProvider->createCaptureSession(cameraDevices[options.cameraIndex]));
     
    @@ -164,6 +170,12 @@ static bool execute(const ExecuteOptions& options)
             iBayerAverageMapSettings->setBayerAverageMapEnable(true);
         }
     
    +    /* Enable sensor private metadata output in the request. */
    +    Ext::ISensorPrivateMetadataRequest *iSensorPrivateMetadataRequest =
    +        interface_cast<Ext::ISensorPrivateMetadataRequest>(request);
    +    EXIT_IF_NULL(iSensorPrivateMetadataRequest, "Failed to get ISensorPrivateMetadata interface");
    +    iSensorPrivateMetadataRequest->setMetadataEnable(true);
    +
         EXIT_IF_NOT_OK(iSession->repeat(request.get()), "Unable to submit repeat() request");
     
         /*
    @@ -196,6 +208,23 @@ static bool execute(const ExecuteOptions& options)
             const ICaptureMetadata* iMetadata = interface_cast<const ICaptureMetadata>(metaData);
             EXIT_IF_NULL(iMetadata, "Failed to get CaptureMetadata Interface");
     
    +        /* Get and dump some of the sensor private metadata. */
    +        const Ext::ISensorPrivateMetadata *iSensorMetadata =
    +            interface_cast<const Ext::ISensorPrivateMetadata>(metaData);
    +        EXIT_IF_NULL(iSensorMetadata, "Failed to get ISensorPrivateMetadata Interface");
    +        size_t sensorMetadataSize = iSensorMetadata->getMetadataSize();
    +        char* sensorMetadata = new char[sensorMetadataSize];
    +        EXIT_IF_NULL(sensorMetadata, "Failed to allocate metadata buffer");
    +        EXIT_IF_NOT_OK(iSensorMetadata->getMetadata(sensorMetadata, sensorMetadataSize),
    +                "Failed to get sensor private metadata");
    +        printf("Sensor private metadata (first 16 of %lu bytes): ", sensorMetadataSize);
    +        for (size_t byte = 0; byte < 16 && byte < sensorMetadataSize; byte++)
    +        {
    +            printf("%02x ", sensorMetadata[byte]);
    +        }
    +        printf("\n");
    +        delete[] sensorMetadata;
    +
             BayerTuple<float> bayerTotals(0.0f);
             BayerTuple<float> bayerAverages(0.0f);
             if (!options.useAverageMap)
    -- 
    2.1.4
    
    1 Like

    Hi ShanneCC,

    Thanks for the patch, I was finally able to get the metadata from the sensor using your patch. The sensor that I’m using has embedded metadata lines at the beginning and at the end of the frame as showed on the picture :

    Will the the metadata capture both embedded metadata lines at the beginning or the end of the frames, just one of them?

    Regards

    Current only support the metadata lines at the beginning.

    Hi ShaneCCC,

    Thanks for your response. Is there any way we can configure the frame to include metadata line to read it from video4linux. I mean can we use the bypass mode and read the metadata from the bottom side of the frame, if so, how do we configure it?

    I sorry to tell it not support for current release. Due to need to modify the vi firmware.

    Hi Shane,

    What if we “not” define the embedded data lines in the DTB and increase the frame height to include the embedded meta data, would the VI pass the frame + embedded data to V4L2?

    Also, could you explain how the “bypass mode” works?

    If we need to modify the VI driver so we can get the embedded data, could you point the starting point.

    Thx!
    -clint

    No matter the witch mode doesn’t support rear embedded data in current design.

    but can we read the embedded meta in front of the frame we use the bypass mode? what do we need to do this?

    Please have reference to below topics.

    https://devtalk.nvidia.com/default/topic/1026581
    https://devtalk.nvidia.com/default/topic/1031140
    https://devtalk.nvidia.com/default/topic/1030163