NVMEDIA_IMAGE_CAPTURE_INPUT_FORMAT_TYPE_RAW8 for ISP

Please provide the following info (check/uncheck the boxes after creating this topic):
Software Version
DRIVE OS Linux 5.2.6
DRIVE OS Linux 5.2.0
DRIVE OS Linux 5.2.0 and DriveWorks 3.5
NVIDIA DRIVE™ Software 10.0 (Linux)
NVIDIA DRIVE™ Software 9.0 (Linux)
other DRIVE OS version
other

Target Operating System
Linux
QNX
other

Hardware Platform
NVIDIA DRIVE™ AGX Xavier DevKit (E3550)
NVIDIA DRIVE™ AGX Pegasus DevKit (E3550)
other

SDK Manager Version
1.6.0.8170
other

Host Machine Version
native Ubuntu 18.04
other

Hi,
I’m trying to build my own SIPL sample that only uses ISP to process my single sample RAW frame in a loop. I’m able to enable the simulator mode and write a custom FillRawBuffer callback.
If I set
platformCfg.deviceBlockList[0].cameraModuleInfoList[0].sensorInfo.vcInfo.inputFormat = NVMEDIA_IMAGE_CAPTURE_INPUT_FORMAT_TYPE_RAW8;
There will be an error “Unknown input format 6” before the pipeline starts
Is there a way to test ISP for converting RAW8 type input?

Hi @leqiang.wang,

May I know calling which function outputs this error message?
How did you get the raw file? What’s your nvsipl_camera command to dump the RAW8 images to it? What’s the platform configuration used?

This error message is printed out when calling SetPlatformCfg()
The RAW file for input is a sample file comes from our camera dev team with RAW8 format
I did not run nvsipl_camera, erros are from sample code I wrote (modified from nvsipl_sample)
I used “SF3325_DPHY_x4” configuration, it comes from the result of NvSIPLQuery. I set the mask to “0x0001 0x0000 0x0000 0x0000” and loaded “/opt/nvidia/nvmedia/nit/sf3325.nito”.

Here is the main.cpp of my sample code

// Standard header files
#include <iostream>
#include <memory>
#include <thread>
#include <mutex>

// SIPL header files
#include "NvSIPLCommon.hpp"
#include "NvSIPLCamera.hpp"
#include "NvSIPLClient.hpp"
#include "NvSIPLPipelineMgr.hpp"
#include "NvSIPLQuery.hpp" // Query

// Sample application header files
#include "CUtils.hpp"
#include "CImageManager.hpp"
#include "QueueHandlers.hpp"
#include "DummyFrameLoader.hpp"

// Platfom configuration header files
#include "platform/common.hpp"
#include "platform/sf3324.hpp"
#include "platform/imx390_c.hpp"
#include "platform/ar0820_b.hpp"
#include "platform/ov2311_c.hpp"

using namespace nvsipl;

/**
 * Control information for a single queue.
 * Only one of @c imageQueueHandler and @c eventQueueHandler will be active,
 * depending on the type of queue being monitored.
 */
struct ThreadData: public FrameCompleteQueueHandler::ICallback,
                   public NotificationQueueHandler::ICallback
{
    std::string threadName;
    FrameCompleteQueueHandler imageQueueHandler;
    NotificationQueueHandler eventQueueHandler;
    std::mutex *printMutex;

    /**
     * Process events from the notification queue (if applicable).
     */
    void process(const NvSIPLPipelineNotifier::NotificationData& event)
    {
        const char *eventName = nullptr;

        const SIPLStatus status = GetEventName(event, eventName);
        if ((status != NVSIPL_STATUS_OK) || (eventName == nullptr)) {
            LOG_ERR("Failed to get event name\n");
        } else {
            printMutex->lock();
            std::cout << threadName << ": " << eventName << std::endl;
            printMutex->unlock();
        }
    }

    /**
     * Process completed images (if applicable).
     */
    void process(INvSIPLClient::INvSIPLBuffer* const & pBuffer)
    {
        INvSIPLClient::INvSIPLNvMBuffer *pNvMBuffer =
            static_cast<INvSIPLClient::INvSIPLNvMBuffer *>(pBuffer);
        if (pNvMBuffer == nullptr) {
            LOG_ERR("Invalid buffer\n");
        } else {
            INvSIPLClient::ImageMetaData metadata = pNvMBuffer->GetImageData();
            if (!metadata.frameSeqNumInfo.frameSeqNumValid) {
                LOG_ERR("Invalid frame sequence number\n");
            } else {
                printMutex->lock();
                std::cout << threadName << ": " \
                          << metadata.frameSeqNumInfo.frameSequenceNumber << std::endl;
                printMutex->unlock();
            }
            const SIPLStatus status = pNvMBuffer->Release();
            if (status != NVSIPL_STATUS_OK) {
                LOG_ERR("Buffer release failed\n");
            }
        }
    }
};

enum ThreadIndex {
    THREAD_INDEX_ICP = 0U,
    THREAD_INDEX_ISP0,
    THREAD_INDEX_ISP1,
    THREAD_INDEX_EVENT,
    THREAD_INDEX_COUNT
};


static SIPLStatus SiplMain(int argc, char *argv[])
{
    SIPLStatus status = NVSIPL_STATUS_OK;
    auto pQuery = nvsipl::INvSIPLQuery::GetInstance();
    status = pQuery->ParseDatabase();
    
    if (status != nvsipl::NVSIPL_STATUS_OK ) {
        printf("status not ok\n");
        return status;
    }

    PlatformCfg platformCfg;
    pQuery->GetPlatformCfg("SF3325_DPHY_x4", platformCfg);

    std::vector<uint32_t> vMasks {{
        0x0001, 0x0000, 0x0000, 0x0000
    }};
    pQuery->ApplyMask(platformCfg, vMasks);
    platformCfg.deviceBlockList[0].isSimulatorModeEnabled = true;


    auto& vcinfo = platformCfg.deviceBlockList[0].cameraModuleInfoList[0].sensorInfo.vcInfo;
    vcinfo.inputFormat = NVMEDIA_IMAGE_CAPTURE_INPUT_FORMAT_TYPE_RAW8;
    vcinfo.resolution.height = 1152;
    vcinfo.resolution.width = 2048;
    vcinfo.embeddedBottomLines = 0;
    vcinfo.embeddedTopLines = 0;
    std::unique_ptr<DummyFrameLoader> fileReader(new DummyFrameLoader());
    fileReader->Init("pg-2k.raw", vcinfo);

    NvSIPLPipelineConfiguration pipelineCfg = {
        .captureOutputRequested = true, 
        .isp0OutputRequested = true, 
        .isp1OutputRequested = false
    }; 
    pipelineCfg.imageGroupWriter = fileReader.get();


    NvSIPLPipelineQueues queues;
    CImageManager imageManager;
    std::vector<uint8_t> blob;
    bool defaultNitoLoaded = true;
    std::mutex threadPrintMutex;
    ThreadData threadDataStructs[THREAD_INDEX_COUNT];
    bool quit = false;
    bool skipNito = false;

    std::unique_ptr<INvSIPLCamera> siplCamera = INvSIPLCamera::GetInstance();
    CHK_PTR_AND_RETURN(siplCamera, "INvSIPLCamera::GetInstance()");

    status = siplCamera->SetPlatformCfg(&platformCfg);
    CHK_STATUS_AND_RETURN(status, "INvSIPLCamera::SetPlatformCfg()");

    status = siplCamera->SetPipelineCfg(SIPL_PIPELINE_ID, pipelineCfg, queues);
    CHK_STATUS_AND_RETURN(status, "INvSIPLCamera::SetPipelineCfg()");

    status = siplCamera->Init();
    CHK_STATUS_AND_RETURN(status, "INvSIPLCamera::Init()");

    status = imageManager.Init(siplCamera.get(), SIPL_PIPELINE_ID, pipelineCfg);
    CHK_STATUS_AND_RETURN(status, "CImageManager::Init()");

    status = LoadNitoFile("/opt/nvidia/nvmedia/nit/", "sf3325", blob, defaultNitoLoaded);
    CHK_STATUS_AND_RETURN(status, "LoadNitoFile()");
    if (defaultNitoLoaded) {
        LOG_ERR("Module-specific NITO file not found\n");
        return NVSIPL_STATUS_NOT_SUPPORTED;
    }

    status = siplCamera->RegisterAutoControlPlugin(SIPL_PIPELINE_ID, NV_PLUGIN, nullptr, blob);
    CHK_STATUS_AND_RETURN(status, "INvSIPLCamera::RegisterAutoControlPlugin()");

    for (uint32_t i = 0U; i < THREAD_INDEX_COUNT; i++) {
        threadDataStructs[i].printMutex = &threadPrintMutex;
        switch (i) {
            case THREAD_INDEX_ICP:
                threadDataStructs[i].threadName = "ICP";
                threadDataStructs[i].imageQueueHandler.Start(queues.captureCompletionQueue,
                                                             &threadDataStructs[i],
                                                             IMAGE_QUEUE_TIMEOUT_US);
                break;
            case THREAD_INDEX_ISP0:
                threadDataStructs[i].threadName = "ISP0";
                threadDataStructs[i].imageQueueHandler.Start(queues.isp0CompletionQueue,
                                                             &threadDataStructs[i],
                                                             IMAGE_QUEUE_TIMEOUT_US);
                break;
            case THREAD_INDEX_ISP1:
                threadDataStructs[i].threadName = "ISP1";
                threadDataStructs[i].imageQueueHandler.Start(queues.isp1CompletionQueue,
                                                             &threadDataStructs[i],
                                                             IMAGE_QUEUE_TIMEOUT_US);
                break;
            case THREAD_INDEX_EVENT:
                threadDataStructs[i].threadName = "Event";
                threadDataStructs[i].eventQueueHandler.Start(queues.notificationQueue,
                                                             &threadDataStructs[i],
                                                             EVENT_QUEUE_TIMEOUT_US);
                break;
            default:
                LOG_ERR("Unexpected thread index\n");
                return NVSIPL_STATUS_ERROR;
        }
    }

    status = siplCamera->Start();
    CHK_STATUS_AND_RETURN(status, "INvSIPLCamera::Start()");

    // Wait for the user's quit command
    while (!quit) {
        std::cout << "Enter 'q' to quit the application\n-\n";
        char line[INPUT_LINE_READ_SIZE];
        std::cin.getline(line, INPUT_LINE_READ_SIZE);
        if (line[0] == 'q') {
            quit = true;
        }
    }

    status = siplCamera->Stop();
    CHK_STATUS_AND_RETURN(status, "INvSIPLCamera::Stop()");

    status = siplCamera->Deinit();
    CHK_STATUS_AND_RETURN(status, "INvSIPLCamera::Deinit()");

    for (uint32_t i = 0U; i < THREAD_INDEX_COUNT; i++) {
        if (threadDataStructs[i].imageQueueHandler.IsRunning()) {
            threadDataStructs[i].imageQueueHandler.Stop();
        }
        if (threadDataStructs[i].eventQueueHandler.IsRunning()) {
            threadDataStructs[i].eventQueueHandler.Stop();
        }
    }

    std::cout << "SUCCESS!" << std::endl;
    return status;
}

int main(int argc, char *argv[])
{
    SIPLStatus status = SiplMain(argc, argv);
    return static_cast<int>(status);
}

Here is the error msg after running it:

Module_id 30 Severity 2 : src/devblk/fusa/cameramodule/common/SensorIF/CNvMSensor.cpp 189
Module_id 30 Severity 2 : Unknown input format 6
Module_id 30 Severity 2 : src/devblk/fusa/cameramodule/common/SensorIF/CNvMSensor.cpp 62
Module_id 30 Severity 2 : CNvMSensor::SetConfig failed with SIPL error 1
Module_id 30 Severity 2 : src/devblk/nonfusa/cameramodule/MAX96712cameramodule/ar0231/CNvMMAX96712_96705_AR0231.cpp 42
Module_id 30 Severity 2 : AR0231RCCB: GetAR0231RccbConfigSet failed with NvMedia error 7
Module_id 30 Severity 2 : src/devblk/nonfusa/cameramodule/MAX96712cameramodule/CNvMMAX96712CameraModule.cpp 404
Module_id 30 Severity 2 : MAX96712: Sensor SetConfig failed with SIPL error 1
Module_id 30 Severity 2 : src/devblk/common/ddi/ModuleIF/CNvMCameraModule.cpp 22
Module_id 30 Severity 2 : CNvMCameraModule::SetConfig failed with SIPL error 1
Module_id 30 Severity 2 : src/devblk/common/core/CNvMDeviceBlock.cpp 226
Module_id 30 Severity 2 : Module SetConfig failed with SIPL error 1
Module_id 30 Severity 2 : src/core/CNvMDeviceBlockMgr.cpp 93
Module_id 30 Severity 2 : SetConfig failed for device block on port 0x000002
Module_id 30 Severity 2 : src/devblk/nonfusa/devices/MAX96712DeserializerDriver/cdi_max96712.c 3623
Module_id 30 Severity 2 : MAX96712: Null handle passed to MAX96712SetDefaults
Module_id 30 Severity 2 : src/devblk/nonfusa/devices/MAX96712DeserializerDriver/CNvMMax96712.cpp 237
Module_id 30 Severity 2 : MAX96712: CDI_CONFIG_MAX96712_DISABLE_REPLICATION failed with NvMedia error 1
Module_id 30 Severity 2 : src/devblk/common/ddi/CNvMDevice.cpp 131
Module_id 30 Severity 2 : Driver failed Deinit() call 1
sample_m: /dvs/git/dirty/git-master_linux/camera/fusa/sipl/src/core/CNvMCamera.cpp: 74: SetPlatformCfg: DeviceBlock manager set config failed
nvsipl_sample: ERROR: INvSIPLCamera::SetPlatformCfg() failed, status: 1

I checked with our team. NVMEDIA_IMAGE_CAPTURE_INPUT_FORMAT_TYPE_RAW8 isn’t supported in DRIVE OS 5.2.6.

1 Like