Crop the input instead of scale (mmapi/Argus)

Hello, is there any way we can achieve the camera input to be cropped instead of scaled?

The current Argus lib does process the image in full resolution as it comes from the sensor, and eventually can provide a scaling, when the requested frame size is different from the native one. I am looking for an option to enforce a CROP (cut-out) instead of scaling. Ideally with zero processing overhead.

The need for a crop comes from the limitation of the NVENC, which requests the frame size to be multiple of 32. The sensor however is able to provide resolutions in multiple of 3, which makes it hard to find a large enough common denominator.

Alternatively, what is the device tree setting for a crop on input to the VI ? Looking to the sources I have a feeling it could provide cropping, but I have not found any example use case.

Thanks!

I found that the issue is rather on the encoder end - wider frames than 4096 pixel are not supported.

The only error we get is a SCF error message:

SCF: Error OverFlow: Temp BufferPool exceeds max expected size: 7 (4104x3046 BL U8_V8 420SP) (in src/services/buffermanager/BufferPool.cpp, function allocBuffer(), line 217)
SCF: Error OverFlow: Possibly due to pipeline slowdown, can cause memory bloat. (in src/services/buffermanager/BufferPool.cpp, function allocBuffer(), line 219)

But is misleading, since it can be a result of the blocked pipeline when the nvenc is not accepting such big frames.

So yes - we still need a way to CROP the ISP generated frames - without processing overhead, in order to encode at least the 4096px of data.

Try Argus::IOutputStreamSettings::setResolution() to set the output size as 3840x2160(4k)

We have the advised code already there:

iOutputStreamSettings->setResolution(Size2D<uint32_t>( CAMERA_RESX, CAMERA_RESY ));

But this infers scaling the output to the requested resolution.
Being the input 4104x3046 and the requested resolution 3840x2160, we see a clear squeeze of the 4:3 content into the 16:9 one.

To reduce processing power within the ISP, it would be better to crop on the VI input. I do not see any device-tree bindings for that, and the resolution is hard-coded in the VI driver as follows:

kernel-4.4/drivers/media/platform/tegra/camera/vi/vi4_fops.c

vi4_channel_write(chan, vnc_id, FRAME_X, width);
vi4_channel_write(chan, vnc_id, FRAME_Y, height);
vi4_channel_write(chan, vnc_id, SKIP_X, 0x0);
vi4_channel_write(chan, vnc_id, CROP_X, width);
vi4_channel_write(chan, vnc_id, OUT_X, width);
vi4_channel_write(chan, vnc_id, SKIP_Y, 0x0);
vi4_channel_write(chan, vnc_id, CROP_Y, height);
vi4_channel_write(chan, vnc_id, OUT_Y, height);

Does the crop unit actually work with the TX2 VI and we can change these registers in order to gain the crop function we need?
The TRM seems to indicate that SKIP_X is in units of 8 pixels, while SKIP_Y,CROP_X,CROP_Y is 1 pixel.

@danieel
For the Argus pipeline we don’t user VI crop but ISP.
Another way is to try Argus::IStreamSettings::getSourceClipRect() or NvVideoConverter::setCropRect()

@ShaneCCC

I have extended the DT information about the crop dimensions, changing the width/height to being the output resolution, since all your user-space code uses those numbers (e.g. to buffer allocation, etc).

But all where I have this working correctly is the RAW capture, using v4l2-ctl. It produces the VI cropped image.

When I tried to run argus_camera, there were SYNCPT errors and no image. Same with gstreamer.

With further inspection, I traced the difference down to the fact that the RAW using bypass_mode=0, and your userspace SCF requesting bypass_mode=1, which in kernel then skips the VI crop setup. And as you have again some undocumented black magic with the userspace libraries configuring the hardware, the numbers of course does not match.

So - the question is, which part of the kernel I must change, so that the input data from CSI will get trimmed by the crop sizes (I have all left/right/top/bottom defined) before entering to ISP ?

Why is the ISP not using the VI in the first place ? Or if it is optional, then can we configure the nvcamerasrc or argus to use bypass_mode=0 ? How?

The Figure 186: System Diagram with VI4 in Parker TRM on page 2774 shows that the VI block can not be bypassed, yet your kernel code uses some undocumented secret bypass mode?

I am sorry the the source of the bypass_mode=1 didn’t public due to IP policy.
It’s better to crop with the sensor REG setting.

@ShaneCCC

The advised crop with:

iStreamSettings->setSourceClipRect(Rectangle<float>( INPUT_CROP_L, INPUT_CROP_T, INPUT_CROP_R, INPUT_CROP_B));

Does not meet the performance requirements. It was clear to me, that with float coordinates, that might be a very sub-optimal solution, as there will be always some scaling involved.

A single stream works - its produced every 33.3 ms (30 fps):

0 > frame 500 @ 645152293  =  33362
0 > frame 501 @ 645185580  =  33287
0 > frame 502 @ 645218923  =  33343
0 > frame 503 @ 645252565  =  33642
0 > frame 504 @ 645285596  =  33031
0 > frame 505 @ 645318927  =  33331
0 > frame 506 @ 645352621  =  33694
0 > frame 507 @ 645385616  =  32995
0 > frame 508 @ 645419289  =  33673
0 > frame 509 @ 645452290  =  33001

But two concurrent cameras do not work - the frame to frame time is about 4 frames!

1 > frame 300 @ 563140097  =  116767
0 > frame 300 @ 563140020  =  116766
0 > frame 301 @ 563169269  =  29249
1 > frame 301 @ 563169343  =  29246
1 > frame 302 @ 563198469  =  29126
0 > frame 302 @ 563198396  =  29127
0 > frame 303 @ 563315129  =  116733
1 > frame 303 @ 563315205  =  116736
1 > frame 304 @ 563344468  =  29263
0 > frame 304 @ 563344378  =  29249
0 > frame 305 @ 563373538  =  29160
1 > frame 305 @ 563373614  =  29146
1 > frame 306 @ 563490350  =  116736
0 > frame 306 @ 563490278  =  116740
0 > frame 307 @ 563519506  =  29228
1 > frame 307 @ 563519580  =  29230
0 > frame 308 @ 563548660  =  29154
1 > frame 308 @ 563548736  =  29156
1 > frame 309 @ 563665450  =  116714
0 > frame 309 @ 563665378  =  116718

Three are naturally not working either. Now you can see why my intention was to crop it in the proper place, in the hardware unit made for that. Any post-processing on a large frame will not work, given that you have these strange performance bugs in the SCF.

Many times I have complained that the ISP stuff is crap, and you are telling me that it is not released because of IP restriction. That is untrue - you are not releasing it, because you know the low quality of that code.

So back to the original intent:
We have made the RAW image to be cropped properly. Now you tell me, how to make the hardware cropper to work with the Argus/SCF. I would really appreciate, if you can arrange access to the source code of otherwise undisclosed parts (argus_camera_daemon, libargus), and that we make our new crop_{l,r,t,b} device tree settings work across all use cases.

Use of your camera scaling partners is out of question here, we are currently fixing their work and making your hardware actually meet our production level requirements. We are not able to wait weeks and months till some thing gets fixed and several other gets broken, or as the things usually happen - that nobody will implement the requested features because there is lack of reason or millions of dollars.

We are willing to implement this ourselves, all we need is access to the source code.
Make it happen, please.

I am sorry to tell as I said due to company policy we don’t have plan to release it even scaling partner.

You are telling this:

So - how do we instruct Argus, to do INPUT CROP, which will naturally result in less load because we ask for processing only of part of the image ?

Every option mentioned above shows that the input is processed in full, and then a scaled version is produced from a part of this output.
This consumes extra processing power and extra memory bandwidth, and works only on 1 stream real-time.

Or admit, that your libraries can not support this totally primitive task and we stop looking for the holy grail.

Here’s example cropped without scaling.

/*
 * Copyright (c) 2016, NVIDIA CORPORATION. All rights reserved.
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions
 * are met:
 *  * Redistributions of source code must retain the above copyright
 *    notice, this list of conditions and the following disclaimer.
 *  * Redistributions in binary form must reproduce the above copyright
 *    notice, this list of conditions and the following disclaimer in the
 *    documentation and/or other materials provided with the distribution.
 *  * Neither the name of NVIDIA CORPORATION nor the names of its
 *    contributors may be used to endorse or promote products derived
 *    from this software without specific prior written permission.
 *
 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
 * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
 * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
 * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER OR
 * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
 * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
 * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
 * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
 * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
 * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
 * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 */

#include <stdio.h>
#include <stdlib.h>
#include <Argus/Argus.h>
#include <EGLStream/EGLStream.h>

#define EXIT_IF_NULL(val,msg)   \
        {if (!val) {printf("%s\n",msg); return EXIT_FAILURE;}}
#define EXIT_IF_NOT_OK(val,msg) \
        {if (val!=Argus::STATUS_OK) {printf("%s\n",msg); return EXIT_FAILURE;}}

/*
 * Program: oneShotCropped
 * Function: Capture a single cropped image from a camera device and write to a JPG file
 * Purpose: To demonstrate the use of the setSourceClipRect and the configuration of the
 *          stream parameters to get an unscaled image as a result.
 */

int main(int argc, char** argv)
{
    const uint64_t FIVE_SECONDS_IN_NANOSECONDS = 5000000000;
    std::vector<Argus::CameraDevice*> cameraDevices;

    /*
     * Set up Argus API Framework, identify available camera devices, and create
     * a capture session for the first available device
     */

    Argus::UniqueObj<Argus::CameraProvider> cameraProvider(Argus::CameraProvider::create());

    Argus::ICameraProvider *iCameraProvider =
        Argus::interface_cast<Argus::ICameraProvider>(cameraProvider);
    EXIT_IF_NULL(iCameraProvider, "Cannot get core camera provider interface");

    Argus::Status status = iCameraProvider->getCameraDevices(&cameraDevices);
    EXIT_IF_NOT_OK(status, "Failed to get camera devices");
    EXIT_IF_NULL(cameraDevices.size(), "No camera devices available");

    Argus::CameraDevice *cameraDevice = cameraDevices[0];
    Argus::ICameraProperties *iCameraProperties = Argus::interface_cast<Argus::ICameraProperties>(cameraDevice);
    EXIT_IF_NULL(iCameraProperties, "Failed to get ICameraProperties interface\n");

    Argus::UniqueObj<Argus::CaptureSession> captureSession(
        iCameraProvider->createCaptureSession(cameraDevice, &status));

    std::vector<Argus::SensorMode*> sensorModes;
    iCameraProperties->getAllSensorModes(&sensorModes);
    EXIT_IF_NULL(sensorModes.size(), "Failed to get sensor modes\n");

    Argus::ISensorMode *iSensorMode = Argus::interface_cast<Argus::ISensorMode>(sensorModes[0]);
    EXIT_IF_NULL(iSensorMode, "Failed to get sensor mode interface\n");

    Argus::ICaptureSession *iSession =
        Argus::interface_cast<Argus::ICaptureSession>(captureSession);
    EXIT_IF_NULL(iSession, "Cannot get Capture Session Interface");

    /*
     * Creates the stream between the Argus camera image capturing
     * sub-system (producer) and the image acquisition code (consumer).  A consumer object is
     * created from the stream to be used to request the image frame.  A successfully submitted
     * capture request activates the stream's functionality to eventually make a frame available
     * for acquisition.
     */

    /*
     * There is incorrect behavior in the processing of the setSourceClipRect() function
     * and the sizing of the buffers.  As this problem is being overcome internnaly and will be
     * released at a later date, a method to allow application development to proceed with
     * the desired result of a non-rescaled crop has been determined.
     *
     * By specifying a full sensor resolution on a seperate OutputStream, the internal
     * behavior of the image processing will be modified to NOT downscale the output
     * image before cropping, and then upscaling it afterwards.
     *
     * So all the following code segments that relate to a fullRes data object or interface
     * are specifically added to insure that no scaling is applied.  This elements are:
     *
     *    fullResOutputStreamSettings
     *    iFullResOutputStreamSettings
     *    fullResStream
     *    fullResConsumer
     *    iFullResFrameConsumer
     *
     */
    Argus::UniqueObj<Argus::OutputStreamSettings> fullResOutputStreamSettings(
        iSession->createOutputStreamSettings());

    Argus::UniqueObj<Argus::OutputStreamSettings> outputStreamSettings(
        iSession->createOutputStreamSettings());

    Argus::IOutputStreamSettings *iFullResOutputStreamSettings =
        Argus::interface_cast<Argus::IOutputStreamSettings>(fullResOutputStreamSettings);
    EXIT_IF_NULL(iFullResOutputStreamSettings, "Cannot get Full Res OutputStreamSettings Interface");
    iFullResOutputStreamSettings->setPixelFormat(Argus::PIXEL_FMT_YCbCr_420_888);

    Argus::IOutputStreamSettings *iOutputStreamSettings =
        Argus::interface_cast<Argus::IOutputStreamSettings>(outputStreamSettings);
    EXIT_IF_NULL(iOutputStreamSettings, "Cannot get OutputStreamSettings Interface");
    iOutputStreamSettings->setPixelFormat(Argus::PIXEL_FMT_YCbCr_420_888);

    // This call sets the stream resolution to the sensor resolution
    iFullResOutputStreamSettings->setResolution(Argus::Size2D<uint32_t>(
        iSensorMode->getResolution().width(), iSensorMode->getResolution().height()));

    // This call sets the stream resolution to the crop size
    iOutputStreamSettings->setResolution(Argus::Size2D<uint32_t>(
        iSensorMode->getResolution().width() / 2, iSensorMode->getResolution().height() / 2));

    Argus::UniqueObj<Argus::OutputStream> fullResStream(
        iSession->createOutputStream(fullResOutputStreamSettings.get()));

    Argus::UniqueObj<Argus::OutputStream> stream(
        iSession->createOutputStream(outputStreamSettings.get()));

    Argus::IStream *iStream = Argus::interface_cast<Argus::IStream>(stream);
    EXIT_IF_NULL(iStream, "Cannot get OutputStream Interface");

    Argus::UniqueObj<EGLStream::FrameConsumer> fullResConsumer(
        EGLStream::FrameConsumer::create(fullResStream.get()));

    EGLStream::IFrameConsumer *iFullResFrameConsumer =
        Argus::interface_cast<EGLStream::IFrameConsumer>(fullResConsumer);
    EXIT_IF_NULL(iFullResFrameConsumer, "Failed to initialize full res Consumer");

    Argus::UniqueObj<EGLStream::FrameConsumer> consumer(
        EGLStream::FrameConsumer::create(stream.get()));

    EGLStream::IFrameConsumer *iFrameConsumer =
        Argus::interface_cast<EGLStream::IFrameConsumer>(consumer);
    EXIT_IF_NULL(iFrameConsumer, "Failed to initialize Consumer");

    Argus::UniqueObj<Argus::Request> request(
        iSession->createRequest(Argus::CAPTURE_INTENT_STILL_CAPTURE));

    Argus::IRequest *iRequest = Argus::interface_cast<Argus::IRequest>(request);
    EXIT_IF_NULL(iRequest, "Failed to get capture request interface");

    Argus::IStreamSettings* iStreamSettings =
        Argus::interface_cast<Argus::IStreamSettings>(
            iRequest->getStreamSettings(stream.get()));
    EXIT_IF_NULL(iStreamSettings, "Cannot get StreamSettings Interface");

    // This calls crops the image by 50% in each dimension and centers the image
    status = iStreamSettings->setSourceClipRect(
        Argus::Rectangle<float>(0.25f, 0.25f, 0.75f, 0.75f));
    EXIT_IF_NOT_OK(status, "Failed setSourceClipRect");

    status = iRequest->enableOutputStream(fullResStream.get());
    EXIT_IF_NOT_OK(status, "Failed to enable Full Res stream in capture request");

    status = iRequest->enableOutputStream(stream.get());
    EXIT_IF_NOT_OK(status, "Failed to enable stream in capture request");

    uint32_t requestId = iSession->capture(request.get());
    EXIT_IF_NULL(requestId, "Failed to submit capture request");

    /*
     * Acquire a frame generated by the capture request, get the image from the frame
     * and create a .JPG file of the captured image
     */

    Argus::UniqueObj<EGLStream::Frame> frame(
        iFrameConsumer->acquireFrame(FIVE_SECONDS_IN_NANOSECONDS, &status));

    EGLStream::IFrame *iFrame = Argus::interface_cast<EGLStream::IFrame>(frame);
    EXIT_IF_NULL(iFrame, "Failed to get IFrame interface");

    EGLStream::Image *image = iFrame->getImage();
    EXIT_IF_NULL(image, "Failed to get Image from iFrame->getImage()");

    EGLStream::IImageJPEG *iImageJPEG = Argus::interface_cast<EGLStream::IImageJPEG>(image);
    EXIT_IF_NULL(iImageJPEG, "Failed to get ImageJPEG Interface");

    status = iImageJPEG->writeJPEG("oneShotCropped.jpg");
    EXIT_IF_NOT_OK(status, "Failed to write JPEG");

    return EXIT_SUCCESS;
}

Thank you, we will try those fullRes* hacks in the specified order and see whether the timing mentioned in my post #8 will get back to normal.

Please also add a new feature request for next Argus releases - for true INPUT crop - specified in bayer pixels within the DT specified sensor resolution, offset might be 2N, to ease the task and keep the bayer color order same.