How to custom tracker in deepstream

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
NVIDIA GeForce GTX 1650

• DeepStream Version
6.2

• JetPack Version (valid for Jetson only)

• TensorRT Version
8.6.1.6

• NVIDIA GPU Driver Version (valid for GPU only)
Driver Version: 525.125.06

• Issue Type( questions, new requirements, bugs)
new requirements

• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)

• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)
I want to custom my tracker to deepstream, like NanoTrack, but the doc is too simple that i cannot understand, etc: In NvMOTContext create code

 m_pLocalizer = LocalizerFactory::getInstance().makeLocalizer(config.customConfigFilePath);

where is “LocalizerFactory”? can you provide detailed documents or samples, thanks very much.

3

Those code is for understanding the API. Please implment your own tracker based on the API. Maybe your tracker don’t need LocalizerFactory. It depends on the algrithm of different tracker

4

Nanotrack

nanotrack_network
nanotrack_network1758×740 108 KB

question

I want to custom this track algo in deepstream, Now backbone and neckhead tensorrt inference has finished, with some opencv offical track code, i can run it as executable as well as shared library. but i have no idea about how to add to deepstream.

the tracker has two api:

void init(cv::Mat, cv::Rect);
cv::Rect update(cv::Mat);

idea

  • what should do in NvMOT_Query?
  • should i allocate memory on gpu resource in NvMOT_Init?
  • should i create backbone engine and neckhead engine in NvMOT_Context?
  • should i run “init” and “update” func in NvMOT_Process?

expect

expected some detailed doc or sample code for custom tracker in deepstream. thank you very much.

finally

Please forgive me for poor English.

5

We haven’t sample code. We only have this document: Gst-nvtracker — DeepStream 6.3 Release documentation

6

hello, now i finish part of custom nanotrack, but got Bus error after some frames(not fixed num frame), with gdb, the backtrack is:

 #0  0x00007fffed2b8440 in NvTrackerProc::updateFrameMeta(_NvDsFrameMeta*, _NvMOTTrackedObjList const&, NvTrackerProc::ProcParams const&) () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_tracker.so
#1  0x00007fffed2b8f67 in NvTrackerProc::processSurfaceStream(NvTrackerProc::DispatchInfo*) () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_tracker.so
#2  0x00007ffff70d279f in std::execute_native_thread_routine(void*) (__p=0x7fff580020f0) at /nv/src/gcc-target-12.1.0/libstdc++-v3/src/c++11/thread.cc:82
#3  0x00007ffff7694b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#4  0x00007ffff7726a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

the context code (for SOT) is:

NvMOTStatus NvMOTContext::processFrame(const NvMOTProcessParams *params, NvMOTTrackedObjBatch *pTrackedObjectsBatch) {
    cv::Mat  cv_frame;
    cv::Rect rect;

    float confi;
    int   label;

    // std::cout << "stream num: " << pTrackedObjectsBatch->numFilled << std::endl;
    for (uint streamIdx = 0; streamIdx < pTrackedObjectsBatch->numFilled; streamIdx++){
        NvMOTTrackedObjList   *trackedObjList = &pTrackedObjectsBatch->list[streamIdx];
        NvMOTFrame            *frame          = &params->frameList[streamIdx];

        /* get tracker per stream */
        if (NanoTrackMap.find(frame->streamID) == NanoTrackMap.end()) {
            NanoTrackMap[frame->streamID] = std::make_shared<TrackerNanoTRT>(custom_config_file);
        }
        
        /* get input frame */
        NvBufSurfaceParams *surfaceParams = frame->bufferList[0];
        cv_frame = cv::Mat(surfaceParams->height, surfaceParams->width, CV_8UC4, (char*)surfaceParams->dataPtr);
        cv::cvtColor(cv_frame, cv_frame, cv::COLOR_RGBA2BGR);

        /* get input rect */
        for (uint32_t numObjects = 0; numObjects < frame->objectsIn.numFilled; numObjects++) {
            NvMOTObjToTrack *objectToTrack = &frame->objectsIn.list[numObjects];

            float confi = objectToTrack->confidence;
            int   label = objectToTrack->classId;
            rect.x      = (int)(objectToTrack->bbox.x);
            rect.y      = (int)(objectToTrack->bbox.y);
            rect.width  = (int)(objectToTrack->bbox.width);
            rect.height = (int)(objectToTrack->bbox.height);

            /* single object track */
            break;
        }

        /* tracking */
        NanoTrackMap.at(frame->streamID)->update(cv_frame, rect);

        /* output */
        NvMOTTrackedObj *trackedObj = new NvMOTTrackedObj;

        trackedObj->confidence                     = 1;
        trackedObj->classId                        = (uint16_t)label;
        trackedObj->trackingId                     = 0;

        trackedObj->bbox.x                         = (float)rect.x;
        trackedObj->bbox.y                         = (float)rect.y;
        trackedObj->bbox.width                     = (float)rect.width;
        trackedObj->bbox.height                    = (float)rect.height;

        /* associatedObjectIn
        trackedObj->age                            = (uint32_t) sTrack.tracklet_len;
        trackedObj->associatedObjectIn             = sTrack.associatedObjectIn;
        trackedObj->associatedObjectIn->doTracking = true;
        */

        std::cout << "frame num: " << frame->frameNum << std::endl;
        // std::cout << "numFilled: " << filled << std::endl;
        trackedObjList->streamID     = frame->streamID;
        trackedObjList->frameNum     = frame->frameNum;
        trackedObjList->valid        = true;
        trackedObjList->list         = trackedObj;
        trackedObjList->numFilled    = 1;
        trackedObjList->numAllocated = 1;
    }
}

the query code is:

NvMOTStatus NvMOT_Query(uint16_t customConfigFilePathSize,
                        char *pCustomConfigFilePath,
                        NvMOTQuery *pQuery) {
    /**
     * Users can parse the low-level config file in pCustomConfigFilePath to check
     * the low-level tracker's requirements
     */
    pQuery->computeConfig = NVMOTCOMP_CPU;       // among {NVMOTCOMP_GPU, NVMOTCOMP_CPU}
    pQuery->numTransforms = 1;                   // 0 for IOU tracker, 1 for NvDCF or DeepSORT tracker as they require the video frames
    pQuery->colorFormats[0] = NVBUF_COLOR_FORMAT_RGBA; // among {NVBUF_COLOR_FORMAT_NV12, NVBUF_COLOR_FORMAT_RGBA}

    // among {NVBUF_MEM_DEFAULT, NVBUF_MEM_CUDA_DEVICE, NVBUF_MEM_CUDA_UNIFIED, NVBUF_MEM_CUDA_PINNED, ... }
#ifdef __aarch64__
    pQuery->memType = NVBUF_MEM_DEFAULT;
#else
    pQuery->memType = NVBUF_MEM_CUDA_UNIFIED;
#endif

    pQuery->batchMode              = NvMOTBatchMode_NonBatch;    // set NvMOTBatchMode_Batch if the low-level tracker supports batch processing mode. Otherwise, NvMOTBatchMode_NonBatch
    pQuery->supportPastFrame       = false;    // set true if the low-level tracker supports the past-frame data or not

    /**
     * return NvMOTStatus_Error if something is wrong
     * return NvMOTStatus_OK if everything went well
     */
    std::cout << "[NanoTrack Initialized]" << std::endl;
    return NvMOTStatus_OK;
}
7

You can get the source code of nvtracker plugin on DeepStream 6.3. So you can debug it with the source code.

root@p4station:/opt/nvidia/deepstream/deepstream-6.3# ls /opt/nvidia/deepstream/deepstream-6.3/sources/gst-plugins/gst-nvtracker/
Makefile README convbufmanager.cpp convbufmanager.h gstnvtracker.cpp gstnvtracker.h invtracker_proc.h logging.h miscdatabufmanager.cpp miscdatabufmanager.h nvtracker_proc.cpp nvtracker_proc.h
root@p4station:/opt/nvidia/deepstream/deepstream-6.3#

8

thank you, the problem has solved, you can close the issue