Custom Byte Tracker was not able to send Terminated Tracks info back to DeepStream

Accelerated Computing Intelligent Video Analytics DeepStream SDK
,
1

Ideally, DeepStream sends a valid pointer for the custom low level Byte Tracker to fill in Terminated Tracks info. Then DeepStream CType Mapping the custom Terminated Tracks data so the the Python app is able to get those data.

Right now the issue is DeepStream always sends a nullptr. The custom low level ByteTracker detects the nullptr, and allocates a new valid object for that pointer and populates the Terminated Tracks data to it. But DeepStream 7.1 is not able to get that newly allocated object values from the Custom code. And the data gets lost.

Tried hard to turn on “output-terminated-tracks” in DeepStream configuration. But DeepStream keeps sending a nullptr. Here is the relevant DeepStream 7.1 code: in deepstream-7.1/sources/gst-plugins/gst-nvtracker/nvtracker_proc.cpp file lines 2075-2147:

2075 NvTrackerMiscDataBuffer pMiscDataBuf = m_MiscDataMgr.pop();
2076 NvMOTTrackerMiscData miscDataResult;
2077 if (pMiscDataBuf)
2078 {
2079 miscDataResult.pPastFrameObjBatch = &pMiscDataBuf->pastFrameObjBatch;
2080
2081 miscDataResult.pTerminatedTrackBatch = m_Config.outputTerminatedTracks?
2082 &pMiscDataBuf->terminatedTrackBatch : nullptr;
2083
2084 miscDataResult.pShadowTrackBatch = m_Config.outputShadowTracks?
2085 &pMiscDataBuf->shadowTracksBatch : nullptr;
2086
2087 procResult.pReidTensorBatch = m_Config.outputReidTensor ?
2088 (&pMiscDataBuf->reidTensorBatch) : nullptr;
2089
2090 }
2091 else
2092 {
2093 miscDataResult.pPastFrameObjBatch = nullptr;
2094 procResult.pReidTensorBatch = nullptr;
2095 miscDataResult.pTerminatedTrackBatch = nullptr;
2096
2097 /* Print warning if user meta is required but unavailable. */
2098 if (m_Config.pastFrame || m_Config.outputReidTensor)
2099 {
2100 LOG_WARNING(
2101 “gstnvtracker: Unable to acquire a user meta buffer. Try increasing user-meta-pool-size\n”);
2102 }
2103 }
2104
2105 for (uint listInd = 0; listInd < batchList.size(); listInd++)
2106 {
2107 std::map<SurfaceStreamId, NvDsFrameMeta *>& frameMap = batchList.at(listInd);
2108
2109 int frameInd = 0;
2110 for (auto it = frameMap.begin(); it != frameMap.end(); it++, frameInd++)
2111 {
2112 fillMOTFrame(it->first, procParams, it->second, procInput.frameList[frameInd],
2113 procResult.list[frameInd]);
2114 }
2115 procInput.numFrames = frameMap.size();
2116 procResult.numFilled = frameMap.size();
2117 procResult.numAllocated = frameMap.size();
2118
2119 / Wait for buffer surface transform to finish. */
2120 if (m_Config.inputTensorMeta == false) {
2121 m_ConvBufMgr.syncBuffer(procParams.pConvBuf, &procParams.bufSetSyncObjs);
2122 }
2123
2124 char contextName[100];
2125 snprintf(contextName, sizeof(contextName), “%s_nvtracker_process(Batch=%u)”,
2126 m_Config.gstName, procParams.batchId);
2127 nvtx_helper_push_pop(contextName);
2128 NvMOTStatus status = m_TrackerLibProcess(m_BatchContextHandle, &procInput, &procResult);
2129 nvtx_helper_push_pop(NULL);
2130 if (NvMOTStatus_OK != status)
2131 {
2132 LOG_ERROR(“gstnvtracker: Low-level tracker lib returned error %d\n”, status);
2133 m_TrackerLibError = true;
2134 }
2135
2136 if (m_TrackerLibRetrieveMiscData){
2137 status = m_TrackerLibRetrieveMiscData(m_BatchContextHandle, &procInput, &miscDataResult);
2138 if (NvMOTStatus_OK != status)
2139 {
2140 LOG_ERROR(“gstnvtracker: When flushing previous frames, low-level tracker lib returned error %d\n”, status);
2141 m_TrackerLibError = true;
2142 }
2143 }
2144 updateBatchMeta(procResult, procParams, frameMap);
2145 }
2146 updateUserMeta(batchList, procParams, pMiscDataBuf);
2147 }

At line 2081, ideally, “output-terminated-tracks” is turned on, “miscDataResult.pTerminatedTrackBatch” and “&pMiscDataBuf->terminatedTrackBatch” should point to the same structure object.

Then line 2137 calls the low level custom Byte Tracker by passing it “miscDataResult”, and the low level code populates “miscDataResult.pTerminatedTrackBatch” with Terminated Tracks info.
Ideally that info should also be populated to “&pMiscDataBuf->terminatedTrackBatch” because DeepStream makes both the same pointer.

Finally at line 2146, DeepStream CType Mapping “pMiscDataBuf” for Python app to get the Terminated Tracks info.

What is broken here is DeepStream always sends a nullptr “miscDataResult.pTerminatedTrackBatch” at line 2137. Even the Custom low level ByteTracker detects the nullptr, and allocates a new valid pointer for “miscDataResult.pTerminatedTrackBatch” and populates the Terminated Tracks data to it, at line 2146, DeepStream CType Mapping “pMiscDataBuf” pointer doesn’t get the Terminated Tracks info. So the data was not able to be sent back to Python app. Because DeepStream is using two pointers for the same thing. The custom code can only fix the one that is passed in. But DeepStream happens to use the other one for CType Mapping.

Now I hope the Moderator can help: please indicate how to get DeepStream to send a valid pointer “miscDataResult.pTerminatedTrackBatch” at line 2137 of nvtracker_proc.cpp? This chunk of code is nearly repeated from lines 2444-2543 of the same file. So if you can show the same at line 2528, that will also do.

Thank you very much!

Jan

3
  1. According to the link below, if you use deepstream-app with nvds_nvmultiobjecttracker instead of your custom low level traker library, can you get the expected data in write_terminated_track_output( location at sources/apps/sample_apps/deepstream-app/deepstream_app.c)?

If the answer is yes, then the problem is probably caused by the custom tracker library.

2.In fact, the value of m_Config.outputTerminatedTracks should be assigned by NvMOT_Query in the custom tracker library. Have you performed the corresponding operation? then in line 2081, pTerminatedTrackBatch will not be nullptr

NvMOTStatus NvMOT_Query(uint16_t customConfigFilePathSize,
                         char* pCustomConfigFilePath,
                         NvMOTQuery *pQuery)
  1. Try changing the value of nvtracker user-meta-pool-size property to 64 or greater.
4

Thank you @junshengy !

I tried your suggestions #2 and #3. After applying suggestion #2, outputTerminatedTracks is now successfully enabled in DeepStream. This is confirmed by two observations:

  1. Previously, DeepStream passed a null pointer pTrackerMiscData->pTerminatedTrackBatch for the low level Byte tracker to return Terminated Tracks. Now, it passes a valid pointer.
  2. In Python, batch_user_meta_list used to have 0 elements. Now, it contains entries (e.g., l_user is non-null).

I am doing more testing to see if there is still any problem. After that I will update on this post here.

5

The problem is solved. Now DeepStream’s support for Terminated Tracks is turned on by @junshengy 's suggestion #2, i.e, outputTerminatedTracks should be assigned by NvMOT_Query in the custom tracker library. Note: setting outputTerminatedTracks in existing config files did not work. Setting it in low level C++ custom tracker code worked. Please close this issue. Thank you!

6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.