No detect when use nvinferserver with yolov5s

czjasasas · August 30, 2023, 3:17pm

• Hardware Platform GeForce RTX 2080 Ti
• DeepStream Version 6.2
• NVIDIA GPU Driver Version 470.63.01
• CUDA Version 10.2
• Issue Type questions
I tried to use nvinferserver with yolov5s to detect my video in Docker Image(DS6.2 with CUDA11.8).
Its not working when i use the libnvdsinfer_custom_impl_Yolo.so generated by this DS6.2 image，so i generated it by another Image(DS6.0 with CUDA11.4), it works but have no result.I still run it in DS6.2 image container.
libnvdsinfer_custom_impl_Yolo.so can be generated by this pro: GitHub - marcoslucianops/DeepStream-Yolo: NVIDIA DeepStream SDK 6.3 / 6.2 / 6.1.1 / 6.1 / 6.0.1 / 6.0 / 5.1 implementation for YOLO models

Blew are my configuration files:
1.config.txt

source-list:
  list: file:///opt/nvidia/deepstream/deepstream-6.2/sources/apps/sample_apps/meeting_demo/test.mp4;

streammux:
  batch-size: 1
  batched-push-timeout: 40000
  width: 1920
  height: 1080

## Inference using nvinferserver:
primary-gie:
  plugin-type: 1
  config-file-path: pgie_config.txt

2.pgie_config.txt

infer_config {
  unique_id: 1
  gpu_ids: [0]
  max_batch_size: 1
  backend {
    inputs: [ {
      name: "input"
    }]
    outputs: [
      {name: "boxes"},
      {name: "scores"},
      {name: "classes"}
    ]
    triton {
      model_name: "Meeting_Person"
      version: -1
      model_repo {
        root: "../../../../samples/triton_model_repo"
        strict_model_config: false
      }
    }
  }

  preprocess {
    network_format: IMAGE_FORMAT_RGB
    tensor_order: TENSOR_ORDER_LINEAR
    tensor_name: "input"
    maintain_aspect_ratio: 1
    normalize {
      scale_factor: 0.0039215697906911373
      channel_offsets: [0,0,0]
    }
  }

  postprocess {
    labelfile_path: "./labels.txt"
    detection {
      num_detected_classes: 1
      custom_parse_bbox_func: "NvDsInferParseYolo"
      nms {
        confidence_threshold: 0.5
        iou_threshold: 0.3
        topk : 200
      }
    }
  }
  custom_lib {
    path: "./libnvdsinfer_custom_impl_Yolo.so"
  }

  extra {
    copy_input_to_host_buffers: false
    output_buffer_pool_size: 2
  }
}
input_control {
  process_mode: PROCESS_MODE_FULL_FRAME
  interval: 0
}

czjasasas · August 30, 2023, 3:41pm

Here is the model.onnx(yolov5s)’s input and output:

fanzh · August 31, 2023, 8:05am

are you testing on DS6.2 with CUDA11.8? what do you mean about “not working”? it can’t run or no detection box? could you share a whole running log?
please set symmetric_padding in configuration. can you add log in NvDsInferParseYolo to check if parsing function was triggered?
here are some yolov5 nvinferserver samples, sample1, sample2.

czjasasas · August 31, 2023, 9:16am

Thank you for your reply, here are my answers:
1.(1)yes, im testing on DS6.2 with CUDA11.8;
(2)I can not use the libnvdsinfer_custom_impl_Yolo.so generated by DS6.2 with CUDA11.8, it will run with error, maybe because my driver version is too low. So i use the libnvdsinfer_custom_impl_Yolo.so generated by DS6.0 with CUDA11.4.
(3)It runs with wrong output, part 2 will show,.
(4)The running log with no error, i didint save logs and i have debuged many times, i can show new logs in part 2.
2.(1)Tried with symmetric_padding, i set 1 but no difference,.
(2)Here is the NvDsInferParseYolo.cpp

/*
 * Copyright (c) 2019, NVIDIA CORPORATION. All rights reserved.
 *
 * Permission is hereby granted, free of charge, to any person obtaining a
 * copy of this software and associated documentation files (the "Software"),
 * to deal in the Software without restriction, including without limitation
 * the rights to use, copy, modify, merge, publish, distribute, sublicense,
 * and/or sell copies of the Software, and to permit persons to whom the
 * Software is furnished to do so, subject to the following conditions:
 *
 * The above copyright notice and this permission notice shall be included in
 * all copies or substantial portions of the Software.
 *
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
 * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
 * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
 * DEALINGS IN THE SOFTWARE.
 *
 * Edited by Marcos Luciano
 * https://www.github.com/marcoslucianops
 */

#include "nvdsinfer_custom_impl.h"

#include "utils.h"

extern "C" bool
NvDsInferParseYolo(std::vector<NvDsInferLayerInfo> const& outputLayersInfo, NvDsInferNetworkInfo const& networkInfo,
    NvDsInferParseDetectionParams const& detectionParams, std::vector<NvDsInferParseObjectInfo>& objectList);

extern "C" bool
NvDsInferParseYoloE(std::vector<NvDsInferLayerInfo> const& outputLayersInfo, NvDsInferNetworkInfo const& networkInfo,
    NvDsInferParseDetectionParams const& detectionParams, std::vector<NvDsInferParseObjectInfo>& objectList);

static NvDsInferParseObjectInfo
convertBBox(const float& bx1, const float& by1, const float& bx2, const float& by2, const uint& netW, const uint& netH)
{
  NvDsInferParseObjectInfo b;

  float x1 = bx1;
  float y1 = by1;
  float x2 = bx2;
  float y2 = by2;

  x1 = clamp(x1, 0, netW);
  y1 = clamp(y1, 0, netH);
  x2 = clamp(x2, 0, netW);
  y2 = clamp(y2, 0, netH);

  b.left = x1;
  b.width = clamp(x2 - x1, 0, netW);
  b.top = y1;
  b.height = clamp(y2 - y1, 0, netH);

  return b;
}

static void
addBBoxProposal(const float bx1, const float by1, const float bx2, const float by2, const uint& netW, const uint& netH,
    const int maxIndex, const float maxProb, std::vector<NvDsInferParseObjectInfo>& binfo)
{
  NvDsInferParseObjectInfo bbi = convertBBox(bx1, by1, bx2, by2, netW, netH);

  if (bbi.width < 1 || bbi.height < 1)
      return;

  bbi.detectionConfidence = maxProb;
  bbi.classId = maxIndex;
  binfo.push_back(bbi);
}

static std::vector<NvDsInferParseObjectInfo>
decodeTensorYolo(const float* boxes, const float* scores, const float* classes, const uint& outputSize, const uint& netW,
    const uint& netH, const std::vector<float>& preclusterThreshold)
{
  std::vector<NvDsInferParseObjectInfo> binfo;

  for (uint b = 0; b < outputSize; ++b) {
    float maxProb = scores[b];
    int maxIndex = (int) classes[b];
    
    std::cout << "Prop " << maxProb << std::endl;
    std::cout << "Class " << maxIndex << std::endl;
    std::cout << "Threshold " << preclusterThreshold[maxIndex] << std::endl;
    //if (maxProb < preclusterThreshold[maxIndex])
    //  continue;

    float bxc = boxes[b * 4 + 0];
    float byc = boxes[b * 4 + 1];
    float bw = boxes[b * 4 + 2];
    float bh = boxes[b * 4 + 3];

    float bx1 = bxc - bw / 2;
    float by1 = byc - bh / 2;
    float bx2 = bx1 + bw;
    float by2 = by1 + bh;
    
    std::cout << "Box " << "[" << bx1 << " " << by1 << " " << bx2 << " " << by2 << "]" <<std::endl;

    addBBoxProposal(bx1, by1, bx2, by2, netW, netH, maxIndex, maxProb, binfo);
  }

  return binfo;
}

static std::vector<NvDsInferParseObjectInfo>
decodeTensorYoloE(const float* boxes, const float* scores, const float* classes, const uint& outputSize, const uint& netW,
    const uint& netH, const std::vector<float>& preclusterThreshold)
{
  std::vector<NvDsInferParseObjectInfo> binfo;

  for (uint b = 0; b < outputSize; ++b) {
    float maxProb = scores[b];
    int maxIndex = (int) classes[b];
    
    if (maxProb < preclusterThreshold[maxIndex])
      continue;

    float bx1 = boxes[b * 4 + 0];
    float by1 = boxes[b * 4 + 1];
    float bx2 = boxes[b * 4 + 2];
    float by2 = boxes[b * 4 + 3];

    addBBoxProposal(bx1, by1, bx2, by2, netW, netH, maxIndex, maxProb, binfo);
  }

  return binfo;
}

static bool
NvDsInferParseCustomYolo(std::vector<NvDsInferLayerInfo> const& outputLayersInfo, NvDsInferNetworkInfo const& networkInfo,
    NvDsInferParseDetectionParams const& detectionParams, std::vector<NvDsInferParseObjectInfo>& objectList)
{
  if (outputLayersInfo.empty()) {
    std::cerr << "ERROR: Could not find output layer in bbox parsing" << std::endl;
    return false;
  }

  std::vector<NvDsInferParseObjectInfo> objects;

  const NvDsInferLayerInfo& boxes = outputLayersInfo[0];
  const NvDsInferLayerInfo& scores = outputLayersInfo[1];
  const NvDsInferLayerInfo& classes = outputLayersInfo[2];

  const uint outputSize = boxes.inferDims.d[0];
  //std::cout << "outputSize " << outputSize << "\n";

  std::vector<NvDsInferParseObjectInfo> outObjs = decodeTensorYolo((const float*) (boxes.buffer),
      (const float*) (scores.buffer), (const float*) (classes.buffer), outputSize, networkInfo.width, networkInfo.height,
      detectionParams.perClassPreclusterThreshold);

  objects.insert(objects.end(), outObjs.begin(), outObjs.end());

  objectList = objects;

  return true;
}

static bool
NvDsInferParseCustomYoloE(std::vector<NvDsInferLayerInfo> const& outputLayersInfo, NvDsInferNetworkInfo const& networkInfo,
    NvDsInferParseDetectionParams const& detectionParams, std::vector<NvDsInferParseObjectInfo>& objectList)
{
  if (outputLayersInfo.empty()) {
    std::cerr << "ERROR: Could not find output layer in bbox parsing" << std::endl;
    return false;
  }

  std::vector<NvDsInferParseObjectInfo> objects;

  const NvDsInferLayerInfo& boxes = outputLayersInfo[0];
  const NvDsInferLayerInfo& scores = outputLayersInfo[1];
  const NvDsInferLayerInfo& classes = outputLayersInfo[2];

  const uint outputSize = boxes.inferDims.d[0];

  std::vector<NvDsInferParseObjectInfo> outObjs = decodeTensorYoloE((const float*) (boxes.buffer),
      (const float*) (scores.buffer), (const float*) (classes.buffer), outputSize, networkInfo.width, networkInfo.height,
      detectionParams.perClassPreclusterThreshold);

  objects.insert(objects.end(), outObjs.begin(), outObjs.end());

  objectList = objects;

  return true;
}

extern "C" bool
NvDsInferParseYolo(std::vector<NvDsInferLayerInfo> const& outputLayersInfo, NvDsInferNetworkInfo const& networkInfo,
    NvDsInferParseDetectionParams const& detectionParams, std::vector<NvDsInferParseObjectInfo>& objectList)
{
  return NvDsInferParseCustomYolo(outputLayersInfo, networkInfo, detectionParams, objectList);
}

extern "C" bool
NvDsInferParseYoloE(std::vector<NvDsInferLayerInfo> const& outputLayersInfo, NvDsInferNetworkInfo const& networkInfo,
    NvDsInferParseDetectionParams const& detectionParams, std::vector<NvDsInferParseObjectInfo>& objectList)
{
  return NvDsInferParseCustomYoloE(outputLayersInfo, networkInfo, detectionParams, objectList);
}

CHECK_CUSTOM_PARSE_FUNC_PROTOTYPE(NvDsInferParseYolo);

In last cpp, i changed code in function decodeTensorYolo like this :

std::cout << "Prop " << maxProb << std::endl;
    std::cout << "Class " << maxIndex << std::endl;
    std::cout << "Threshold " << preclusterThreshold[maxIndex] << std::endl;
    //if (maxProb < preclusterThreshold[maxIndex])
    //  continue;

The log is blew:

Now playing : file:///opt/nvidia/deepstream/deepstream-6.2/sources/apps/sample_apps/meeting_demo/test.mp4
In create_source_bin
index : 0
WARNING: Overriding infer-config batch-size (0) with number of sources (1)
Added elements to bin
Using file: dsmeeting_config.yml
INFO: infer_trtis_backend.cpp:218 TrtISBackend id:1 initialized model: Meeting_Person
Decodebin child added: source

(meeting_demo:45442): GLib-GObject-WARNING **: 09:07:51.029: g_object_set_is_valid_property: object class 'GstFileSrc' has no property named 'drop-on-latency'
Decodebin child added: decodebin0
Running...
Decodebin child added: qtdemux0
Decodebin child added: multiqueue0
Decodebin child added: h264parse0
Decodebin child added: capsfilter0
Decodebin child added: nvv4l2decoder0
In cb_newpad
Prop 0
Class 0
Threshold 0.5
Box [-3.68488 -6.09062 9.25938 16.9058]
Prop 0
Class 0
Threshold 0.5
Box [0.257194 -3.61133 20.3475 15.6011]
Prop 0
Class 0
Threshold 0.5
Box [7.60119 -4.33493 31.6653 16.812]
Prop 0
Class 0
Threshold 0.5
Box [13.8528 -4.36356 42.1008 17.4075]
Prop 0
Class 0
Threshold 0.5
Box [19.4193 -3.64592 51.9924 17.6033]
Prop 0
Class 0
Threshold 0.5
Box [26.2744 -3.11224 58.982 16.0277]
....and so on

I found the score outputs are all 0! I dont konw why.
3.Thank you for your samples, im learning from them.

fanzh · September 3, 2023, 8:56am

please compile libnvdsinfer_custom_impl_Yolo on DS6.2 with CUDA11.8.

Sorry for the late reply, Is this still an DeepStream issue to support? Thanks. here is a yolov7 nvinferserver sample.

system · September 17, 2023, 8:56am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.