BodyPose3DNet pose3d tensor output

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

• Hardware Platform : GTX 1660
• DeepStream Version : deepstream:7.0-samples-multiarch
• NVIDIA GPU Driver Version : 535.183.01
• Issue Type( bugs, questions )

I am working on deepstream-bodypose3d example, and I want to get the data of the “pose3d” output layer. Currently the example has demonstrated the output of “pose25d” layer. I have tried with the following addition in the code,

    
    /* Process 3d tensor */
    if (!strcmp(info->layerName, "pose3d")) {
      float *data = nullptr;
      NvDsInferDims embeddingDimensions = tensor_meta->output_layers_info[m].inferDims;
      int size = embeddingDimensions.d[0];
      data = (float *)g_malloc0(size * sizeof(float));
      cudaMemcpy (
          data,
          (float*)(tensor_meta->out_buf_ptrs_dev[m]),
          size * sizeof(float), 
          cudaMemcpyDeviceToHost
      );

      /* Init filters */
      if (g_filter_pose3d.find(obj_meta->object_id) == g_filter_pose3d.end()) { 
        const float m_oneEuroSampleRate = 30.0f;
        const float m_oneEuroDerivCutoffFreq = 1.0f;// Hz
        std::vector <OneEuroFilter> filter_vec;

        for (int j=0; j < numKeyPoints*3; j++) {
        /** TODO: Pending delete especially when object goes out of view, or ID switch
          * will cause memleak, cleanup required wrap into class
          * filter_vec.push_back(SF1eFilterCreate(30, 1.0, 0.0, 1.0));
          */
          filter_vec.push_back(OneEuroFilter(m_oneEuroSampleRate, 0.1f, 0.05, m_oneEuroDerivCutoffFreq));
          filter_vec.push_back(OneEuroFilter(m_oneEuroSampleRate, 0.1f, 0.05, m_oneEuroDerivCutoffFreq));
          filter_vec.push_back(OneEuroFilter(m_oneEuroSampleRate, 0.5f, 0.05, m_oneEuroDerivCutoffFreq));
        }
        g_filter_pose3d[obj_meta->object_id] = filter_vec;

        // Filters depth of root keypoint
        m_filterRootDepth.reset(m_oneEuroSampleRate, 0.1f, 0.05f, m_oneEuroDerivCutoffFreq);
      }
      
      /* Filter data {x, y, z} and populate keypoints */
      int batchSize_offset = 0;
      json pose;
      std::vector<OneEuroFilter> &filt_val = g_filter_pose3d[obj_meta->object_id];
      for (int i = 0; i < numKeyPoints; i++) {
        int index = batchSize_offset + i * 3;
        pose.push_back(filt_val[3*i].filter(data[index]));
        pose.push_back(filt_val[3*i+1].filter(data[index+1]));
        pose.push_back(filt_val[3*i+2].filter(data[index+2]));
      }
      stdout_log("debug", pose.dump());
      final_output_3d[std::to_string(obj_meta->object_id)].push_back(pose);
    }

but I am getting unusable values, here are some of the values I am receiving,

[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,1.0714328058227551e-41,-7.105549330345406e-15,1.436190796086505e-41,-7.105549330345406e-15,-1.0157697851067885e-16,1.8048724220503644e-42,0.0,-4.1598838360513285e-15,2.479597632622764e-41,-6.494745825267332e-16,0.0,9.31155152983365e-12,-1.156277397107455e-16,0.0,0.0,0.0,0.0,0.0009550009854137897,-1.192997598859909e-16,0.0,0.0,0.0,0.0,0.0,-3.655413300221092e-16,0.0,0.0,-3.7389056025297442e-16,0.0004924209788441658,0.0,0.0,0.06306854635477066,0.05800691619515419,0.0157929677516222,0.06530284136533737,0.06209205836057663,0.015823783352971077,0.05800691619515419,0.06458855420351028,-3.6893488147419103e+19,0.06205235794186592,0.02288777567446232,-3.6893488147419103e+19,9.80908925027372e-45,4.694349855488137e-43,-3.6893488147419103e+19,9.80908925027372e-45,0.06474225223064423,-3.6893488147419103e+19,0.012945625931024551,0.007515321020036936,-3.6893488147419103e+19,0.00106204638723284,0.07308079302310944,-3.6893488147419103e+19,0.008311464451253414,9.80908925027372e-45,-3.6893488147419103e+19,0.059373773634433746,0.06267625838518143,-3.6893488147419103e+19,0.06185149401426315,0.05936209112405777,-3.6893488147419103e+19,0.06304371356964111,0.05718471482396126,-3.6893488147419103e+19,0.06372728943824768,0.06440367549657822,-3.6893488147419103e+19,0.0611739456653595,0.06424808502197266,-3.6893488147419103e+19,0.061644963920116425,0.06403374671936035,-3.6893488147419103e+19]

The size of the tensor is 102 which is correct 1*3*34. Can anyone please help me regarding how to parse this output?

3

There might be some issue with the pose3d output layer currently. For now, we recommend using the pose25d in our demo.

4

Thanks for the clarification @yuweiw

5

Hey @yuweiw, what is this part of the code at line: 334 to 339 in deepstream_pose_estimation_app.cpp doing,

    for (int i = 0; i < XY1.rows(); i++) {
        float x = XY1(i, 0);
        float y = XY1(i, 1);
        float z = XY1(i, 2);
        XY1.row(i) << x, y, z;
    }

It seems redundant, can you please explain?
Thanks.

6

It does seem redundant. You can comment out that and try. We’ll confirm it further. Thanks

7

I see no changes in output without it.

8

Thanks. We’ll consider removing that code snippet later.