Does CoarseToFineStereoDepth sample require extrinsics and Intrinsincs?

Hello,
I am trying to run the stereo_matching_depth sample but I get this error :
2020-07-30 16:13:54.752 PANIC packages/stereo_depth/CoarseToFineStereoDepth.cpp@113: Failed to create stereo matcher (error code 15)
In the isaac.stereo_depth.CoarseToFineStereoDepth Proto doc say
baseline [double] [default=0.12]: default baseline for the stereo camera (in meters) if no extrinsics provided so I thought that no extrinsics data are required but I still get the same error when I delete the extrincs edge { "source": "TXn/TX/extrinsics", "target": "depth_estimation/isaac.stereo_depth.CoarseToFineStereoDepth/extrinsics" }, so I tryed to feed extrincs data with the following cpp codlet :

namespace isaac {
void TX::start() {  tickOnMessage(rx_inLcam());}
void TX::tick() {
//Extrinsincs
  Pose3d extrinsics;
  double qq[4] = {0,0,0,0};
  extrinsics.rotation = SO3<double>::FromQuaternion(Quaterniond(qq));
  extrinsics.translation = Vector3d(0.06,0,0);
  const auto ext = tx_extrinsics().initProto();
  ToProto(extrinsics, ext);
  tx_extrinsics().publish();
  LOG_INFO("OK \n");
}}  // namespace isaac

But still the same error so I tried to modify the Intrinsincs data computed with matlab stereo calibration app with the following cpp :

#include "TX.hpp"
#include <memory>
#include <utility>
#include "engine/core/assert.hpp"
#include "engine/gems/image/conversions.hpp"
#include "engine/gems/image/utils.hpp"
#include "messages/camera.hpp"
#include "messages/math.hpp"

namespace isaac {
void TX::start() {  tickOnMessage(rx_inLcam());}
void TX::tick() {
//Extrinsincs
  Pose3d extrinsics;
  double qq[4] = {0,0,0,0};
  extrinsics.rotation = SO3<double>::FromQuaternion(Quaterniond(qq));
  extrinsics.translation = Vector3d(0.06,0,0);
  const auto ext = tx_extrinsics().initProto();
  ToProto(extrinsics, ext);
  tx_extrinsics().publish();

//Intrinsincs
  auto out = tx_outLcam().initProto();
  // Pinhole
  auto pinhole = out.initPinhole();
  ToProto(Vector2d(3147.54098361, 3160.49382716), pinhole.getFocal());//1280*12/4.86 & 960*12/3.66  Foc len [8.525929042561033e+02,8.549701455135702e+02]
  ToProto(Vector2d(519.4719202793973, 693.0736876181506), pinhole.getCenter());//principal point [6.930736876181506e+02,5.194719202793973e+02]
  pinhole.setCols(1280);
  pinhole.setRows(960);
  // Distortion
  auto distortion = out.initDistortion();
  distortion.setModel(DistortionProto::DistortionModel::BROWN);
  Vector5d disto;
  disto << 0.035, 0.035, 0, 0, 0;
  ToProto(disto, distortion.getCoefficients());
  //Get Cam
  auto Lin= rx_inLcam().getProto();
  auto Rin = rx_inRcam().getProto();
  //auto Lout = tx_outLcam().initProto();
  //auto Rout = tx_outRcam().initProto();
  //Lout.setColorSpace(ColorCameraProto::ColorSpace::RGB);
  //Rout.setColorSpace(ColorCameraProto::ColorSpace::RGB);

  //Publish CamProto
  ToProto(out,Lin);
  ToProto(out,Rin);
  tx_inLcam().publish();
  tx_inRcam().publish();

  LOG_INFO("OK \n");
}}  // namespace isaac

I don’t know how to modify the Intrinsincs data of the camera proto but I tried and the code don’t compile because of the wrong proto handeling, I could probably make it work with some more work but I don’t know if it is really necessary. If the error come from something else I don’t want to get into that now because my camera have a 12mm lens with no distortion.
I did this code by looking at the zed.cpp code and the opencv tuto, I modified the opencv tuto to make a rgb to grey script
So do I need to make a the cpp code to send extrinsics and Intrinsincs data ? If I need to do it, could you help me a bit with my code please ?

Thank you !
regards
Planktos

Hi, you can check that the custom codelet ticks with the same frequency as the CoarseToFineStereoDepth using the Sight Statistics widget.

1 Like

Thanks but indeed my code don’t run so frequency is 0, I will create a new post with my issue in my code because I suppose that CoarseToFineStereoDepth sample need a custom Intrinsincs