Some general questions regarding PathNet

[kn10]

Hardware Platform: DRIVE AGX Pegasus™ Developer Kit
Software Version: DRIVE Software 10
Host Machine Version: Ubuntu 18.04
SDK Manager Version: 1.1.0.6343

Hello,

we are planning on using the PathNet in one of our applications. Therefore, we have some general questions:

1. Is it correct that PathNet requires a rectified camera image as input?

2. Right now, we are using the sf3324 camera. If a rectified image is required, then we have to calibrate the camera. Do we have to do the intrinsic calibration individually for every sf3324 camera? Or is it possible to do the intrinsic calibration once for one sf3324 camera and use the parameters obtained from the calibration for the other sf3324 cameras as well? Or are there maybe some provided rig files with the sf3324 camera parameters already available?

3. The PathNet sample uses the dwPathDetector_initializeFromPathNetWithCameraRig function which needs a dwTransformation3f cam2rig variable as input. In the sample, the transformation is obtained by using the dwRig_getSensorToRigTransformation function. For this function, the documentation says that it “relates the sensor and the rig coordinate system”. Now, my question is: How is the origin of the rig coordinate system defined? I was not able to find the definition of the origin in the documentation.

Answering the three questions above would help us a lot! Thanks in advance!

[shayNV]

Hello @kn10,

@kn10

1. Is it correct that PathNet requires a rectified camera image as input?

No. PathNet uses a direct 60FOV image from 2MP AR0231 front camera. The 2MP image is downscaled to 960x604 (0.5MP) and then the top portion is cropped out to remove sky pixels to 960x400 and fed to the network.

@kn10

2. Right now, we are using the sf3324 camera. If a rectified image is required, then we have to calibrate the camera. Do we have to do the intrinsic calibration individually for every sf3324 camera? Or is it possible to do the intrinsic calibration once for one sf3324 camera and use the parameters obtained from the calibration for the other sf3324 cameras as well? Or are there maybe some provided rig files with the sf3324 camera parameters already available?

So the intrinsic values of the camera should be the same between the cameras but the extrinsic parameters will be different due to their different orientation. So I fear you will have to calibrate all cameras. The extrinsic calibration parameters are only required to project the image space predictions of PathNet to world space.
But mind that PathNet is only trained to infer from images from the front facing camera. Its performance will not be good with side or rear cameras.

@kn10

3. The PathNet sample uses the dwPathDetector_initializeFromPathNetWithCameraRig function which needs a dwTransformation3f cam2rig variable as input. In the sample, the transformation is obtained by using the dwRig_getSensorToRigTransformation function. For this function, the documentation says that it “relates the sensor and the rig coordinate system”. Now, my question is: How is the origin of the rig coordinate system defined? I was not able to find the definition of the origin in the documentation.

The center of the rear axle of the car is taken as the origin of the rig coordinate system.
this is described in the documentation of DW: Tutorials->Basic Tutorials → DriveWorks 101 → Conventions ->Coordinate Systems