Intrinsic and Extrinsic Camera Calibration

[kn10]

Please provide the following info (check/uncheck the boxes after creating this topic):
Software Version
DRIVE OS Linux 5.2.6
DRIVE OS Linux 5.2.6 and DriveWorks 4.0
DRIVE OS Linux 5.2.0
DRIVE OS Linux 5.2.0 and DriveWorks 3.5
NVIDIA DRIVE™ Software 10.0 (Linux)
NVIDIA DRIVE™ Software 9.0 (Linux)
other DRIVE OS version
other

Target Operating System
Linux
QNX
other

Hardware Platform
NVIDIA DRIVE™ AGX Xavier DevKit (E3550)
NVIDIA DRIVE™ AGX Pegasus DevKit (E3550)
other

SDK Manager Version
1.8.0.10363
other

Host Machine Version
native Ubuntu 18.04
other

Hello,

we want to calibrate eight cameras we have mounted onto our vehicle. We followed the tutorial “static camera calibration” in the documentation. However, we have some issues.

According to the documentation, the images validation-<camera>.jpg generated after executing ./calibration-graph-cli --dir=<calib_data_path> should show a green overlay which indicate the quality of the calibration.

After executing ./calibration-graph-cli --dir=<calib_data_path>, in our case, the validation-<camera>.jpg images only show the blue dots on the targets and do not show the green overlay. The image below is an example of the output we obtain after executing the calibration:

Also, the validation-intrinsics-<camera>.jpg images only show the blue points, but the red points and the yellow lines are not visible (which should be visible according to the documentation):

This obviously indicates that the calibration was not successful.

What is the problem here? How can we solve this issue?

[kayccc]

so while using external camera, you have make sure the followings, right?

• Auto focus must be disabled.
• Any image stabilization must be disabled
• Zoom, focus and aperture should not change

What’s the camera module?

[kn10]

Hello @kayccc ,

@kayccc

What’s the camera module?

We are using a Sony Alpha 6400 (ILCE-6400) with a sony 18-135mm objective (SEL 18135).
We used a fixed focal length of 18mm during the process.

@kayccc

Auto focus must be disabled.

It seems like we used auto focus during the capturing of the external camera images.
Can this be the reason for the failure of the calibration process (and the reason why the green validation rectangles are missing on the validation images)?
Why is an auto focus not allowed?

[kn10]

Hello,

is there any updates you can provide us on our questions above?

Also, can you please tell us which camera and objective you are using exactly for the camera calibration? We would then purchase the exact same camera for the camera calibration.

[VickNV]

Hi @kn10

Please try with the tools in DriveWorks 4.0 and see if it works well.

[kn10]

Hi @VickNV ,

we tried installing DriveWorks 4.0 but had big issues doing so. We can’t afford spending more time on trying to install DriveWorks 4.0, which is why we are still using the NVIDIA DRIVE Software 10.

Nevertheless, can you please answer this question from my previous post:

@kn10

Also, can you please tell us which camera and objective you are using exactly for the camera calibration? We would then purchase the exact same camera for the camera calibration.

Thank you very much.

[VickNV]

@kn10

After executing ./calibration-graph-cli --dir=<calib_data_path>, in our case, the validation-<camera>.jpg images only show the blue dots on the targets and do not show the green overlay. The image below is an example of the output we obtain after executing the calibration:

Also, the validation-intrinsics-<camera>.jpg images only show the blue points, but the red points and the yellow lines are not visible (which should be visible according to the documentation):

Please share the output messages in the console.

@kn10

we tried installing DriveWorks 4.0 but had big issues doing so. We can’t afford spending more time on trying to install DriveWorks 4.0, which is why we are still using the NVIDIA DRIVE Software 10.

Please create a topic for this. Maybe we can help you with it.

@kn10

Also, can you please tell us which camera and objective you are using exactly for the camera calibration? We would then purchase the exact same camera for the camera calibration.

The most used cameras should be those in Hyperion. They are ON Semi - AR0231 ones listed in https://developer.nvidia.com/drive/ecosystem-hw-sw. You can refer to DRIVE Hyperion Kit Quick Start Guide.

[kn10]

Hi @VickNV ,

@VickNV

The most used cameras should be those in Hyperion. They are ON Semi - AR0231 ones listed in DRIVE Ecosystem Hardware and Software Components | NVIDIA Developer. You can refer to DRIVE Hyperion Kit Quick Start Guide.

I did not mean the car cameras. My question was regarding the external camera you are using for the camera calibration. Can you please tell us the exact model and objective of the external camera you are using for the camera calibration process?

[VickNV]

Please refer to External Camera model required for calibration. Thanks.

[kn10]

Hello @VickNV ,

@VickNV

Please share the output messages in the console.

In the following the output messages of the console in a text file:
console_output_camera_calib (243.1 KB)

We cannot really tell why the calibration does not work.
Can you please provide us with information as soon as possible. I hope the console output messages help you.

Thanks in advance!

[VickNV]

The information is in the forum topic I shared. Thansk.

[kn10]

As we still have not been able to successfully complete the camera calibration, I am writing in this forum post again.

In the meantime, we bought the same camera and lens you are using, namely the Sony A6000 with the SEL-20F28 lens, which you specified here:

External Camera model required for calibration

Please provide the following info (check/uncheck the boxes after creating this topic): Software Version DRIVE OS Linux 5.2.6 DRIVE OS Linux 5.2.6 and DriveWorks 4.0 DRIVE OS Linux 5.2.0 DRIVE OS Linux 5.2.0 and DriveWorks 3.5 [+] NVIDIA DRIVE™ Software 10.0 (Linux) NVIDIA DRIVE™ Software 9.0 (Linux) other DRIVE OS version other Target Operating System [+] Linux QNX other Hardware Platform [+] NVIDIA DRIVE™ AGX Xavier DevKit (E3550) NVIDIA DRIVE™ AGX Pegasus DevKit (E355…

With this new camera we collected new calibration data.
However, this still did not lead to a successful calibration.

In the following, I will again show the new results we obtained and also provide our data to you and would kindly ask you to check what is wrong with our calibration process.

First, it can be seen in the following image, that the corners of the calibration target can be detected by the calibration tool. But still, as mentioned above, only the blue dots are visible and the green overlay is not:

What is the reason for this?

Further, the validation-intrinsics-<camera>.jpg images again only show the blue points, but the red points and the yellow lines are not visible (which should be visible according to the documentation):

What is the reason for this?

Also, I would like to note that the the json-files of the car cameras, which are generated after the intrinsic calibration (where one needs to record a video of a target moving in front of every car camera), are approximately 400kB ~ 850kB. This file sizes are quite big (for text files) and the files cannot be opened (the PC crashes). Is it normal that these json-files are that big? If not, what could be the reason for this? How big should these files approximately be? How long should the recorded videos be for the intrinsic calibration?

In the following, you can find the data we used for the camera calibration: Download File (dlr.de)

Also, in the following you can find the terminal output after executing ./calibration-graph-cli --dir=<calib_data_path>:
terminal_output_camera_calibration (275.1 KB)

Can you please examine why the calibration is not working?

Thank you very much in advance!

[VickNV]

May I know which DRIVE OS and DriveWorks versions you are using? Did you make sure autofocus is disabled?

[kn10]

We use DRIVE Software 10.0 and we disabled autofocus.

[VickNV]

Besides focus, did you make sure you didn’t change any intrinsics during the process (e.g. zoom, images-stabilization)?
Do you have a chance to try on DRIVE OS 5.2.6 and DriveWorks 4.0? DRIVE Software 10 is quite an old release.

[kn10]

@VickNV

Besides focus, did you make sure you didn’t change any intrinsics during the process (e.g. zoom, images-stabilization)?

Yes, we made sure that we did not change any intrinsics during the process.

@VickNV

Do you have a chance to try on DRIVE OS 5.2.6 and DriveWorks 4.0? DRIVE Software 10 is quite an old release.

We already tried to install DRIVE OS 5.2.6 and DriveWorks 4.0 many times. However, we were not able to successfully update to the new release. For now, we cannot afford to spend more time on trying to update to the new release, but we will try to do so again in the near future. Also, all the other software we developed is already running on DRIVE Software 10.0. The only thing we need to do now is to calibrate the cameras. Besides, we already checked the forum and confirmed that other developers have been able to successfully calibrate the cameras with the DRIVE Software 10.0 release, which should therefore also work in our case.

[VickNV]

Thanks for your confirmation. I’ll check it with our team.

[VickNV]

Please check the information from 4.3.3 Extrinsic Calibration Validation and 4.1.1 targets.json respectively, and see if correcting the unit of the bar lengths can help with this issue. Thanks.

[kn10]

Thank you for your answer, that helped. We now get the green overlay on the targets, as seen in the following picture:

However, the ground plane is leaning towards the right and is not horizontal. Does this mean that the data we collected is not good and that we have to collect new data?

[VickNV]

The coordinate frame is defined by the ground and wheel targets, so you should double-check that the special-targets.json contains the correct entries.