Thanks for your suggestion regarding using OpenCV’s camera calibration method to generate the projection matrix. I have implemented it and tested the results. Unfortunately, I am still facing the same issue.
To ensure correctness, I am attaching the following:
The Python script I used to generate the projection matrix. The script includes both:
A function to compute the projection matrix from four 3D-2D correspondences.
A verification function that projects the 3D points onto the image to check the accuracy of the matrix.
import numpy as np
import cv2
def project_world_points_on_image(image_path, world_points, P, output_path):
img = cv2.imread(image_path)
image_points = []
# Project each world point
for point in world_points:
ph = np.append(point, 1).reshape(4, 1)
a = P @ ph
a = a / a[2]
x = float(a[0][0])
y = float(a[1][0])
image_points.append([x, y])
point_color = (0, 0, 255)
point_size = 5
thickness = -1
for i, point_2d in enumerate(image_points):
x_int, y_int = int(round(point_2d[0])), int(round(point_2d[1]))
cv2.circle(img, (x_int, y_int), point_size, point_color, thickness)
cv2.imshow('Image with Projected Points', img)
cv2.waitKey(0)
cv2.destroyAllWindows()
if output_path:
cv2.imwrite(output_path, img)
def compute_projection_matrix(K, points_3D, points_2D):
dist_coeffs = np.zeros((4, 1))
success, rvec, tvec = cv2.solvePnP(points_3D, points_2D, K, dist_coeffs, flags=cv2.SOLVEPNP_P3P)
R, _ = cv2.Rodrigues(rvec)
# Concatenate [R | t] to form the 3x4 extrinsic matrix
Rt = np.hstack((R, tvec))
# Compute the projection matrix: P = K * [R | t]
P = np.dot(K, Rt)
return P
K = np.array([
[330.76614, 0.0, 321.2513],
[0.0, 330.76614, 163.69617],
[0.0, 0.0, 1.0]
], dtype=np.float32)
points_3D = np.array([
[2.48, 2, 0],
[2.82, 0.542, 0],
[0.9346, 3.029, 0.95],
[2.0695, 1.9345, 0.95]
]) * 100 # Multiply by 100 to be in cm
points_2D = np.array([
[328, 139],
[405, 171],
[203, 97],
[300, 95]
], dtype=np.float64)
P = compute_projection_matrix(K, points_3D, points_2D)
print("Projection Matrix:\n", P)
print("projectionMatrix_3x4:")
for row in P:
print(" -", list(row)[0])
print(" -", list(row)[1])
print(" -", list(row)[2])
print(" -", list(row)[3])
image_path = r"C:\Users\AsafShim\Downloads\j-transparent.jpg"
output_path = "projected_points_cv.jpg"
project_world_points_on_image(image_path, points_3D, P, output_path)
This image shows
The 3D points in world coordinates.
and the corresponding manually marked 2D points on the image.
I verified that when using the projection matrix to project the 3D points, they align exactly with their corresponding pixel locations in the image.
This is the projection matrix I placed in the .yml file.
projectionMatrix_3x4_w2p:
- 398.2512522753727
- 62.09416837270102
- -223.9395048330216
- 69863.75993218269
- -67.27429151301806
- -26.7476309597462
- -361.8859086935327
- 98758.5434165381
- 0.3695215531960892
- 0.7299604267387418
- -0.5749883452027038
- 314.34493574459873
the height and radius of the cylinder model
modelInfo:
height: 205
radius: 33
Here is the result video, as you can see the bounding boxes remain too large and incorrect, and the ID assignment is still unstable (people frequently change IDs).
Since the verification process confirms that the projection matrix is correct, but the tracker still fails, I suspect something else might be affecting the tracking. Do you have any insights on additional parameters or settings that might influence SV3DT’s behavior?
I would appreciate any further guidance on what could be causing this issue. Thanks again for your help!
