Assigning Classes to Object Sides

Hi everyone!

I have a model of this object and can generate data with it. However, I now need to assign classes to each side of this object to train a YOLO model. I used the Bounding_box_2d_tight and instance_segmentation annotators to create segmented annotations, successfully segmenting each object in the image. But when I try to assign a class to each object, I don’t have enough information to determine which class corresponds to the current position of the object.

I attempted to use the transformation matrix from Bounding_box_3d to calculate the object’s rotation and assign the class based on the angle of rotation, but this often assigns the wrong class.

Does anyone have any ideas on how to approach this task?
Thank you in advance!

Here is the picture of my object: