Training the Isaac SDK Pose CNN Decoder with Isaac Sim Omniverse

I am currently trying to use the Pose CNN Decoder with isaac-sim, but while training on the generated data the decoder just turns all white. With reducing the learning rate to e-6 and add_noise_to_image=true the edges of the decoder stay black but are also slowly turning white.

The training object is the armadillo figure from the Stanford 3D Scanning Repository:
The data is generated by placing multiple of these figures on a table and using the offline dataset generation with isaac-sdk. How can I stop the decoder from saturating ?
Is it a problem to have multiple objects on the table because they sometimes overlap? Any help is greatly appreciated.

Update: I tried to train on the default Isaac-Sim pose estimation example in python_samples/isaac_sdk/ still no convergence after ~30000 steps.

Solved: When using isaac-sim with pose cnn encoder the there has to be a uniform mesh background for the decoder training data that is always visible otherwise the background of the decoder training image is 255 not 0 like it should be, maybe the example python_samples/isaac_sdk/ should be corrected.