To resole this issue
I update my version to 2022.1.1
Howveer, there is a bug in the get_ground_truth portion of Synthetic data helper.
All odd number of viewports return the same images (viewport 1, 3, 5 … ), However, the camera matrix are not the same, and what I see in the non-headless mode actually makes sense
After reading this issue:
I realized this is a multi-gpu issue. I switched to only using one GPU this error disappeared. Originally I was uing a 2 GPU machine