No its not normal in the general case. You might wish to file a bug. If you do, you’ll be asked for a complete test case demonstrating the difference.
I would also recommend that Jetson AGX Xavier questions get posted on the AGX Xavier forum. You generally have AGX Xavier experts there as well as people familiar with those kinds of discussions. For example, there might be a specific known performance regression that the experts there would be familiar with. Jetson also has a number of system performance settings that might be appropriate to review, and the experts there can point you to those.