Realsense D435 stream stuck on Xavier NX 16GB

I am trying to run my realsense D435 on my Jetson Xavier NX16 using pyrealsense2.

First of all it seems like when I am able to start the camera stream, something is off about the auto exposure on the RGB camera.
Secondly, if my program crashes and I have to restart it, I am met with an error because the stream does not get released.

self.profile = self.pipeline.start(self.config)
RuntimeError: xioctl(VIDIOC_S_FMT) failed Last Error: Device or resource busy

I have build librealsense2 and pyrealsense2 from source code to make it work on the Xavier NX16 and the realsense-viewer works fine, but when using it with pyrealsense2 it does not work so well.
Here is my setup code for the stream with pyrealsense:

import pyrealsense2 as rs
self.camera_config = camera_config
            self.profile = None

            context = rs.context()
            self.pipeline = rs.pipeline(context)
            self.config = rs.config()

            fps = camera_config.fps
            width, height = camera_config.resolution
            rgb_width, rgb_height = 1920, 1080 # 1280, 720 for D455
            self.config.enable_stream(, width, height, rs.format.z16, fps)
            self.config.enable_stream(, rgb_width, rgb_height, rs.format.rgb8, 30)

            self.profile = self.pipeline.start(self.config)  # This is where it crashes
            profile = self.profile.get_stream( # Fetch stream profile for color stream
            intr = profile.as_video_stream_profile().get_intrinsics() # Downcast to video_stream_profile and fetch intrinsics
            # print(intr.ppx, intr.ppy, intr.fx, intr.fy)
            roisensor = self.profile.get_device().first_roi_sensor()
            roi = roisensor.get_region_of_interest()

This seems to be the pyrealsense2 issue.

Have reference to below relative topic