Using endoscopic_tool_tracking app with /dev/video0 V4L

Healthcare Holoscan SDK
1

Hi,
I have the holohub working and i am running the tool tracking on files
The sample is written for AJA of replayer.
What should I do to run it with V4L /dev/video0 camera?
The tao_peaplenet uses V4L but its hard to understand from it how to modify the endoscopic_tool_tracking
This is the app I am trying to use: /holohub/applications/endoscopy_tool_tracking/python
Cheers,
Raz

2

Hi Raz, the tao_peoplenet app and the endoscopy_tool_tracking app are composed of different operators. tao_peoplenet demonstrates using the V4L2VideoCaptureOp, the source in the SDK is here. The endoscopy_tool_tracking app demonstrates using either the AJA operator or the video replayer operator. You could replace the source operator in endoscopy_tool_tracking app with V4L2VideoCaptureOp for source (and modify downstream operators/parameters) for your use case.

3

I estimated that its possible, i didnt find any guidance how to do it.
The entire holohub pipeline building is not very clear. For instance which formater do i need to use after V4L. Which part of the code I need to modify. Are you able to provide guidance for that. Do you know from whom can I get it.

4

For the format converter parameters, generally it’s to suit the need for the ML model being run, you could refer to tao_peoplenet.yaml#L20-L26 and endoscopy_tool_tracking.yaml#L42-L55 to see how those two apps defined the format_converter parameters. Since you’re modifying the endoscopy_tool_tracking app, likely you’d want to keep these same parameters:

  out_tensor_name: source_video
  out_dtype: "float32"
  scale_min: 0.0
  scale_max: 255.0
  resize_width: 854
  resize_height: 480

For connecting the operators and defining the app graph, you could refer to the add_flow calls, and the SDK documentation Creating an Application - NVIDIA Docs.

5

The code here support AJA and repleayer , what is the code that i need for the V4L:
if is_aja:
aja_kwargs = self.kwargs(“aja”)
source = AJASourceOp(self, name=“aja”, **aja_kwargs)

        # 4 bytes/channel, 4 channels
        width = aja_kwargs["width"]
        height = aja_kwargs["height"]
        is_rdma = aja_kwargs["rdma"]
        is_aja_overlay_enabled = is_aja and aja_kwargs["enable_overlay"]
        source_block_size = width * height * 4 * 4
        source_num_blocks = 3 if is_rdma else 4
    else:
        width = 854
        height = 480
        video_dir = self.sample_data_path
        if not os.path.exists(video_dir):
            raise ValueError(f"Could not find video data: {video_dir=}")
        source = VideoStreamReplayerOp(
            self,
            name="replayer",
            directory=video_dir,
            **self.kwargs("replayer"),
        )
        # 4 bytes/channel, 3 channels
        source_block_size = width * height * 3 * 4
        source_num_blocks = 2
6

For using the V4L2 operator within an application similar to the above for AJA, please see tao_peoplenet’s application compose method, especially the definition of https://github.com/nvidia-holoscan/holohub/blob/main/applications/tao_peoplenet/tao_peoplenet.py#L238.