Jetson Multicamera Pipelines: Project Showcase

Autonomous Machines Jetson & Embedded Systems Jetson Projects
1

Jetson Multicamera Pipelines

Hi Everyone!

I’d like to share with you my intern project completed during my time here at Nvidia.

It’s a python package which:

  • Makes it easy to compose multi-camera pipelines
  • Lets you take advantage of different HW accelerators on Nvidia Jetson modules for minimal CPU usage
  • Allows you to build custom logic on top of the detections from multiple cameras

Demo

Below is a demo video of person-following logic built on top of jetmulticam package:

External Media

In this demo we have 3 intependent cameras with total ~270° field of view, like in the figure below. Red Boxes correspond to DashCamNet detections, green ones to PeopleNet. The PeopleNet detections are used to perform person following logic. Source code for this example is available on our github: example-person-following.py

camera_layout_1
camera_layout_11212×843 20.4 KB

Getting Started

The project allows you to implement complex logic (capture, batching, HW inference, HW encoding) with multiple cameras using relatively simple python code. A typical pipeline can look like in the flowchart below:

flow
flow1506×399 38.7 KB

Below is a code snippet that presents you the capabilities of the package:

import time
from jetmulticam import CameraPipelineDNN
from jetmulticam.models import PeopleNet, DashCamNet

if __name__ == "__main__":

    pipeline = CameraPipelineDNN(
        cameras=[2, 5, 8],
        models=[
            PeopleNet.DLA1,
            DashCamNet.DLA0,
            # PeopleNet.GPU
        ],
        save_video=True,
        save_video_folder="/home/nx/logs/videos",
        display=True,
    )

    while pipeline.running():
        arr = pipeline.images[0] # np.array with shape (1080, 1920, 3), i.e. (1080p RGB image)
        dets = pipeline.detections[0] # Detections from the DNNs
        time.sleep(1/30)

With this snippet we create the pipeline that allows us to:

  • Capture video from 3 cameras
  • Batch the images for DNN inference
  • Deploy 2 DNNs (PeopleNet and DashCamNet) on 2 different HW accelerators
  • Encode the video with h264 HW encoder and save it to mkv file
  • Display video on the screen for debugging
  • Write your own logic in python using parsed detentions in the main thread.

If you’re interested, please check out the project’s github repo and considering leaving a star!
Detailed info on how to install and get started with the project are available in the Quickstart in the repo.

If you have any questions, feel free to post them below!

1 Like
2

would it work on Nvida Jetson Nano 2GB?

1 Like