CSI Camera Video is Spatially Shifted (BGR Padding Issue) When Using subprocess on Jetson Orin Nano / CSIカメラ映像がsubprocess利用時に左右にズレる（BGRパディング問題）

[drufin.iruka]

Hello,

This message has been translated using an online tool. Please excuse any awkward phrasing.

I am currently trying to use a CSI camera on my system. I was able to display the video feed by executing GStreamer via subprocess and reading the raw data from stdout into NumPy.
(The colors are correct with both BGRx and I420 pipelines.)

However, the video output is spatially shifted and appears split into two halves, as shown in the attached screenshot.

I suspect this spatial shift is caused by a padding issue (extra bytes at the end of each line). To resolve this, what would be the most effective solution (a workaround or specific setting) on either the GStreamer pipeline side (e.g., in caps or filesink settings) or the NumPy reading side?

I apologize for the intrusion on your busy schedule, but I would be grateful for any advice you can offer.

Thank you very much.

HW: NVIDIA Jetson Orin Nano Super Developer Kit
Camera: Raspberry Pi Camera Module 2
OS: Ubuntu 22.04.5 LTS
SW: OpenCV 4.12.0

[Attached Image of the Spatially Shifted Output]

[Attached Code]

import cv2
import subprocess as sp
import numpy as np

GSTREAMER_COMMAND = [
    "gst-launch-1.0",
    "nvarguscamerasrc",
    "sensor-mode=4",
    "!",
    "video/x-raw(memory:NVMM),width=1280,height=720,framerate=60/1,format=NV12",
    "!",
    "nvvidconv",
    "!",
    "video/x-raw,format=BGRx", 
    "!",
    "videoconvert",           
    "!",
    "video/x-raw,format=BGR",
    "!",
    "filesink",
    "location=/dev/stdout"
]

WIDTH, HEIGHT = 1280, 720
CHANNELS = 3 
# GStreamerプロセスをPythonから起動する
try:
    pipe = sp.Popen(GSTREAMER_COMMAND, stdout=sp.PIPE, bufsize=WIDTH*HEIGHT*CHANNELS)
except Exception as e:
    print(f"エラー：GStreamerプロセスの起動に失敗しました。{e}")
    exit()

print(f"カメラを開きました。映像が表示されたら、ウィンドウをクリックして 'q' で終了できます。")

while True:
    raw_frame_size = WIDTH * HEIGHT * CHANNELS
    
    # 標準出力から必要なバイト数だけを正確に読み込む
    raw_image = pipe.stdout.read(raw_frame_size)
    
    if not raw_image or len(raw_image) < raw_frame_size: # フレームサイズが足りない場合もエラーにする
        print("エラー：フレームの取得に失敗しました。GStreamerパイプラインが停止しました。")
        break
        
    # バイトデータをNumPyの配列に変換する
    frame = np.frombuffer(raw_image, dtype=np.uint8).reshape((HEIGHT, WIDTH, CHANNELS))

    cv2.imshow('Camera Stream (Press "q" to exit)', frame)

    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

pipe.terminate()
cv2.destroyAllWindows()

────────────────────────────────
こんにちは。
この文章は翻訳ツールを介しています。読みづらい点があればご容赦ください。

私は現在CSIカメラを使用しようとしています。
GStreamerをsubprocessで実行し、標準出力からNumPyで読み取る方法で映像は表示できました。
（BGRxとI420どちらのパイプラインでも色は正常です）
しかし、映像が添付画像のように左右にずれて二分割された状態になってしまいます。

この空間的なズレ（パディングに起因すると推測）を解消するには、GStreamerパイプライン（capsやfilesinkの設定）側、またはNumPy読み込み側で、どのような設定（裏技）が最も効果的でしょうか？

お忙しいところ恐縮ですが、ご助言をいただけると幸いです。
よろしくお願いいたします。

HW NVIDIA Jetson Orin Nano Super Developer Kit
Cam Raspberry Pi Camera Module 2
OS Ubuntu 22.04.5 LTS
SW OpenCV 4.12.0

[drufin.iruka]

[English Translation for the Forum]

Update: I have found a temporary fix for the spatial shift.
By applying np.roll(frame, -486, axis=1) (or +800px) to each frame, the video is now displayed correctly.

The current code is attached below.

This means that the shift is exactly 486 pixels to the left, which is 6 pixels (18 bytes) more than the expected 480-pixel difference (1280 - 800).

This suggests a hard-coded padding of 6 pixels/18 bytes is being added by GStreamer’s filesink or the underlying memory handling.

Could anyone advise on how to eliminate this 6-pixel/18-byte padding in the GStreamer pipeline itself to prevent the shift and remove the need for np.roll()?

Thank you again for your time.

根本的に解決してませんが、元々ズレている映像を+800pxまたは-486px矯正して暫定対応できました。
下記が現状コードです。

import cv2
import subprocess as sp
import numpy as np
import sys

# I420を経由し、色が正常に出たパイプライン
GSTREAMER_COMMAND = [
    "gst-launch-1.0",
    "nvarguscamerasrc",
    "sensor-mode=4",
    "!",
    "video/x-raw(memory:NVMM),width=1280,height=720,framerate=30/1,format=NV12",
    "!",
    "nvvidconv",
    "!",
    "video/x-raw,format=I420",
    "!",
    "videoconvert",
    "!",
    "video/x-raw,format=BGR", 
    "!",
    "filesink",
    "location=/dev/stdout" 
]

WIDTH, HEIGHT = 1280, 720
CHANNELS = 3
pipe = None
try:
    pipe = sp.Popen(GSTREAMER_COMMAND, stdout=sp.PIPE, bufsize=WIDTH*HEIGHT*CHANNELS)
except Exception as e:
    print(f"エラー：GStreamerプロセスの起動に失敗しました。{e}", file=sys.stderr)
    sys.exit(1)

if pipe is None:
    print("致命的なエラー：GStreamerプロセスが起動できませんでした。", file=sys.stderr)
    sys.exit(1)


print(f"カメラを開きました。映像が表示されたら、ウィンドウをクリックして 'q' で終了できます。")

# --- vvv 変更箇所 ------------------------------------------

ZURE_PIXELS = -486 # ズレのピクセル数
READ_CHANNELS = 3 # 読み込みは3チャンネルで行う

while True:
    raw_frame_size = WIDTH * HEIGHT * READ_CHANNELS
    
    # データを読み込む
    raw_image = pipe.stdout.read(raw_frame_size)
    
    if not raw_image or len(raw_image) < raw_frame_size: 
        print("\nエラー：フレームの取得に失敗しました。GStreamerパイプラインが停止しました。", file=sys.stderr)
        break
        
    # 1. BGR (3チャンネル) としてNumPy配列に変換する
    frame = np.frombuffer(raw_image, dtype=np.uint8).reshape((HEIGHT, WIDTH, READ_CHANNELS))
    
    # 2. 映像を横方向（axis=1）に、ズレているピクセル数分ローテーションさせる
    # 'roll'は、ズレたデータを元の位置に戻す、NumPyの強力な機能だよ！
    frame = np.roll(frame, ZURE_PIXELS, axis=1)
    
    # 3. 必要なら、ローテーション後にできた余分な領域を切り捨てる処理も加える
    cv2.imshow('Camera Stream (Press "q" to exit)', frame)

    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

# --- ^^^ ------------------------------------------

pipe.terminate()
cv2.destroyAllWindows()

[proventusnova]

Hello @drufin.iruka,

Couple of ideas:

1. Can you try capturing with a GStreamer pipeline to remove Python, Numpy and all that complexity from the equation?

gst-launch-1.0 nvarguscamerasrc ! nvvidconv ! ximagesink

• If you still see the shifting issue. Then the problem is probably a driver problem. Something in the device tree configuration or the camera register table is causing the video to shift.

• If the problem does not show on GStreaerm. Then the issue is how you are trying to capture using Python.

2. I believe running GStreamer with subprocess is a bit complicated. I would suggest you use our sample Python script from our How to Capture Images From Nvidia Jetson Using Python wiki.. You can simply comment the print(f"Received frame of resolution: {width}x{height}") line to avoid noise and uncomment the following section:

# Optional: convert to numpy array
# import numpy as np
# np_frame = np.frombuffer(frame_data, dtype=np.uint8).reshape((height, width, 3))

That will give you the numpy array, after that you can simply display it with OpenCV.

best regards,
Andrew
Embedded Software Engineer at ProventusNova

Go From Broken Jetson Builds to Production-Ready AI