Hello, we have an application running inside the docker of a Jetson Xavier device. It uses gstreamer to save camera images to video files with:
path = f"appsrc ! video/x-raw, format=(string)BGR ! videoconvert ! video/x-raw,format=BGRx ! nvvidconv ! nvv4l2h264enc profile=4 preset-level=3 control-rate=0 cabac-entropy-coding=1 bitrate={bitrate} ! video/x-h264, streamformat=(string)byte-stream ! h264parse ! qtmux ! filesink location={str(full_path)}"
fourcc = 0
self.writer = cv2.VideoWriter(path, fourcc, self.video_fps, (image.shape[1], image.shape[0]))
...
self.writer.write(image)
The docker is launched with
sudo docker run --rm --runtime nvidia --privileged --net=host ... <docke_image>
After several hours of running the application inside the docker, the application will stop writing frames to the video files, and the thread got stuck there. Restart the docker or reboot the system doesn’t solve the problem. And the gstreamer outside of the docker is also no longer working. Here is the verbose output from our test command using gst-launch:
nvidia@miivii-tegra-00044bcc0754:~$ gst-launch-1.0 filesrc location=test.mp4 ! decodebin name=dec ! queue ! nvvidconv ! nvv4l2h264enc profile=4 preset-level=3 control-rate=0 cabac-entropy-coding=1 bitrate=2000000 ! video/x-h264, streamformat=\(string\)byte-stream ! h264parse ! qtmux ! filesink location=out.mp4 -v
Setting pipeline to PAUSED ...
Opening in BLOCKING MODE
Pipeline is PREROLLING ...
/GstPipeline:pipeline0/GstDecodeBin:dec/GstTypeFindElement:typefind.GstPad:src: caps = video/quicktime, variant=(string)iso
/GstPipeline:pipeline0/GstDecodeBin:dec/GstTypeFindElement:typefind.GstPad:src: caps = NULL
/GstPipeline:pipeline0/GstDecodeBin:dec/GstMultiQueue:multiqueue0.GstPad:src_0: caps = video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)4.1, profile=(string)high, codec_data=(buffer)01640029ffe1001967640029acd94050065a1000000300100000030080f183196001000568ebecb22c, width=(int)1280, height=(int)800, framerate=(fraction)4/1, pixel-aspect-ratio=(fraction)1/1
/GstPipeline:pipeline0/GstDecodeBin:dec/GstH264Parse:h264parse1.GstPad:sink: caps = video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)4.1, profile=(string)high, codec_data=(buffer)01640029ffe1001967640029acd94050065a1000000300100000030080f183196001000568ebecb22c, width=(int)1280, height=(int)800, framerate=(fraction)4/1, pixel-aspect-ratio=(fraction)1/1
/GstPipeline:pipeline0/GstDecodeBin:dec/GstMultiQueue:multiqueue0: max-size-buffers = 5
/GstPipeline:pipeline0/GstDecodeBin:dec/GstMultiQueue:multiqueue0: max-size-time = 0
/GstPipeline:pipeline0/GstDecodeBin:dec/GstMultiQueue:multiqueue0: max-size-bytes = 2097152
/GstPipeline:pipeline0/GstDecodeBin:dec/GstMultiQueue:multiqueue0.GstMultiQueuePad:sink_0: caps = video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)4.1, profile=(string)high, codec_data=(buffer)01640029ffe1001967640029acd94050065a1000000300100000030080f183196001000568ebecb22c, width=(int)1280, height=(int)800, framerate=(fraction)4/1, pixel-aspect-ratio=(fraction)1/1
/GstPipeline:pipeline0/GstDecodeBin:dec/GstH264Parse:h264parse1.GstPad:src: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, level=(string)4.1, profile=(string)high, width=(int)1280, height=(int)800, framerate=(fraction)4/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
/GstPipeline:pipeline0/GstDecodeBin:dec/GstMultiQueue:multiqueue0: max-size-buffers = 5
/GstPipeline:pipeline0/GstDecodeBin:dec/GstMultiQueue:multiqueue0: max-size-time = 0
/GstPipeline:pipeline0/GstDecodeBin:dec/GstMultiQueue:multiqueue0: max-size-bytes = 2097152
Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
/GstPipeline:pipeline0/GstDecodeBin:dec/nvv4l2decoder:nvv4l2decoder0.GstPad:sink: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, level=(string)4.1, profile=(string)high, width=(int)1280, height=(int)800, framerate=(fraction)4/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
/GstPipeline:pipeline0/GstDecodeBin:dec/GstCapsFilter:capsfilter1.GstPad:src: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, level=(string)4.1, profile=(string)high, width=(int)1280, height=(int)800, framerate=(fraction)4/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
/GstPipeline:pipeline0/GstDecodeBin:dec/GstCapsFilter:capsfilter1.GstPad:sink: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, level=(string)4.1, profile=(string)high, width=(int)1280, height=(int)800, framerate=(fraction)4/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
/GstPipeline:pipeline0/GstDecodeBin:dec/nvv4l2decoder:nvv4l2decoder0.GstPad:src: caps = video/x-raw(memory:NVMM), format=(string)NV12, width=(int)1280, height=(int)800, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)4/1
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:sink: caps = video/x-raw(memory:NVMM), format=(string)NV12, width=(int)1280, height=(int)800, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)4/1
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:sink: caps = video/x-raw(memory:NVMM), format=(string)NV12, width=(int)1280, height=(int)800, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)4/1
/GstPipeline:pipeline0/GstDecodeBin:dec.GstDecodePad:src_0.GstProxyPad:proxypad2: caps = video/x-raw(memory:NVMM), format=(string)NV12, width=(int)1280, height=(int)800, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)4/1
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1280, height=(int)800, framerate=(fraction)4/1, pixel-aspect-ratio=(fraction)1/1, format=(string)NV12, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono
/GstPipeline:pipeline0/nvv4l2h264enc:nvv4l2h264enc0.GstPad:src: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, profile=(string)NULL, level=(string)NULL, width=(int)1280, height=(int)800, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)4/1, interlace-mode=(string)progressive, colorimetry=(string)bt709, chroma-site=(string)mpeg2, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-h264, streamformat=(string)byte-stream, stream-format=(string)byte-stream, alignment=(string)au, profile=(string)NULL, level=(string)NULL, width=(int)1280, height=(int)800, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)4/1, interlace-mode=(string)progressive, colorimetry=(string)bt709, chroma-site=(string)mpeg2, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono
/GstPipeline:pipeline0/GstH264Parse:h264parse0.GstPad:sink: caps = video/x-h264, streamformat=(string)byte-stream, stream-format=(string)byte-stream, alignment=(string)au, profile=(string)NULL, level=(string)NULL, width=(int)1280, height=(int)800, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)4/1, interlace-mode=(string)progressive, colorimetry=(string)bt709, chroma-site=(string)mpeg2, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, profile=(string)NULL, level=(string)NULL, width=(int)1280, height=(int)800, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)4/1, interlace-mode=(string)progressive, colorimetry=(string)bt709, chroma-site=(string)mpeg2, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono
Redistribute latency...
NvMMLiteOpen : Block : BlockType = 4
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4
/GstPipeline:pipeline0/nvv4l2h264enc:nvv4l2h264enc0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1280, height=(int)800, framerate=(fraction)4/1, pixel-aspect-ratio=(fraction)1/1, format=(string)NV12, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:sink: caps = video/x-raw(memory:NVMM), format=(string)NV12, width=(int)1280, height=(int)800, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)4/1
H264: Profile = 100, Level = 0
^Chandling interrupt.
Interrupt: Stopping pipeline ...
ERROR: pipeline doesn't want to preroll.
Setting pipeline to NULL ...
^C
Below are some version information:
nvidia@miivii-tegra-00044bcc0754:~$ gst-launch-1.0 --gst-version
GStreamer Core Library version 1.14.5
nvidia@miivii-tegra-00044bcc0754:~$ uname -a
Linux miivii-tegra-00044bcc0754 4.9.140+ #1 SMP PREEMPT Wed Dec 11 20:10:04 CST 2019 aarch64 aarch64 aarch64 GNU/Linux
nvidia@miivii-tegra-00044bcc0754:~$cat /etc/nv_tegra_release
# R32 (release), REVISION: 2.1, GCID: 16294929, BOARD: t186ref, EABI: aarch64, DATE: Tue Aug 13 04:45:36 UTC 2019
nvidia@miivii-tegra-00044bcc07d2:~$ docker version
Client:
Version: 3.0.13+azure
API version: 1.40
Go version: go1.13.11
Git commit: dd360c7c0de8d9132a3965db6a59d3ae74f43ba7
Built: Thu May 28 20:41:25 2020
OS/Arch: linux/arm64
Experimental: false
Server:
Engine:
Version: 3.0.13+azure
API version: 1.40 (minimum version 1.12)
Go version: go1.13.11
Git commit: 77e06fda0c
Built: Mon Jun 1 20:23:47 2020
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: v1.2.13
GitCommit: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683
We will very appreciate if anyone can suggest the fix for this issue.