Deepstream UCS test sample fails Invalid argument

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform ASUS Bare Metal
. Ubuntu 22.04.5 LTS
. GPU RTX-6000ADA
. Driver Version: 560.35.05
• DeepStream Version 7.1.0
• TensorRT Version 10.6
• NVIDIA GPU Driver Version 560.35.05
• Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7]
• hca_id: mlx5_0
• transport: InfiniBand (0)
• state: PORT_ACTIVE (4)
• Issue Type questions
• How to reproduce the issue ?
I followed the README in /opt/nvidia/deepstream/deepstream-7.1/sources/apps/sample_apps/deepstream-ucx-test

I ran these commands to set a static IP for the Mellanox ConnectX-7 card,

sudo ip link set dev enp2s0f0np0 down
sudo ip addr add 192.168.168.67/24 dev enp2s0f0np0
sudo ip link set dev enp2s0f0np0 mtu 9000
sudo ip link set dev enp2s0f0np0 up
ip -f inet addr show enp2s0f0np0

Running Server I get:

bill@H4Mrd3:/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-ucx-test$ ./deepstream-ucx-test-app -t 1 -s 192.168.168.67 4000 1920 1080 file:////opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4

Using URI: file:////opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4
Server listening on: 192.168.168.67 : 4000
[1734976545.138070] [H4Mrd3:16076:0] ucp_context.c:1774 UCX WARN UCP version is incompatible, required: 1.13, actual: 1.12 (release 1)

Running Client I get:

bill@H4Mrd3:/opt/nvidia/deepstream/deepstream-7.1/sources/apps/sample_apps/deepstream-ucx-test$ ./deepstream-ucx-test-app -t 1 -c 192.168.168.67 4000 1920 1080 output.mp4

Now saving stream to: output.mp4
Failed to query video capabilities: Invalid argument
[1734979244.542099] [H4Mrd3:19416:0] ucp_context.c:1774 UCX WARN UCP version is incompatible, required: 1.13, actual: 1.12 (release 1)
Running...
End of stream
Returned, stopping playback
Deleting pipeline
bill@H4Mrd3:/opt/nvidia/deepstream/deepstream-7.1/sources/apps/sample_apps/deepstream-ucx-test$

and the Server gives:
Decodebin child added: source
Decodebin child added: decodebin0
Running...
Decodebin child added: qtdemux0
Decodebin child added: multiqueue0
Decodebin child added: h264parse0
Decodebin child added: capsfilter0
Decodebin child added: aacparse0
Decodebin child added: avdec_aac0
Decodebin child added: nvv4l2decoder0
Failed to query video capabilities: Invalid argument
In cb_newpad
In cb_newpad

The Server appears to load the video file and run it but fails with Invalid Argument.

How can I get the Nvidia provided sample app to work?

@gpolaillon @hsinha @garethsb @pravins @sseitz

3

Have you read and followed the /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-ucx-test/README instruction of NVIDIA ConnectX-6 DX, RDMA OFED network stack and Unified Communication-X Library before you start with DeepStream sample app?

4

Yes. I did that as I said in the • How to reproduce the issue ? at the top pf the page.

5

Please use “export GST_DEBUG=3” to get more log.

6

Of course I also run:
sudo mst start
Starting MST (Mellanox Software Tools) driver set
Loading MST PCI module - Success
Loading MST PCI configuration module - Success
Create devices
Unloading MST PCI module (unused) - Success

Then I run:
deepstream-ucx-test-app -t 1 -s 192.168.168.67 4000 1920 1080 file:////opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4

Using URI: file:////opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4
Server listening on: 192.168.168.67 : 4000
[1735054753.607000] [H4Mrd3:6474 :0] ucp_context.c:1774 UCX WARN UCP version is incompatible, required: 1.13, actual: 1.12 (release 1)
0:01:01.567531798 6474 0x60129a7cad70 WARN basesrc gstbasesrc.c:3688:gst_base_src_start_complete: pad not activated yet
Decodebin child added: source
Decodebin child added: decodebin0
0:01:01.567777381 6474 0x60129a7cad70 WARN basesrc gstbasesrc.c:3688:gst_base_src_start_complete: pad not activated yet
Running…
Decodebin child added: qtdemux0
0:01:01.572121530 6474 0x7f507c001050 WARN qtdemux qtdemux.c:3121:qtdemux_parse_trex: failed to find fragment defaults for stream 1
0:01:01.572202653 6474 0x7f507c001050 WARN qtdemux qtdemux.c:3121:qtdemux_parse_trex: failed to find fragment defaults for stream 2
Decodebin child added: multiqueue0
Decodebin child added: h264parse0
Decodebin child added: capsfilter0
Decodebin child added: aacparse0
Decodebin child added: avdec_aac0
Decodebin child added: nvv4l2decoder0
Failed to query video capabilities: Invalid argument
0:01:01.630394604 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:01:01.630407235 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat MJPG
0:01:01.630414525 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:01:01.630420464 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat MJPG
0:01:01.630436630 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:01:01.630443142 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat AV10
0:01:01.630447861 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:01:01.630453938 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat AV10
0:01:01.630466500 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:01:01.630472496 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat DVX5
0:01:01.630477547 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:01:01.630482727 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat DVX5
0:01:01.630491182 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:01:01.630496591 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat DVX4
0:01:01.630501722 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:01:01.630508890 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat DVX4
0:01:01.630518940 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:01:01.630524601 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat MPG4
0:01:01.630529435 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:01:01.630534230 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat MPG4
0:01:01.630542862 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:01:01.630548141 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat MPG2
0:01:01.630552797 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:01:01.630557705 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat MPG2
0:01:01.630567165 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:01:01.630572915 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat H265
0:01:01.630578129 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:01:01.630583430 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat H265
0:01:01.630592934 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:01:01.630598489 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat VP90
0:01:01.630604078 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:01:01.630609425 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat VP90
0:01:01.630617347 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:01:01.630622632 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat VP80
0:01:01.630627633 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:01:01.630633276 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat VP80
0:01:01.630642766 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:01:01.630648598 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat H264
0:01:01.630653785 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:01:01.630658889 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat H264
0:01:01.630906605 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:src Unable to try format: Unknown error -1
0:01:01.630920854 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:src Could not probe minimum capture size for pixelformat Y444
0:01:01.630926787 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:src Unable to try format: Unknown error -1
0:01:01.630932163 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:src Could not probe maximum capture size for pixelformat Y444
0:01:01.630944114 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:src Unable to try format: Unknown error -1
0:01:01.630950243 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:src Could not probe minimum capture size for pixelformat P410
0:01:01.630955840 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:src Unable to try format: Unknown error -1
0:01:01.630960994 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:src Could not probe maximum capture size for pixelformat P410
0:01:01.630971269 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:src Unable to try format: Unknown error -1
0:01:01.630976874 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:src Could not probe minimum capture size for pixelformat PM10
0:01:01.630981928 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:src Unable to try format: Unknown error -1
0:01:01.630986878 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:src Could not probe maximum capture size for pixelformat PM10
0:01:01.630996822 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:src Unable to try format: Unknown error -1
0:01:01.631002272 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:src Could not probe minimum capture size for pixelformat NM12
0:01:01.631007442 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:src Unable to try format: Unknown error -1
0:01:01.631012834 6474 0x7f507c0012c0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:src Could not probe maximum capture size for pixelformat NM12
In cb_newpad
In cb_newpad
0:01:01.825918042 6474 0x7f507c0012c0 WARN v4l2videodec gstv4l2videodec.c:2297:gst_v4l2_video_dec_decide_allocation: Duration invalid, not setting latency
0:01:01.825931887 6474 0x7f507c0012c0 WARN v4l2bufferpool gstv4l2bufferpool.c:1130:gst_v4l2_buffer_pool_start:nvv4l2decoder0:pool:src Uncertain or not enough buffers, enabling copy threshold
0:01:01.826755132 6474 0x7f507c0017a0 WARN v4l2bufferpool gstv4l2bufferpool.c:1607:gst_v4l2_buffer_pool_dqbuf:nvv4l2decoder0:pool:src Driver should never set v4l2_buffer.field to ANY
0:01:01.836072295 6474 0x7f507c001de0 ERROR ucxcommon gstucx.c:736:request_finalize: unable to process UCX message (Connection reset by remote peer)
0:01:01.836083144 6474 0x7f507c001de0 ERROR ucxserversink gstucxserversink.c:374:do_server_sink_work: failed to send with status Connection reset by remote peer batch_id 0

./deepstream-ucx-test-app -t 1 -c 192.168.168.67 4000 1920 1080 output.mp4
Now saving stream to: output.mp4
The UCX receiver gives :

Now saving stream to: output.mp4
Failed to query video capabilities: Invalid argument
0:00:00.011801572 7352 0x6211df8f0fa0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:h264-hw-enc:sink Unable to try format: Unknown error -1
0:00:00.011811942 7352 0x6211df8f0fa0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:h264-hw-enc:sink Could not probe minimum capture size for pixelformat YM12
0:00:00.011815299 7352 0x6211df8f0fa0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:h264-hw-enc:sink Unable to try format: Unknown error -1
0:00:00.011818879 7352 0x6211df8f0fa0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:h264-hw-enc:sink Could not probe maximum capture size for pixelformat YM12
0:00:00.011824326 7352 0x6211df8f0fa0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:h264-hw-enc:sink Unable to try format: Unknown error -1
0:00:00.011828279 7352 0x6211df8f0fa0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:h264-hw-enc:sink Could not probe minimum capture size for pixelformat Y444
0:00:00.011833402 7352 0x6211df8f0fa0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:h264-hw-enc:sink Unable to try format: Unknown error -1
0:00:00.011835314 7352 0x6211df8f0fa0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:h264-hw-enc:sink Could not probe maximum capture size for pixelformat Y444
0:00:00.011840446 7352 0x6211df8f0fa0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:h264-hw-enc:sink Unable to try format: Unknown error -1
0:00:00.011843922 7352 0x6211df8f0fa0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:h264-hw-enc:sink Could not probe minimum capture size for pixelformat P410
0:00:00.011845624 7352 0x6211df8f0fa0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:h264-hw-enc:sink Unable to try format: Unknown error -1
0:00:00.011849392 7352 0x6211df8f0fa0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:h264-hw-enc:sink Could not probe maximum capture size for pixelformat P410
0:00:00.011854560 7352 0x6211df8f0fa0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:h264-hw-enc:sink Unable to try format: Unknown error -1
0:00:00.011857775 7352 0x6211df8f0fa0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:h264-hw-enc:sink Could not probe minimum capture size for pixelformat PM10
0:00:00.011859582 7352 0x6211df8f0fa0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:h264-hw-enc:sink Unable to try format: Unknown error -1
0:00:00.011862777 7352 0x6211df8f0fa0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:h264-hw-enc:sink Could not probe maximum capture size for pixelformat PM10
0:00:00.011867597 7352 0x6211df8f0fa0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:h264-hw-enc:sink Unable to try format: Unknown error -1
0:00:00.011870487 7352 0x6211df8f0fa0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:h264-hw-enc:sink Could not probe minimum capture size for pixelformat NM12
0:00:00.011872101 7352 0x6211df8f0fa0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:h264-hw-enc:sink Unable to try format: Unknown error -1
0:00:00.011874175 7352 0x6211df8f0fa0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:h264-hw-enc:sink Could not probe maximum capture size for pixelformat NM12
0:00:00.011900140 7352 0x6211df8f0fa0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:h264-hw-enc:src Unable to try format: Unknown error -1
0:00:00.011902616 7352 0x6211df8f0fa0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:h264-hw-enc:src Could not probe minimum capture size for pixelformat H264
0:00:00.011904684 7352 0x6211df8f0fa0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:h264-hw-enc:src Unable to try format: Unknown error -1
0:00:00.011906931 7352 0x6211df8f0fa0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:h264-hw-enc:src Could not probe maximum capture size for pixelformat H264
[1735055797.003609] [H4Mrd3:7352 :0] ucp_context.c:1774 UCX WARN UCP version is incompatible, required: 1.13, actual: 1.12 (release 1)
0:00:00.199035452 7352 0x7731b8001070 FIXME default gstutils.c:4025:gst_pad_create_stream_id_internal:ucxclientsrc:src Creating random stream-id, consider implementing a deterministic way of creating a stream-id
Running…
0:00:00.467738271 7352 0x7731b8001300 ERROR ucxcommon gstucx.c:736:request_finalize: unable to process UCX message (Connection reset by remote peer)
0:00:00.485796556 7352 0x7731b8000df0 WARN v4l2bufferpool gstv4l2bufferpool.c:1130:gst_v4l2_buffer_pool_start:h264-hw-enc:pool:src Uncertain or not enough buffers, enabling copy threshold
0:00:00.485885866 7352 0x7731b8000b70 FIXME basesink gstbasesink.c:3395:gst_base_sink_default_event: stream-start event without group-id. Consider implementing group-id handling in the upstream elements
0:00:00.486023589 7352 0x7731b8000b70 FIXME aggregator gstaggregator.c:1410:gst_aggregator_aggregate_func: Subclass should call gst_aggregator_selected_samples() from its aggregate implementation.
End of stream
Returned, stopping playback
Deleting pipeline

7

The height should be in front of the width. Please read the /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-ucx-test/README carefully.

deepstream-ucx-test-app -t 1 -s 192.168.168.67 4000 1080 1920 file:////opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4

./deepstream-ucx-test-app -t 1 -c 192.168.168.67 4000 1080 1920 output.mp4
8

I did read the README carefully. I copied and pasted the sample exactly. The example in the README has the height and width reversed. Please correct the README example in the next version. Here is the text from the README:

Example:
Server:
$ ./deepstream-ucx-test-app -t 1 -s 192.168.100.1 4000 1920 1080
file:////opt/nvidia/deepstream/deepstream/samples/streams/
sample_1080p_h264.mp4

Client:
$ ./deepstream-ucx-test-app -t 1 -c 192.168.100.1 4000 1920 1080
output.mp4

@H4Mrd3:/opt/nvidia/deepstream/deepstream-7.1/sources/apps/sample_apps/deepstream-ucx-test$ export GST_DEBUG=3
bill@H4Mrd3:/opt/nvidia/deepstream/deepstream-7.1/sources/apps/sample_apps/deepstream-ucx-test$ deepstream-ucx-test-app -t 1 -s 192.168.168.67 4000 1080 1920 file:////opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4
Using URI: file:////opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4
Server listening on: 192.168.168.67 : 4000
[1735091280.609904] [H4Mrd3:7051 :0] ucp_context.c:1774 UCX WARN UCP version is incompatible, required: 1.13, actual: 1.12 (release 1)
0:00:29.335543609 7051 0x57b00f3d1d70 WARN basesrc gstbasesrc.c:3688:gst_base_src_start_complete: pad not activated yet
Decodebin child added: source
Decodebin child added: decodebin0
0:00:29.335686948 7051 0x57b00f3d1d70 WARN basesrc gstbasesrc.c:3688:gst_base_src_start_complete: pad not activated yet
Running…
Decodebin child added: qtdemux0
0:00:29.339180468 7051 0x739b4c001050 WARN qtdemux qtdemux.c:3121:qtdemux_parse_trex: failed to find fragment defaults for stream 1
0:00:29.339247602 7051 0x739b4c001050 WARN qtdemux qtdemux.c:3121:qtdemux_parse_trex: failed to find fragment defaults for stream 2
Decodebin child added: multiqueue0
Decodebin child added: h264parse0
Decodebin child added: capsfilter0
Decodebin child added: aacparse0
Decodebin child added: avdec_aac0
Decodebin child added: nvv4l2decoder0
Failed to query video capabilities: Invalid argument
0:00:29.358467485 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:29.358474999 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat MJPG
0:00:29.358478328 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:29.358481253 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat MJPG
0:00:29.358489505 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:29.358492532 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat AV10
0:00:29.358495002 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:29.358498133 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat AV10
0:00:29.358503750 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:29.358506671 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat DVX5
0:00:29.358509109 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:29.358511852 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat DVX5
0:00:29.358516061 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:29.358518756 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat DVX4
0:00:29.358521326 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:29.358525003 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat DVX4
0:00:29.358529305 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:29.358532522 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat MPG4
0:00:29.358534990 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:29.358537719 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat MPG4
0:00:29.358541732 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:29.358544545 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat MPG2
0:00:29.358547017 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:29.358549964 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat MPG2
0:00:29.358554409 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:29.358557344 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat H265
0:00:29.358559930 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:29.358562845 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat H265
0:00:29.358567111 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:29.358570034 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat VP90
0:00:29.358572388 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:29.358575189 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat VP90
0:00:29.358579050 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:29.358581781 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat VP80
0:00:29.358584278 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:29.358587145 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat VP80
0:00:29.358591706 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:29.358595057 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat H264
0:00:29.358597502 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:29.358600146 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat H264
0:00:29.358765742 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:src Unable to try format: Unknown error -1
0:00:29.358772437 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:src Could not probe minimum capture size for pixelformat Y444
0:00:29.358774953 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:src Unable to try format: Unknown error -1
0:00:29.358777867 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:src Could not probe maximum capture size for pixelformat Y444
0:00:29.358783114 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:src Unable to try format: Unknown error -1
0:00:29.358786061 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:src Could not probe minimum capture size for pixelformat P410
0:00:29.358788710 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:src Unable to try format: Unknown error -1
0:00:29.358791465 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:src Could not probe maximum capture size for pixelformat P410
0:00:29.358796216 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:src Unable to try format: Unknown error -1
0:00:29.358799193 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:src Could not probe minimum capture size for pixelformat PM10
0:00:29.358801578 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:src Unable to try format: Unknown error -1
0:00:29.358804382 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:src Could not probe maximum capture size for pixelformat PM10
0:00:29.358808909 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:src Unable to try format: Unknown error -1
0:00:29.358811854 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:src Could not probe minimum capture size for pixelformat NM12
0:00:29.358814401 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:src Unable to try format: Unknown error -1
0:00:29.358817200 7051 0x739b4c0012c0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:src Could not probe maximum capture size for pixelformat NM12
In cb_newpad
In cb_newpad
0:00:29.593350339 7051 0x739b4c0012c0 WARN v4l2videodec gstv4l2videodec.c:2297:gst_v4l2_video_dec_decide_allocation: Duration invalid, not setting latency
0:00:29.593363635 7051 0x739b4c0012c0 WARN v4l2bufferpool gstv4l2bufferpool.c:1130:gst_v4l2_buffer_pool_start:nvv4l2decoder0:pool:src Uncertain or not enough buffers, enabling copy threshold
0:00:29.593901678 7051 0x739b4c0017a0 WARN v4l2bufferpool gstv4l2bufferpool.c:1607:gst_v4l2_buffer_pool_dqbuf:nvv4l2decoder0:pool:src Driver should never set v4l2_buffer.field to ANY
[1735091310.246066] [H4Mrd3:7051 :1] ib_md.c:349 UCX ERROR ibv_reg_mr(address=0x739ae4000000, length=4423680, access=0xf) failed: Bad address
[1735091310.246077] [H4Mrd3:7051 :1] ucp_mm.c:158 UCX ERROR failed to register address 0x739ae4000000 mem_type bit 0x1 length 4423680 on md[4]=mlx5_0: Input/output error (md reg_mem_types 0x1)
[1735091310.246080] [H4Mrd3:7051 :1] ucp_request.c:498 UCX ERROR failed to register user buffer datatype 0x8 address 0x739ae4000000 len 4423680: Input/output error
[H4Mrd3:7051 :1:7088] rndv.c:2072 Assertion `status == UCS_OK’ failed
==== backtrace (tid: 7088) ====
0 /lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2e4) [0x739b783a4fc4]
1 /lib/x86_64-linux-gnu/libucs.so.0(ucs_fatal_error_message+0xb6) [0x739b783a5176]
2 /lib/x86_64-linux-gnu/libucs.so.0(ucs_fatal_error_format+0x11a) [0x739b783a529a]
3 /lib/x86_64-linux-gnu/libucp.so.0(ucp_rndv_rtr_handler+0xb0b) [0x739b7831423b]
4 /lib/x86_64-linux-gnu/ucx/libuct_ib.so.0(+0x3e5a3) [0x739b781b25a3]
5 /lib/x86_64-linux-gnu/libucp.so.0(ucp_worker_progress+0x7a) [0x739b782dc28a]
6 /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_ucx.so(_Z12gst_ucp_sendP10ucp_workerP6ucp_ep14GstNvDsUcxAmIdPvmS4_m+0x98) [0x739b7a216d58]
7 /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_ucx.so(+0xffb1) [0x739b7a21bfb1]
8 /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_ucx.so(+0x134c3) [0x739b7a21f4c3]
9 /lib/x86_64-linux-gnu/libglib-2.0.so.0(+0x8a384) [0x739b7dd62384]
10 /lib/x86_64-linux-gnu/libglib-2.0.so.0(+0x89ac1) [0x739b7dd61ac1]
11 /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x739b7d694ac3]
12 /lib/x86_64-linux-gnu/libc.so.6(+0x126850) [0x739b7d726850]

Aborted (core dumped)
bill@H4Mrd3:/opt/nvidia/deepstream/deepstream-7.1/sources/apps/sample_apps/deepstream-ucx-test$

bill@H4Mrd3:/opt/nvidia/deepstream/deepstream-7.1/sources/apps/sample_apps/deepstream-ucx-test$ export GST_DEBUG=3
bill@H4Mrd3:/opt/nvidia/deepstream/deepstream-7.1/sources/apps/sample_apps/deepstream-ucx-test$ ./deepstream-ucx-test-app -t 1 -c 192.168.168.67 4000 1080 1920 output.mp4
Now saving stream to: output.mp4
Failed to query video capabilities: Invalid argument
0:00:00.011702886 7063 0x5ab68f6a7fc0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:h264-hw-enc:sink Unable to try format: Unknown error -1
0:00:00.011719419 7063 0x5ab68f6a7fc0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:h264-hw-enc:sink Could not probe minimum capture size for pixelformat YM12
0:00:00.011721966 7063 0x5ab68f6a7fc0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:h264-hw-enc:sink Unable to try format: Unknown error -1
0:00:00.011724753 7063 0x5ab68f6a7fc0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:h264-hw-enc:sink Could not probe maximum capture size for pixelformat YM12
0:00:00.011730339 7063 0x5ab68f6a7fc0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:h264-hw-enc:sink Unable to try format: Unknown error -1
0:00:00.011733798 7063 0x5ab68f6a7fc0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:h264-hw-enc:sink Could not probe minimum capture size for pixelformat Y444
0:00:00.011735590 7063 0x5ab68f6a7fc0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:h264-hw-enc:sink Unable to try format: Unknown error -1
0:00:00.011737521 7063 0x5ab68f6a7fc0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:h264-hw-enc:sink Could not probe maximum capture size for pixelformat Y444
0:00:00.011741513 7063 0x5ab68f6a7fc0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:h264-hw-enc:sink Unable to try format: Unknown error -1
0:00:00.011745214 7063 0x5ab68f6a7fc0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:h264-hw-enc:sink Could not probe minimum capture size for pixelformat P410
0:00:00.011749116 7063 0x5ab68f6a7fc0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:h264-hw-enc:sink Unable to try format: Unknown error -1
0:00:00.011751564 7063 0x5ab68f6a7fc0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:h264-hw-enc:sink Could not probe maximum capture size for pixelformat P410
0:00:00.011770624 7063 0x5ab68f6a7fc0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:h264-hw-enc:sink Unable to try format: Unknown error -1
0:00:00.011773372 7063 0x5ab68f6a7fc0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:h264-hw-enc:sink Could not probe minimum capture size for pixelformat PM10
0:00:00.011776531 7063 0x5ab68f6a7fc0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:h264-hw-enc:sink Unable to try format: Unknown error -1
0:00:00.011778687 7063 0x5ab68f6a7fc0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:h264-hw-enc:sink Could not probe maximum capture size for pixelformat PM10
0:00:00.011783131 7063 0x5ab68f6a7fc0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:h264-hw-enc:sink Unable to try format: Unknown error -1
0:00:00.011786399 7063 0x5ab68f6a7fc0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:h264-hw-enc:sink Could not probe minimum capture size for pixelformat NM12
0:00:00.011788393 7063 0x5ab68f6a7fc0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:h264-hw-enc:sink Unable to try format: Unknown error -1
0:00:00.011790558 7063 0x5ab68f6a7fc0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:h264-hw-enc:sink Could not probe maximum capture size for pixelformat NM12
0:00:00.011812392 7063 0x5ab68f6a7fc0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:h264-hw-enc:src Unable to try format: Unknown error -1
0:00:00.011814842 7063 0x5ab68f6a7fc0 WARN v4l2 gstv4l2object.c:2993:gst_v4l2_object_probe_caps_for_format:h264-hw-enc:src Could not probe minimum capture size for pixelformat H264
0:00:00.011816965 7063 0x5ab68f6a7fc0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:h264-hw-enc:src Unable to try format: Unknown error -1
0:00:00.011819025 7063 0x5ab68f6a7fc0 WARN v4l2 gstv4l2object.c:2999:gst_v4l2_object_probe_caps_for_format:h264-hw-enc:src Could not probe maximum capture size for pixelformat H264
[1735091309.745556] [H4Mrd3:7063 :0] ucp_context.c:1774 UCX WARN UCP version is incompatible, required: 1.13, actual: 1.12 (release 1)
Running…
0:00:00.201716768 7063 0x7b5cd4001070 FIXME default gstutils.c:4025:gst_pad_create_stream_id_internal:ucxclientsrc:src Creating random stream-id, consider implementing a deterministic way of creating a stream-id
0:00:00.492002700 7063 0x7b5cd4000df0 WARN v4l2bufferpool gstv4l2bufferpool.c:1130:gst_v4l2_buffer_pool_start:h264-hw-enc:pool:src Uncertain or not enough buffers, enabling copy threshold
0:00:04.718701644 7063 0x7b5cd4001300 ERROR ucxcommon gstucx.c:736:request_finalize: unable to process UCX message (Connection reset by remote peer)
0:00:04.718893189 7063 0x7b5cd4000b70 FIXME basesink gstbasesink.c:3395:gst_base_sink_default_event: stream-start event without group-id. Consider implementing group-id handling in the upstream elements
0:00:04.719073109 7063 0x7b5cd4000b70 FIXME aggregator gstaggregator.c:1410:gst_aggregator_aggregate_func: Subclass should call gst_aggregator_selected_samples() from its aggregate implementation.
End of stream
Returned, stopping playback
Deleting pipeline
bill@H4Mrd3:/opt/nvidia/deepstream/deepstream-7.1/sources/apps/sample_apps/deepstream-ucx-test$

Please fix your README.

Please fix this bug or let me know what I am doing wrong.

9

The UCX failed with some kernel interface. Please check with the Introduction - NVIDIA Docs and Adapter Software

10

Thanks. I’ll provide more info.

11

UCX is working fine:
(make browser full screen to see data properly)

bill@H4Mrd3:~/Downloads/ucx/src/tools/perf$ ucx_perftest -c 0
Waiting for connection…
Accepted connection from 10.0.0.136:59706
±---------------------------------------------------------------------------------------------------------+
| API: protocol layer |
| Test: tag match latency |
| Data layout: (automatic) |
| Send memory: host |
| Recv memory: host |
| Message size: 8 |
| Window size: 1 |
±---------------------------------------------------------------------------------------------------------+

bill@H4Mrd3:~/Downloads/ucx/src/tools/perf$ ucx_perftest 10.0.0.136 -t tag_lat -c 1
+--------------+--------------+------------------------------+---------------------+-----------------------+
| | | latency (usec) | bandwidth (MB/s) | message rate (msg/s) |
+--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
| Stage | # iterations | 50.0%ile | average | overall | average | overall | average | overall |
+--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
Final: 1000000 0.082 0.083 0.083 91.47 91.47 11989395 11989395

12

The problem is in gstreamer:

deepstream-ucx-test-app -t 1 -s 192.168.168.67 4000 1080 1920 file:////opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4
Using URI: file:////opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4
Server listening on: 192.168.168.67 : 4000
[1735151542.023666] [H4Mrd3:6835 :0] ucp_context.c:1774 UCX WARN UCP version is incompatible, required: 1.13, actual: 1.12 (release 1)
Decodebin child added: source
Decodebin child added: decodebin0
Running…
Decodebin child added: qtdemux0
Decodebin child added: multiqueue0
Decodebin child added: h264parse0
Decodebin child added: capsfilter0
Decodebin child added: aacparse0
Decodebin child added: avdec_aac0
Decodebin child added: nvv4l2decoder0
Failed to query video capabilities: Invalid argument

bill@H4Mrd3:/opt/nvidia/deepstream/deepstream-7.1/sources/apps/sample_apps/deepstream-ucx-test$ ./deepstream-ucx-test-app -t 1 -c 192.168.168.67 4000 1080 1920 output.mp4
Now saving stream to: output.mp4
Failed to query video capabilities: Invalid argument
[1735151578.379731] [H4Mrd3:6881 :0] ucp_context.c:1774 UCX WARN UCP version is incompatible, required: 1.13, actual: 1.12 (release 1)
Running…

My understanding is that UCX needs to pass and get “caps”

13

DS_plugin_gst-nvdsucx

This works and it is trying to play. It sees the receiver. The receiver starts but the receiver doesn’t see caps so it quits. The sender hangs because the receiver never starts receiving data without caps.

gst-launch-1.0 uridecodebin uri=“file:////opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4” ! decodebin name=dec ! queue ! nvvideoconvert ! ‘video/x-raw(memory:NVMM),format=NV12,width=1920,height=1080’ ! nvdsucxserversink addr=192.168.168.67 port=4000 buf-type=nvdsucx-buf-video
Setting pipeline to PAUSED …
[1735184184.170197] [H4Mrd3:33460:0] ucp_context.c:1774 UCX WARN UCP version is incompatible, required: 1.13, actual: 1.12 (release 1)
Pipeline is PREROLLING …
Failed to query video capabilities: Invalid argument
Pipeline is PREROLLED …
Setting pipeline to PLAYING …
Redistribute latency…
New clock: GstSystemClock
0:00:00.0 / 0:00:48.1 (0.1 %)

Here is the receiver:

gst-launch-1.0 nvdsucxclientsrc addr=192.168.168.67 port=4000 nvbuf-memory-type=2 num-nvbuf=4 buf-type=nvdsucx-buf-video ! nvvideoconvert ! autovideosink

Setting pipeline to PAUSED …
[1735184189.976282] [H4Mrd3:33474:0] ucp_context.c:1774 UCX WARN UCP version is incompatible, required: 1.13, actual: 1.12 (release 1)
Pipeline is PREROLLING …

(gst-launch-1.0:33474): GStreamer-CRITICAL **: 19:36:30.152: gst_buffer_pool_config_set_params: assertion ‘caps == NULL || gst_caps_is_fixed (caps)’ failed
ERROR: from element /GstPipeline:pipeline0/GstUcxClientSrc:ucxclientsrc0: Internal data stream error.
Additional debug info:
…/libs/gst/base/gstbasesrc.c(3127): gst_base_src_loop (): /GstPipeline:pipeline0/GstUcxClientSrc:ucxclientsrc0:
streaming stopped, reason not-negotiated (-4)
ERROR: pipeline doesn’t want to preroll.
Setting pipeline to NULL …

(gst-launch-1.0:33474): GStreamer-CRITICAL **: 19:36:30.207: gst_buffer_pool_set_active: assertion ‘GST_IS_BUFFER_POOL (pool)’ failed

(gst-launch-1.0:33474): GStreamer-CRITICAL **: 19:36:30.207: gst_object_unref: assertion ‘((GObject *) object)->ref_count > 0’ failed
Freeing pipeline …

This dumps core.
gst-launch-1.0 nvdsucxclientsrc addr=192.168.168.67 port=4000 nvbuf-memory-type=2 num-nvbuf=4 buf-type=nvdsucx-buf-video ! ‘video/x-raw(memory:NVMM),format=NV12,width=1920,height=1080,framerate=30/1’ ! queue ! nvvideoconvert ! nvv4l2h264enc ! h264parse ! qtmux name=mux_0 ! filesink sync=1 async=0 qos=0 location=./out_1080p.mp4

14

RDMA is working fine

image
image1083×828 125 KB

image
image1038×749 120 KB

15

If I play the file it plays fine:

gst-launch-1.0 uridecodebin uri=“file:////opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4” ! decodebin name=dec ! queue ! nvvideoconvert ! autovideosink

If I send the file with nvdsucxserversink it fails when I start the receiver:

gst-launch-1.0 uridecodebin uri=“file:///testing/sandbox/originalnv12a.mp4” ! decodebin name=dec ! queue ! nvvideoconvert ! nvdsucxserversink addr=192.168.168.67 port=4000 buf-type=nvdsucx-buf-video
Setting pipeline to PAUSED …
[1735249462.103601] [H4Mrd3:57052:0] ucp_context.c:1774 UCX WARN UCP version is incompatible, required: 1.13, actual: 1.12 (release 1)
0:00:04.696629821 57052 0x6535abe402d0 WARN basesrc gstbasesrc.c:3688:gst_base_src_start_complete: pad not activated yet
0:00:04.696763886 57052 0x6535abe402d0 WARN basesrc gstbasesrc.c:3688:gst_base_src_start_complete: pad not activated yet
Pipeline is PREROLLING …
0:00:04.699789551 57052 0x74e5cc001070 WARN qtdemux qtdemux.c:3121:qtdemux_parse_trex: failed to find fragment defaults for stream 1
Failed to query video capabilities: Invalid argument
0:00:04.719342106 57052 0x74e5cc0012e0 WARN v4l2 gstv4l2object.c:3108:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1

The receiver doesn’t get “caps”

gst-launch-1.0 nvdsucxclientsrc addr=192.168.168.67 port=4000 nvbuf-memory-type=2 num-nvbuf=4 buf-type=nvdsucx-buf-video ! nvvideoconvert ! autovideosink
Setting pipeline to PAUSED …
[1735249466.597234] [H4Mrd3:57058:0] ucp_context.c:1774 UCX WARN UCP version is incompatible, required: 1.13, actual: 1.12 (release 1)
Pipeline is PREROLLING …
0:00:00.192728216 57058 0x7105ac000b70 FIXME default gstutils.c:4025:gst_pad_create_stream_id_internal:ucxclientsrc0:src Creating random stream-id, consider implementing a deterministic way of creating a stream-id

(gst-launch-1.0:57058): GStreamer-CRITICAL : 13:44:26.775: gst_buffer_pool_config_set_params: assertion ‘caps == NULL || gst_caps_is_fixed (caps)’ failed
0:00:00.192978879 57058 0x7105ac000b70 WARN nvdspool gstnvdsbufferpool.c:529:gst_nvds_buffer_pool_set_config: no caps in config
0:00:00.192987125 57058 0x7105ac000b70 ERROR ucxcommon gstucx.c:2634:ucx_src_video_buffer_pool_new: bufferpool configuration failed
0:00:00.193013868 57058 0x7105ac000b70 WARN basesrc gstbasesrc.c:3127:gst_base_src_loop: error: Internal data stream error.
0:00:00.193019359 57058 0x7105ac000b70 WARN basesrc gstbasesrc.c:3127:gst_base_src_loop: error: streaming stopped, reason not-negotiated (-4)
ERROR: from element /GstPipeline:pipeline0/GstUcxClientSrc:ucxclientsrc0: Internal data stream error.

16

I figured out the answer. It is a gstreamer problem. gstreamer is very particular about everything. The mp4 files play fine with ffmpeg and gstreamer but deepstream-ucx-test-app and nvdsucxclientsrc and nvdsucxclientsrc are very sensitive to format and metadata. They want a clean and tidy stream with nothing added. I had to filter the incoming stream to make it clean and tidy.

Apparently MainConcept adds metadata to the stream which causes gstreamer to barf. I had to strip that out by using gstreamer to re-encode with the generic VideoHandler encoder rather than MainConcept.

gst-launch-1.0 uridecodebin uri="file:///testing/sandbox/original.mp4" ! decodebin name=dec ! queue ! nvvideoconvert ! nvv4l2h264enc ! h264parse ! qtmux name=mux_0 ! filesink sync=1 async=0 qos=0 location=original_1080p.mp4

image
image1191×767 132 KB

This one works.

gst-launch-1.0 uridecodebin uri="file:////opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4" async-handling=1 name=src1 src1. ! queue ! nvvideoconvert ! 'video/x-raw(memory:NVMM),format=NV12,width=1920,height=1080' ! nvdsucxserversink addr=192.168.168.67 port=4000 buf-type=nvdsucx-buf-video

new Terminal

gst-launch-1.0 nvdsucxclientsrc addr=192.168.168.67 port=4000 nvbuf-memory-type=2 num-nvbuf=4 buf-type=nvdsucx-buf-video ! 'video/x-raw(memory:NVMM),format=NV12,width=1920,height=1080,framerate=30/1' ! queue ! nvvideoconvert ! nvv4l2h264enc ! h264parse ! qtmux name=mux_0 ! filesink sync=1 async=0 qos=0 location=~/out_1080p.mp4

another

gst-launch-1.0 uridecodebin uri="file:////testing/sandbox/original_1080p.mp4" async-handling=1 name=src1 src1. ! queue ! nvvideoconvert ! 'video/x-raw(memory:NVMM),format=NV12,width=1920,height=1080' ! nvdsucxserversink addr=192.168.168.67 port=4000 buf-type=nvdsucx-buf-video

gst-launch-1.0 nvdsucxclientsrc addr=192.168.168.67 port=4000 nvbuf-memory-type=2 num-nvbuf=4 buf-type=nvdsucx-buf-video ! 'video/x-raw(memory:NVMM),format=NV12,width=1920,height=1080,framerate=60000/1001' ! queue ! nvvideoconvert ! nvv4l2h264enc ! h264parse ! qtmux name=mux_0 ! filesink sync=1 async=0 qos=0 location=~/out_1080p.mp4

UCX build and install

Perf Test Package

Installing MLNX_OFED with RDMA

@gpolaillon @hsinha @garethsb @pravins @sseitz

17

Today those don’t work after reboot.

RDMA fails in a container saying that the resource is in use.

ERROR ucxcommon gstucx.c:1768:start_server_listener: failed to listen (Device is busy)

basesink gstbasesink.c:5878:gst_base_sink_change_state:<ucxserversink0> error: Failed to start

ERROR: from element /GstPipeline:pipeline0/GstUcxServerSink:ucxserversink0: GStreamer error: state change failed and some element failed to post a proper error message with the reason for the failure.

Additional debug info:
../libs/gst/base/gstbasesink.c(5878): gst_base_sink_change_state (): /GstPipeline:pipeline0/GstUcxServerSink:ucxserversink0:
Failed to start

How do I get/make a usable IP address inside a container?

18

Hi, @bill58

What is your docker container run command? Have you added “–network=host” option in you docker run command?

19

sudo mst start

sudo docker run -v /run/docker/plugins:/run/docker/plugins -v /etc/docker:/etc/docker -v /var/run:/var/run --net=host --cap-add CAP_SYS_PTRACE --shm-size="8g" --device=/dev/infiniband/uverbs0 --device=/dev/infiniband/rdma_cm --ipc=host -e CUDA_CACHE_DISABLE=0 -v /dev/infiniband:/dev/infiniband --privileged rdma/sriov-plugin

sudo docker network create -d sriov --subnet=194.168.1.0/24 -o netdevice=enp2s0f0np0 mode=passthrough mynet

sudo docker run --rm -it -v /dev/snd:/dev/snd --env ALSA_CARD=PCH --privileged --gpus all --net=mynet --runtime nvidia -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix -v /var/tnp/gxf/.cache/gxf_registry/:/var/tmp/gxf/.cache/gxf_registry/ --entrypoint bash nvcr.io/azqptjkaaotv/holoscan-for-media/nvds-nmos-bin:0.2.0

Whatever I do I get this error when running ucxserversink :

ERROR ucxcommon gstucx.c:1768:start_server_listener: failed to listen (Device is busy)

basesink gstbasesink.c:5878:gst_base_sink_change_state:<ucxserversink0> error: Failed to start

Thanks for your reply.

It’s the same using nvcr.io/nvidia/deepstream:7.1-gc-triton-devel

image
image682×207 34 KB

image
image1850×188 72.9 KB

image
image1853×245 52.1 KB

20

The containers you used are not provided by us. Can you run with DeepStream docker container?

21

I’m using Nvidia provided containers from the Nvidia Holoscan for Media developer nvcr.io repository.

It’s the same using nvcr.io/nvidia/deepstream:7.1-gc-triton-devel

As I stated above.