Hi,
We’re trying to use MicroK8s (Also tried K3s) to run a single node (i.e. one laptop) deployment of our full offering for demo purposes.
We’re unable to access our local IP cameras from inside the Kubernetes pod. This is probably a pure K8s issue as it works outside the pod, both on the machine and inside a normal docker container.
We’ve not been able to connect via gstreamer (using even the most basic pipelines without any deepstream elements), nor via ffmpeg or using OpenCV capture.
We are able to access over TCP, and the initial negotiation in gstreamer seems to work.
This is the Gstreamer error:
“Could not receive any UDP packets for 5.0000 seconds, maybe your firewall is blocking it. No other protocols to try.”
If we allow it to continue over TCP it works.
The machine used is a fresh machine with Ubuntu 20.04, and Microk8s with channel=1.19 and and k3s 1.20. Cameras are Samsung Wisenet type cameras, fairly new (1-2 years maybe). Runtime is nvidia docker runtime.
Tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
13:37:00.235659 IP 10.42.0.116.45232 > 192.168.17.21.rtsp: Flags [S], seq 2716624008, win 64860, options [mss 1410,sackOK,TS val 1662877534 ecr 0,nop,wscale 7], length 0
13:37:00.235669 IP 10.42.0.116.45232 > 192.168.17.21.rtsp: Flags [S], seq 2716624008, win 64860, options [mss 1410,sackOK,TS val 1662877534 ecr 0,nop,wscale 7], length 0
13:37:00.236527 IP 192.168.17.21.rtsp > 10.42.0.116.45232: Flags [S.], seq 4286557816, ack 2716624009, win 14480, options [mss 1460,sackOK,TS val 30680063 ecr 1662877534,nop,wscale 5], length 0
13:37:00.236529 IP 192.168.17.21.rtsp > 10.42.0.116.45232: Flags [S.], seq 4286557816, ack 2716624009, win 14480, options [mss 1460,sackOK,TS val 30680063 ecr 1662877534,nop,wscale 5], length 0
13:37:00.236537 IP 10.42.0.116.45232 > 192.168.17.21.rtsp: Flags [.], ack 1, win 507, options [nop,nop,TS val 1662877534 ecr 30680063], length 0
13:37:00.236539 IP 10.42.0.116.45232 > 192.168.17.21.rtsp: Flags [.], ack 1, win 507, options [nop,nop,TS val 1662877534 ecr 30680063], length 0
13:37:00.236624 IP 10.42.0.116.45232 > 192.168.17.21.rtsp: Flags [P.], seq 1:465, ack 1, win 507, options [nop,nop,TS val 1662877534 ecr 30680063], length 464: RTSP: OPTIONS rtsp://192.168.17.21:554/profile3/media.smp RTSP/1.0
13:37:00.236626 IP 10.42.0.116.45232 > 192.168.17.21.rtsp: Flags [P.], seq 1:465, ack 1, win 507, options [nop,nop,TS val 1662877534 ecr 30680063], length 464: RTSP: OPTIONS rtsp://192.168.17.21:554/profile3/media.smp RTSP/1.0
13:37:00.237450 IP 192.168.17.21.rtsp > 10.42.0.116.45232: Flags [.], ack 465, win 486, options [nop,nop,TS val 30680063 ecr 1662877534], length 0
13:37:00.237451 IP 192.168.17.21.rtsp > 10.42.0.116.45232: Flags [.], ack 465, win 486, options [nop,nop,TS val 30680063 ecr 1662877534], length 0
13:37:00.245650 IP 192.168.17.21.rtsp > 10.42.0.116.45232: Flags [P.], seq 1:231, ack 465, win 486, options [nop,nop,TS val 30680064 ecr 1662877534], length 230: RTSP: RTSP/1.0 401 Unauthorized
13:37:00.245654 IP 192.168.17.21.rtsp > 10.42.0.116.45232: Flags [P.], seq 1:231, ack 465, win 486, options [nop,nop,TS val 30680064 ecr 1662877534], length 230: RTSP: RTSP/1.0 401 Unauthorized
13:37:00.245688 IP 10.42.0.116.45232 > 192.168.17.21.rtsp: Flags [.], ack 231, win 506, options [nop,nop,TS val 1662877544 ecr 30680064], length 0
13:37:00.245694 IP 10.42.0.116.45232 > 192.168.17.21.rtsp: Flags [.], ack 231, win 506, options [nop,nop,TS val 1662877544 ecr 30680064], length 0
13:37:00.246101 IP 10.42.0.116.45232 > 192.168.17.21.rtsp: Flags [P.], seq 465:1447, ack 231, win 506, options [nop,nop,TS val 1662877544 ecr 30680064], length 982: RTSP: OPTIONS rtsp://192.168.17.21:554/profile3/media.smp RTSP/1.0
13:37:00.246111 IP 10.42.0.116.45232 > 192.168.17.21.rtsp: Flags [P.], seq 465:1447, ack 231, win 506, options [nop,nop,TS val 1662877544 ecr 30680064], length 982: RTSP: OPTIONS rtsp://192.168.17.21:554/profile3/media.smp RTSP/1.0
13:37:00.248338 IP 192.168.17.21.rtsp > 10.42.0.116.45232: Flags [P.], seq 231:455, ack 1447, win 548, options [nop,nop,TS val 30680064 ecr 1662877544], length 224: RTSP: RTSP/1.0 200 OK
13:37:00.248342 IP 192.168.17.21.rtsp > 10.42.0.116.45232: Flags [P.], seq 231:455, ack 1447, win 548, options [nop,nop,TS val 30680064 ecr 1662877544], length 224: RTSP: RTSP/1.0 200 OK
13:37:00.248368 IP 10.42.0.116.45232 > 192.168.17.21.rtsp: Flags [.], ack 455, win 505, options [nop,nop,TS val 1662877546 ecr 30680064], length 0
13:37:00.248373 IP 10.42.0.116.45232 > 192.168.17.21.rtsp: Flags [.], ack 455, win 505, options [nop,nop,TS val 1662877546 ecr 30680064], length 0
13:37:00.248800 IP 10.42.0.116.45232 > 192.168.17.21.rtsp: Flags [P.], seq 1447:1807, ack 455, win 505, options [nop,nop,TS val 1662877547 ecr 30680064], length 360: RTSP: DESCRIBE rtsp://192.168.17.21:554/profile3/media.smp RTSP/1.0
13:37:00.248810 IP 10.42.0.116.45232 > 192.168.17.21.rtsp: Flags [P.], seq 1447:1807, ack 455, win 505, options [nop,nop,TS val 1662877547 ecr 30680064], length 360: RTSP: DESCRIBE rtsp://192.168.17.21:554/profile3/media.smp RTSP/1.0
13:37:00.285088 IP 192.168.17.21.rtsp > 10.42.0.116.45232: Flags [.], ack 1807, win 609, options [nop,nop,TS val 30680068 ecr 1662877547], length 0
13:37:00.285094 IP 192.168.17.21.rtsp > 10.42.0.116.45232: Flags [.], ack 1807, win 609, options [nop,nop,TS val 30680068 ecr 1662877547], length 0
13:37:00.320691 IP 192.168.17.21.rtsp > 10.42.0.116.45232: Flags [P.], seq 455:1428, ack 1807, win 609, options [nop,nop,TS val 30680071 ecr 1662877547], length 973: RTSP: RTSP/1.0 200 OK
13:37:00.320697 IP 192.168.17.21.rtsp > 10.42.0.116.45232: Flags [P.], seq 455:1428, ack 1807, win 609, options [nop,nop,TS val 30680071 ecr 1662877547], length 973: RTSP: RTSP/1.0 200 OK
13:37:00.320730 IP 10.42.0.116.45232 > 192.168.17.21.rtsp: Flags [.], ack 1428, win 502, options [nop,nop,TS val 1662877619 ecr 30680071], length 0
13:37:00.320737 IP 10.42.0.116.45232 > 192.168.17.21.rtsp: Flags [.], ack 1428, win 502, options [nop,nop,TS val 1662877619 ecr 30680071], length 0
13:37:00.323652 IP 10.42.0.116.45232 > 192.168.17.21.rtsp: Flags [P.], seq 1807:2211, ack 1428, win 502, options [nop,nop,TS val 1662877622 ecr 30680071], length 404: RTSP: SETUP rtsp://192.168.17.21:554/profile3/media.smp/trackID=v RTSP/1.0
13:37:00.323666 IP 10.42.0.116.45232 > 192.168.17.21.rtsp: Flags [P.], seq 1807:2211, ack 1428, win 502, options [nop,nop,TS val 1662877622 ecr 30680071], length 404: RTSP: SETUP rtsp://192.168.17.21:554/profile3/media.smp/trackID=v RTSP/1.0
13:37:00.324643 IP 192.168.17.21.rtsp > 10.42.0.116.45232: Flags [.], ack 2211, win 671, options [nop,nop,TS val 30680071 ecr 1662877622], length 0
13:37:00.324649 IP 192.168.17.21.rtsp > 10.42.0.116.45232: Flags [.], ack 2211, win 671, options [nop,nop,TS val 30680071 ecr 1662877622], length 0
13:37:00.326298 IP 192.168.17.21.rtsp > 10.42.0.116.45232: Flags [P.], seq 1428:1668, ack 2211, win 671, options [nop,nop,TS val 30680072 ecr 1662877622], length 240: RTSP: RTSP/1.0 200 OK
13:37:00.326303 IP 192.168.17.21.rtsp > 10.42.0.116.45232: Flags [P.], seq 1428:1668, ack 2211, win 671, options [nop,nop,TS val 30680072 ecr 1662877622], length 240: RTSP: RTSP/1.0 200 OK
13:37:00.334129 IP 10.42.0.116.45232 > 192.168.17.21.rtsp: Flags [P.], seq 2211:2631, ack 1668, win 502, options [nop,nop,TS val 1662877632 ecr 30680072], length 420: RTSP: SETUP rtsp://192.168.17.21:554/profile3/media.smp/trackID=m RTSP/1.0
13:37:00.334144 IP 10.42.0.116.45232 > 192.168.17.21.rtsp: Flags [P.], seq 2211:2631, ack 1668, win 502, options [nop,nop,TS val 1662877632 ecr 30680072], length 420: RTSP: SETUP rtsp://192.168.17.21:554/profile3/media.smp/trackID=m RTSP/1.0
13:37:00.346061 IP 192.168.17.21.rtsp > 10.42.0.116.45232: Flags [P.], seq 1668:1908, ack 2631, win 732, options [nop,nop,TS val 30680074 ecr 1662877632], length 240: RTSP: RTSP/1.0 200 OK
13:37:00.346066 IP 192.168.17.21.rtsp > 10.42.0.116.45232: Flags [P.], seq 1668:1908, ack 2631, win 732, options [nop,nop,TS val 30680074 ecr 1662877632], length 240: RTSP: RTSP/1.0 200 OK
13:37:00.351182 IP 10.42.0.116.45232 > 192.168.17.21.rtsp: Flags [P.], seq 2631:2995, ack 1908, win 502, options [nop,nop,TS val 1662877649 ecr 30680074], length 364: RTSP: PLAY rtsp://192.168.17.21:554/profile3/media.smp RTSP/1.0
13:37:00.351198 IP 10.42.0.116.45232 > 192.168.17.21.rtsp: Flags [P.], seq 2631:2995, ack 1908, win 502, options [nop,nop,TS val 1662877649 ecr 30680074], length 364: RTSP: PLAY rtsp://192.168.17.21:554/profile3/media.smp RTSP/1.0
13:37:00.351348 IP 10.42.0.116.32854 > 192.168.17.21.41568: UDP, length 200
13:37:00.351376 IP 10.42.0.116.32854 > 192.168.17.21.41568: UDP, length 200
13:37:00.351656 IP 10.42.0.116.32854 > 192.168.17.21.41568: UDP, length 200
13:37:00.351662 IP 10.42.0.116.32854 > 192.168.17.21.41568: UDP, length 200
13:37:00.351688 IP 10.42.0.116.37670 > 192.168.17.21.41572: UDP, length 200
13:37:00.351715 IP 10.42.0.116.37670 > 192.168.17.21.41572: UDP, length 200
13:37:00.351720 IP 10.42.0.116.32854 > 192.168.17.21.41568: UDP, length 200
13:37:00.351723 IP 10.42.0.116.32854 > 192.168.17.21.41568: UDP, length 200
13:37:00.351769 IP 10.42.0.116.32854 > 192.168.17.21.41568: UDP, length 200
13:37:00.351772 IP 10.42.0.116.32854 > 192.168.17.21.41568: UDP, length 200
13:37:00.351807 IP 10.42.0.116.37670 > 192.168.17.21.41572: UDP, length 200
13:37:00.351813 IP 10.42.0.116.37670 > 192.168.17.21.41572: UDP, length 200
13:37:00.351822 IP 10.42.0.116.32854 > 192.168.17.21.41568: UDP, length 200
13:37:00.351825 IP 10.42.0.116.32854 > 192.168.17.21.41568: UDP, length 200
13:37:00.351868 IP 10.42.0.116.37670 > 192.168.17.21.41572: UDP, length 200
13:37:00.351872 IP 10.42.0.116.37670 > 192.168.17.21.41572: UDP, length 200
13:37:00.351919 IP 10.42.0.116.37670 > 192.168.17.21.41572: UDP, length 200
13:37:00.351922 IP 10.42.0.116.37670 > 192.168.17.21.41572: UDP, length 200
13:37:00.351971 IP 10.42.0.116.37670 > 192.168.17.21.41572: UDP, length 200
13:37:00.351974 IP 10.42.0.116.37670 > 192.168.17.21.41572: UDP, length 200
13:37:00.353914 IP 192.168.17.21.rtsp > 10.42.0.116.45232: Flags [P.], seq 1908:2256, ack 2995, win 793, options [nop,nop,TS val 30680074 ecr 1662877649], length 348: RTSP: RTSP/1.0 200 OK
13:37:00.353919 IP 192.168.17.21.rtsp > 10.42.0.116.45232: Flags [P.], seq 1908:2256, ack 2995, win 793, options [nop,nop,TS val 30680074 ecr 1662877649], length 348: RTSP: RTSP/1.0 200 OK
13:37:00.396068 IP 10.42.0.116.45232 > 192.168.17.21.rtsp: Flags [.], ack 2256, win 502, options [nop,nop,TS val 1662877694 ecr 30680074], length 0
13:37:00.396082 IP 10.42.0.116.45232 > 192.168.17.21.rtsp: Flags [.], ack 2256, win 502, options [nop,nop,TS val 1662877694 ecr 30680074], length 0
13:37:05.362161 IP 10.42.0.116.45232 > 192.168.17.21.rtsp: Flags [P.], seq 2995:3346, ack 2256, win 502, options [nop,nop,TS val 1662882660 ecr 30680074], length 351: RTSP: TEARDOWN rtsp://192.168.17.21:554/profile3/media.smp RTSP/1.0
13:37:05.362177 IP 10.42.0.116.45232 > 192.168.17.21.rtsp: Flags [P.], seq 2995:3346, ack 2256, win 502, options [nop,nop,TS val 1662882660 ecr 30680074], length 351: RTSP: TEARDOWN rtsp://192.168.17.21:554/profile3/media.smp RTSP/1.0
Which looks similar from the docker container from which it’s working, except that instead of TCP TEARDOWN it has
13:47:38.899885 IP 192.168.17.21.rtsp > 172.18.0.3.48610: Flags [P.], seq 1908:2256, ack 2995, win 793, options [nop,nop,TS val 30743926 ecr 1655054582], length 348: RTSP: RTSP/1.0 200 OK
13:47:38.899887 IP 192.168.17.21.rtsp > 172.18.0.3.48610: Flags [P.], seq 1908:2256, ack 2995, win 793, options [nop,nop,TS val 30743926 ecr 1655054582], length 348: RTSP: RTSP/1.0 200 OK
13:47:38.940101 IP 172.18.0.3.48610 > 192.168.17.21.rtsp: Flags [.], ack 2256, win 501, options [nop,nop,TS val 1655054624 ecr 30743926], length 0
13:47:38.940111 IP 172.18.0.3.48610 > 192.168.17.21.rtsp: Flags [.], ack 2256, win 501, options [nop,nop,TS val 1655054624 ecr 30743926], length 0
13:47:38.947192 IP 192.168.17.21.41592 > 172.18.0.3.37596: UDP, length 36
13:47:38.947197 IP 192.168.17.21.41592 > 172.18.0.3.37596: UDP, length 36
13:47:38.947203 IP 192.168.17.21.41592 > 172.18.0.3.37596: UDP, length 42
13:47:38.947204 IP 192.168.17.21.41592 > 172.18.0.3.37596: UDP, length 42
13:47:38.947248 IP 192.168.17.21.41592 > 172.18.0.3.37596: UDP, length 19
13:47:38.947249 IP 192.168.17.21.41592 > 172.18.0.3.37596: UDP, length 19
13:47:38.947252 IP 192.168.17.21.41592 > 172.18.0.3.37596: UDP, length 21
13:47:38.947254 IP 192.168.17.21.41592 > 172.18.0.3.37596: UDP, length 21
13:47:38.947758 IP 192.168.17.21.41592 > 172.18.0.3.37596: UDP, length 1414
13:47:38.947764 IP 192.168.17.21.41592 > 172.18.0.3.37596: UDP, length 1414
13:47:38.947829 IP 192.168.17.21.41592 > 172.18.0.3.37596: UDP, length 1414
13:47:38.947832 IP 192.168.17.21.41592 > 172.18.0.3.37596: UDP, length 1414
13:47:38.947944 IP 192.168.17.21.41592 > 172.18.0.3.37596: UDP, length 1414
13:47:38.947947 IP 192.168.17.21.41592 > 172.18.0.3.37596: UDP, length 1414
Any suggestions?
I tried to flush iptables, and create services for accessing the cameras, modifying network policies, but nothing seems to help.