I am exploring options to restrict or limit the exposure of the device path associated with the Precision Time Protocol (PTP) hardware clock of BlueField-2 (BF2) network interface cards (NICs) to virtual machines (VMs) managed by KubeVirt on a Kubernetes cluster.
The BF2 NICs have inherent PTP hardware clocks, and their device paths are currently visible to all provisioned VMs. This is unlike other NICs, such as StingRay, where this is not the case. My objective is to ensure that the VMs only have access to specific device paths, enhancing control and security.
However, various limitations and constraints have been identified. Directly unbinding the PCI device or using udev rules to ignore the device is not a viable option, as it impacts other functionalities and is no longer supported in the latest udev versions, respectively. Additionally, granular control via KubeVirt or SR-IOV Network Device Plugin configurations has proven ineffective in achieving this level of restriction.
A potential workaround involves manipulating the hardware timestamping capabilities of the BF2 NICs, but this needs to be approached with caution to avoid unintended consequences on other functionalities.
I am interested in any insights, alternative approaches, or solutions that can help achieve this objective without compromising system integrity and functionality.
kubevirt VM on K8S node with BF2 nic
# ls -lah /sys/class/ptp/
total 0
drwxr-xr-x 2 root root 0 Oct 12 23:01 .
drwxr-xr-x 57 root root 0 Oct 12 23:01 ..
lrwxrwxrwx 1 root root 0 Oct 12 23:01 ptp0 -> ../../devices/pci0000:00/0000:00:02.3/0000:04:00.0/ptp/ptp0
lrwxrwxrwx 1 root root 0 Oct 12 23:01 ptp1 -> ../../devices/virtual/ptp/ptp1
K8S node with BF2 nic
# ethtool -T p0
Time stamping parameters for p0:
Capabilities:
hardware-transmit (SOF_TIMESTAMPING_TX_HARDWARE)
hardware-receive (SOF_TIMESTAMPING_RX_HARDWARE)
hardware-raw-clock (SOF_TIMESTAMPING_RAW_HARDWARE)
PTP Hardware Clock: 0
Hardware Transmit Timestamp Modes:
off (HWTSTAMP_TX_OFF)
on (HWTSTAMP_TX_ON)
Hardware Receive Filter Modes:
none (HWTSTAMP_FILTER_NONE)
all (HWTSTAMP_FILTER_ALL)
#
# ethtool -T p1
Time stamping parameters for p1:
Capabilities:
hardware-transmit (SOF_TIMESTAMPING_TX_HARDWARE)
hardware-receive (SOF_TIMESTAMPING_RX_HARDWARE)
hardware-raw-clock (SOF_TIMESTAMPING_RAW_HARDWARE)
PTP Hardware Clock: 1
Hardware Transmit Timestamp Modes:
off (HWTSTAMP_TX_OFF)
on (HWTSTAMP_TX_ON)
Hardware Receive Filter Modes:
none (HWTSTAMP_FILTER_NONE)
all (HWTSTAMP_FILTER_ALL)
#
# ethtool -T ovsbr0
Time stamping parameters for ovsbr0:
Capabilities:
software-transmit (SOF_TIMESTAMPING_TX_SOFTWARE)
software-receive (SOF_TIMESTAMPING_RX_SOFTWARE)
software-system-clock (SOF_TIMESTAMPING_SOFTWARE)
PTP Hardware Clock: none
Hardware Transmit Timestamp Modes: none
Hardware Receive Filter Modes: none
# ls -lah /sys/class/ptp/
total 0
drwxr-xr-x 2 root root 0 Oct 12 22:52 .
drwxr-xr-x 66 root root 0 Oct 12 22:52 ..
lrwxrwxrwx 1 root root 0 Oct 12 22:52 ptp0 -> ../../devices/pci0000:00/0000:00:00.0/0000:01:00.0/0000:02:00.0/0000:03:00.0/ptp/ptp0
lrwxrwxrwx 1 root root 0 Oct 12 22:52 ptp1 -> ../../devices/pci0000:00/0000:00:00.0/0000:01:00.0/0000:02:00.0/0000:03:00.1/ptp/ptp1
kubevirt VM on K8S node with NON BF2 nic
# ls -lah /sys/class/ptp/
total 0
drwxr-xr-x 2 root root 0 Oct 12 18:38 .
drwxr-xr-x 55 root root 0 Oct 12 18:38 ..
lrwxrwxrwx 1 root root 0 Oct 12 18:38 ptp0 -> ../../devices/virtual/ptp/ptp0
K8S node with NON BF2 nic
# ethtool -T p0
Time stamping parameters for p0:
Cannot get device time stamping settings: No such device
# ethtool -T p1
Time stamping parameters for p1:
Cannot get device time stamping settings: No such device
# ls -lah /sys/class/ptp/
total 0
drwxr-xr-x 2 root root 0 Jun 22 2018 .
drwxr-xr-x 62 root root 0 Jun 22 2018 ..