Nvethernet: Bug in SICGHWTSTAMP ioctl

Jetson AGX Orin devkit with Jetpack 7.2, stock BSP installed.

The following sequence initializing PTP timestamping by linuxptp fails: linuxptp/sk.c at e312b959a7dd3e8db0c6c8e282917c89909f8f76 · nwtime/linuxptp · GitHub .

What happens is that the program calls SIOCGHWTSTAMP with cfg.flags = HWTSTAMP_FLAG_BONDED_PHC_INDEX. However, the nvethernet driver doesn’t respond with EOPNOTSUPP (which would mean VLAN over bond is not supported on this interface). The nvethernet driver code in ether_linux.c , function ether_ioctl() does just this:

case SIOCGHWTSTAMP:
		config = &pdata->ptp_config;
		ret = copy_to_user(rq->ifr_data, config, sizeof(*config)) ? -EFAULT : 0;
		break;

So it does not check the cfg.flags at all. This means linuxptp thinks that HWTSTAMP_FLAG_BONDED_PHC_INDEX flag is supported.

In the next round, linuxptp tries to set the timestamping mode with SIOCSHWTSTAMP with cfg.flags set to HWTSTAMP_FLAG_BONDED_PHC_INDEX.

However, in nvethernet driver, file ptp.c, function ether_handle_hwtstamp_ioctl(), this is what happens:

/* reserved for future extensions */
	if (config.flags) {
		return -EINVAL;
	}

So linuxptp is unhappy because GHWTSTAMP told it that the flag is supported, but SHWTSTAMP fails when it is set.

The driver should be fixed to either correctly implement the VLAN over bond behavior, or to at least not pretend it support it.

One possible solution could be (I didn’t check this compiles):

case SIOCGHWTSTAMP:
        struct hwtstamp_config in_config;
        if (copy_from_user(&in_config, ifr->ifr_data, sizeof(struct hwtstamp_config))) {
    		return -EFAULT;
	    }
        if (in_config.flags > 0)
            return -EOPNOTSUPP;

		config = &pdata->ptp_config;
		ret = copy_to_user(rq->ifr_data, config, sizeof(*config)) ? -EFAULT : 0;
		break;

Hmm, maybe it could be something else. linuxptp receives error value -1 which would be EPERM and not EINVAL. However, maybe something mangles this error on its way, because when I set flags = 0, the call succeeds and timestamping is set up.

In what kind of usecase did you reproduce the issue?

We want to use PTP over this port to sync time with a device. But linuxptp cannot use the integrated port due to this bug.

You can see with `sudo ptp4l -i end0 -H -m -l6`. The port immediately falls into an error state because timestamping cannot be set up.

What kind of error did you see when running this?

For what connection did you do on this? I mean where did you connect Orin AGX devkit ethernet port to?

The port is connected to an Ouster lidar. The connection itself is good.

STR

Download, build and install master branch of GitHub - nwtime/linuxptp: Linux PTP Project · GitHub .

Our NIC is renamed to eth_ouster via udev rules:

$ dmesg | grep eth_ouster
[   48.378749] nvethernet 6800000.ethernet eth_ouster: renamed from eth1
[   48.853657] nvethernet 6800000.ethernet eth_ouster: ether_get_wol: phydev is null check iface up status
[   59.236862] nvethernet 6800000.ethernet eth_ouster: Link is Up - 1Gbps/Full - flow control rx

Command:

/usr/local/sbin/ptp4l -i eth_ouster -f /etc/linuxptp/automotive-bc.cfg --step_threshold=0.001 -m -l6

Config /etc/linuxptp/automotive-bc.cfg:

[global]
gmCapable               1
priority1               100
priority2               248
logSyncInterval         -3
syncReceiptTimeout      3
neighborPropDelayThresh 800
min_neighbor_prop_delay -20000000
assume_two_step         1
path_trace_enabled      1
ptp_dst_mac             01:80:C2:00:00:0E
BMCA                    noop
inhibit_announce        1
asCapable               true

# Defaults for upstream (slave) port
clientOnly              1
ignore_source_id        1

# VRAS custom
servo_offset_threshold   100
servo_num_offset_values  10
boundary_clock_jbod      1
time_stamping            hardware
clock_servo              linreg
uds_ro_file_mode         0777
step_threshold           0.001
follow_up_info           1
transportSpecific        0x1
network_transport        L2
delay_mechanism          P2P
tx_timestamp_timeout     10

[eth_ouster]
serverOnly 1

Output of the command:

ptp4l[724.812]: selected /dev/ptp0 as PTP clock
ptp4l[724.812]: driver rejected most general HWTSTAMP filter
ptp4l[724.819]: ioctl SIOCSHWTSTAMP failed: Numerical result out of range
ptp4l[724.848]: port 1 (eth_ouster): INITIALIZING to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)

Corresponding lines in dmesg:

[  724.829516] nvethernet 6800000.ethernet: config.flags = 0x1, tx_type = 0x1,rx_filter = 0xc
[  724.829545] nvethernet 6800000.ethernet: config.flags = 0x0, tx_type = 0x1,rx_filter = 0x9
[  724.829549] nvethernet 6800000.ethernet: rx_filter is out of range

If I rebuild linuxptp with this patch that zeros-out the flags field, it starts and runs correctly. However, the usage of the flags by linuxptp (without the patch) seems correct to me so it should work with the driver.

diff --git a/sk.c b/sk.c
index 7e59281..34dff0c 100644
--- a/sk.c
+++ b/sk.c
@@ -100,6 +100,7 @@ static int hwts_init(int fd, const char *device, int rx_filter,
        case HWTS_FILTER_NORMAL:
                cfg.tx_type   = tx_type;
                cfg.rx_filter = orig_rx_filter = rx_filter;
+               cfg.flags = 0;
                err = ioctl(fd, SIOCSHWTSTAMP, &ifreq);
                if (err < 0) {
                        pr_info("driver rejected most general HWTSTAMP filter");

Sorry, just want to confirm.

What is the ptp4l command that you want to try here?

Is this still related to SIOCSHWTSTAMP? I tried to run your previous ptp4l command. Indeed SIOCSHWTSTAMP failed but the there is a fallback option to continue.

This:

This command triggers the SIOCSHWTSTAMP error which rejects rx filter 0xc (because of the wrong handling of flags):

ptp4l[724.812]: driver rejected most general HWTSTAMP filter

The fallback that ptp4l tries next says:

ptp4l[724.819]: ioctl SIOCSHWTSTAMP failed: Numerical result out of range

which is correct because the driver does not, indeed, advertise support for rx filter 0x9.

The port then remains in FAULTY state and never syncs.

please check if this resolves your issue.

diff --git a/drivers/net/ethernet/nvidia/nvethernet/ethtool.c b/drivers/net/ethernet/nvidia/nvethernet/ethtool.c
index a1c4b2b..6cd6488 100644
--- a/drivers/net/ethernet/nvidia/nvethernet/ethtool.c
+++ b/drivers/net/ethernet/nvidia/nvethernet/ethtool.c
@@ -1392,6 +1392,7 @@
 			  (1 << HWTSTAMP_TX_ONESTEP_SYNC));
 
 	info->rx_filters |= ((1 << HWTSTAMP_FILTER_PTP_V1_L4_SYNC) |
+			     (1 << HWTSTAMP_FILTER_PTP_V2_L2_EVENT) |
 			     (1 << HWTSTAMP_FILTER_PTP_V2_L2_SYNC) |
 			     (1 << HWTSTAMP_FILTER_PTP_V2_L4_SYNC) |
 			     (1 << HWTSTAMP_FILTER_PTP_V1_L4_DELAY_REQ) |
diff --git a/drivers/net/ethernet/nvidia/nvethernet/ptp.c b/drivers/net/ethernet/nvidia/nvethernet/ptp.c
index d6342e6..dd6c4ff 100644
--- a/drivers/net/ethernet/nvidia/nvethernet/ptp.c
+++ b/drivers/net/ethernet/nvidia/nvethernet/ptp.c
@@ -327,6 +327,23 @@
 }
 #endif /* !OSI_STRIPPED_LIB */
 
+static void ether_normalize_hwtstamp_config(struct hwtstamp_config *config)
+{
+	switch (config->rx_filter) {
+	case HWTSTAMP_FILTER_PTP_V2_L2_EVENT:
+		config->rx_filter = HWTSTAMP_FILTER_PTP_V2_EVENT;
+		break;
+	case HWTSTAMP_FILTER_PTP_V2_L2_SYNC:
+		config->rx_filter = HWTSTAMP_FILTER_PTP_V2_SYNC;
+		break;
+	case HWTSTAMP_FILTER_PTP_V2_L2_DELAY_REQ:
+		config->rx_filter = HWTSTAMP_FILTER_PTP_V2_DELAY_REQ;
+		break;
+	default:
+		break;
+	}
+}
+
 int ether_handle_hwtstamp_ioctl(struct ether_priv_data *pdata,
 		struct ifreq *ifr)
 {
@@ -355,11 +372,17 @@
 		 "rx_filter = %#x\n", config.flags, config.tx_type,
 		 config.rx_filter);
 
+#ifdef HWTSTAMP_FLAG_BONDED_PHC_INDEX
+	if (config.flags & ~HWTSTAMP_FLAG_MASK) {
+#else
 	/* reserved for future extensions */
 	if (config.flags) {
+#endif
 		return -EINVAL;
 	}
 
+	ether_normalize_hwtstamp_config(&config);
+
 	if (memcmp(&config, &pdata->ptp_config, sizeof(struct hwtstamp_config)) == 0) {
 		goto skip;
 	}

@WayneWWW is there more information or tests you need from me to help resolving this issue?

Okay, I’ll try to give this a test once I figure how to compile this patch.

Just a quick async question: do you still think it’s a good idea to check the value of the reserved field flags when HWTSTAMP_FLAG_BONDED_PHC_INDEX is not defined? (i.e. on older kernels). Usually, I’d say that the values of reserved fields should not be checked at all.

ok. Please change my patch to

diff --git a/drivers/net/ethernet/nvidia/nvethernet/ethtool.c b/drivers/net/ethernet/nvidia/nvethernet/ethtool.c
index a1c4b2bbf..6cd648889 100644
--- a/drivers/net/ethernet/nvidia/nvethernet/ethtool.c
+++ b/drivers/net/ethernet/nvidia/nvethernet/ethtool.c
@@ -1392,6 +1392,7 @@ int ether_get_ts_info(struct net_device *ndev,
                          (1 << HWTSTAMP_TX_ONESTEP_SYNC));

        info->rx_filters |= ((1 << HWTSTAMP_FILTER_PTP_V1_L4_SYNC) |
+                            (1 << HWTSTAMP_FILTER_PTP_V2_L2_EVENT) |
                             (1 << HWTSTAMP_FILTER_PTP_V2_L2_SYNC) |
                             (1 << HWTSTAMP_FILTER_PTP_V2_L4_SYNC) |
                             (1 << HWTSTAMP_FILTER_PTP_V1_L4_DELAY_REQ) |
diff --git a/drivers/net/ethernet/nvidia/nvethernet/ptp.c b/drivers/net/ethernet/nvidia/nvethernet/ptp.c
index d6342e652..630c75899 100644
--- a/drivers/net/ethernet/nvidia/nvethernet/ptp.c
+++ b/drivers/net/ethernet/nvidia/nvethernet/ptp.c
@@ -327,6 +327,23 @@ static void ether_config_slot_function(struct ether_priv_data *pdata, u32 set)
 }
 #endif /* !OSI_STRIPPED_LIB */

+static void ether_normalize_hwtstamp_config(struct hwtstamp_config *config)
+{
+       switch (config->rx_filter) {
+       case HWTSTAMP_FILTER_PTP_V2_L2_EVENT:
+               config->rx_filter = HWTSTAMP_FILTER_PTP_V2_EVENT;
+               break;
+       case HWTSTAMP_FILTER_PTP_V2_L2_SYNC:
+               config->rx_filter = HWTSTAMP_FILTER_PTP_V2_SYNC;
+               break;
+       case HWTSTAMP_FILTER_PTP_V2_L2_DELAY_REQ:
+               config->rx_filter = HWTSTAMP_FILTER_PTP_V2_DELAY_REQ;
+               break;
+       default:
+               break;
+       }
+}
+
 int ether_handle_hwtstamp_ioctl(struct ether_priv_data *pdata,
                struct ifreq *ifr)
 {
@@ -355,10 +372,7 @@ int ether_handle_hwtstamp_ioctl(struct ether_priv_data *pdata,
                 "rx_filter = %#x\n", config.flags, config.tx_type,
                 config.rx_filter);

-       /* reserved for future extensions */
-       if (config.flags) {
-               return -EINVAL;
-       }
+       ether_normalize_hwtstamp_config(&config);

        if (memcmp(&config, &pdata->ptp_config, sizeof(struct hwtstamp_config)) == 0) {
                goto skip;

Thanks, this patch resolves the issue. I’ve tested it on a live Orin AGX devkit.

The last thing to resolve is how to handle the flags == HWTSTAMP_FLAG_BONDED_PHC_INDEX case. If a program would actually want to use this feature, the fixed driver would not fail, but I guess it would return wrong values. However, I’m not sure if nvethernet supports bonding at all, so maybe this case couldn’t happen?

Will this fix be integrated into the next L4T release?