Obtaining and building Linux kernel source for DOCA 1.0

Hi Team

I am very excited by the DOCA release. My project will require making Linux kernel modifications and then installing this modified kernel on the BlueField 2. To that end I have a couple of questions.

  1. Where can I obtain the source code for the DOCA 1.0 kernel. Ideally this is a git repository but a tarball works too.

  2. Does this source include a .config for the DOCA 1.0 kernel? If not can such a .config be provided?

  3. Are there instructions to build and install this kernel? My team is very comfortable building kernels (we do 10-100 per day) but I am not sure if there are special considerations for the BlueField-2. Also instructions on how to integrate this kernel image and modules into a BF2 disk image would be useful.

Thanks!

Stevie Bates

The official DOCA 1.0 BFB uses the Ubuntu distro, which is built and released by Canonical, and as such we are still working out the details on how to make the kernel source available.

Be that as it may, DOCA also comes in other distros, such as CentOS (for the nonce), Debian, and Fedora. These distros are built by Nvidia and we can supply the kernel source. If you use any of these community distros, then the kernel source and the .config used to build it can be had by asking. Please contact your friendly neighborhood SE/FAE to make a request for it.

Note that the kernel source used in Ubuntu is slightly different from the kernel used in CentOS/Debian/Fedora. While they all have the necessary BF2 patches, the Ubuntu kernel has additional patches provided by Canonical.

1 Like

Thanks @jtau

I was not aware that all the driver code for BF-2 was upstreamed into the distro kernels. That is very useful. Does this include driver code for all the accelerator engines (e.g. compression, rege, nvme-of offload)?

I will work with our SE/FAE to get access to the kernel source and .config.

One final question for now. In DOCA is NVMe-oF target mode (i.e. on the storage server side) implemented? If so, is that implemented in kernel-space (via the NVMe-oF kernel driver) or via SPDK?

Stevie

No, the BF-2 specific patches are not yet upstreamed. We did not mean that they are. What we meant was that we take the CentOS/Debian/Fedora distros and install BF-2-patched kernel and roll it all up in the installation image. For the CentOS/Debian/Fedora distros Nvidia builds the kernel and the image. For the Ubuntu distro, Canonical does it.

Nvidia can provide the kernel source used in the CentOS/Debian/Fedora installation images for the BF-2.

As for NVMe-oF target mode, do you mean using the BF-2 as a controller or the (smart)NIC and the host as the controller? For the former, it is nominally part of DOCA, but there are no new APIs for it. The procedure to configure the BF2 as an NVMe-oF target controller under DOCA is the same as without DOCA. To use the BF-2 as an NVMe-oF target requires that the BF-2 controller card be used. This card cannot be plugged in an x86 server and be an NVMe-oF target controller. The BF2 needs to be the root complex as it would be in a JBOF.

But if the host is the NVMe-oF target controller and the BF-2 is just a (smart)NIC, then whether the BF2 is DOCA-enabled is irrelevant.

@jtau

Thanks again for the great response!

Nvidia can provide the kernel source used in the CentOS/Debian/Fedora installation images for the BF-2.

Great! This is what we need access too. Who do we need to talk too to get approval for access to this code.

and roll it all up in the installation image

Do you have a tool for the creation of these installation images? I see you have a docker hub container for this and if that is still up to date we would love access to the Dockerfile itself so we can do this generation in-house.

To use the BF-2 as an NVMe-oF target requires that the BF-2 controller card be used.

Yes this is what we are doing. We are using BF-2 as the NVMe-oF target at the front end of a Ethernet attached JBOF from an ODM we are working with. So yes, the firmware on the card will put the BF-2 into PCIe Root-Complex mode and we will be hanging NVMe SSDs and some other PCIe devices off the BF-2. There is no other CPU in the JBOF so the BF-2 will be used to terminate the NVMe-oF commands and then issue PCIe-based NVMe commands to the SSDs inside the JBOF and then NVMe-oF RDMA that data back to the initiators.

Stevie