I am very excited by the DOCA release. My project will require making Linux kernel modifications and then installing this modified kernel on the BlueField 2. To that end I have a couple of questions.
Where can I obtain the source code for the DOCA 1.0 kernel. Ideally this is a git repository but a tarball works too.
Does this source include a .config for the DOCA 1.0 kernel? If not can such a .config be provided?
Are there instructions to build and install this kernel? My team is very comfortable building kernels (we do 10-100 per day) but I am not sure if there are special considerations for the BlueField-2. Also instructions on how to integrate this kernel image and modules into a BF2 disk image would be useful.
The official DOCA 1.0 BFB uses the Ubuntu distro, which is built and released by Canonical, and as such we are still working out the details on how to make the kernel source available.
Be that as it may, DOCA also comes in other distros, such as CentOS (for the nonce), Debian, and Fedora. These distros are built by Nvidia and we can supply the kernel source. If you use any of these community distros, then the kernel source and the .config used to build it can be had by asking. Please contact your friendly neighborhood SE/FAE to make a request for it.
Note that the kernel source used in Ubuntu is slightly different from the kernel used in CentOS/Debian/Fedora. While they all have the necessary BF2 patches, the Ubuntu kernel has additional patches provided by Canonical.
I was not aware that all the driver code for BF-2 was upstreamed into the distro kernels. That is very useful. Does this include driver code for all the accelerator engines (e.g. compression, rege, nvme-of offload)?
I will work with our SE/FAE to get access to the kernel source and .config.
One final question for now. In DOCA is NVMe-oF target mode (i.e. on the storage server side) implemented? If so, is that implemented in kernel-space (via the NVMe-oF kernel driver) or via SPDK?
No, the BF-2 specific patches are not yet upstreamed. We did not mean that they are. What we meant was that we take the CentOS/Debian/Fedora distros and install BF-2-patched kernel and roll it all up in the installation image. For the CentOS/Debian/Fedora distros Nvidia builds the kernel and the image. For the Ubuntu distro, Canonical does it.
Nvidia can provide the kernel source used in the CentOS/Debian/Fedora installation images for the BF-2.
As for NVMe-oF target mode, do you mean using the BF-2 as a controller or the (smart)NIC and the host as the controller? For the former, it is nominally part of DOCA, but there are no new APIs for it. The procedure to configure the BF2 as an NVMe-oF target controller under DOCA is the same as without DOCA. To use the BF-2 as an NVMe-oF target requires that the BF-2 controller card be used. This card cannot be plugged in an x86 server and be an NVMe-oF target controller. The BF2 needs to be the root complex as it would be in a JBOF.
But if the host is the NVMe-oF target controller and the BF-2 is just a (smart)NIC, then whether the BF2 is DOCA-enabled is irrelevant.
Nvidia can provide the kernel source used in the CentOS/Debian/Fedora installation images for the BF-2.
Great! This is what we need access too. Who do we need to talk too to get approval for access to this code.
and roll it all up in the installation image
Do you have a tool for the creation of these installation images? I see you have a docker hub container for this and if that is still up to date we would love access to the Dockerfile itself so we can do this generation in-house.
To use the BF-2 as an NVMe-oF target requires that the BF-2 controller card be used.
Yes this is what we are doing. We are using BF-2 as the NVMe-oF target at the front end of a Ethernet attached JBOF from an ODM we are working with. So yes, the firmware on the card will put the BF-2 into PCIe Root-Complex mode and we will be hanging NVMe SSDs and some other PCIe devices off the BF-2. There is no other CPU in the JBOF so the BF-2 will be used to terminate the NVMe-oF commands and then issue PCIe-based NVMe commands to the SSDs inside the JBOF and then NVMe-oF RDMA that data back to the initiators.
How did you set the firmware to make BF-2 as a Root-Complex?
I’m trying to do this but still when I access the Arm core, I cannot find any PCIe devices plugged in other slots.
If you can provide more informations about this, it would be helpful!
*>> How did you set the firmware to make BF-2 as a Root-Complex? *
This is device specific and set in the FW in the factory. It’s not something a user can change at will. So if your BF2 device is a controller card, then it should already be configured as a root complex. No additional configuration is needed to configure it as a root-complex. If your BF2 device is not a controller card, you have a DPU, in which case it can never be a root-complex.
For this issue, please open a support case so that we can properly resolve it.
I would check the link as you provided for the kernel source.
For storage-related offloads, I mean the deduplication and compression not the storage target. Where can I find those acceleration related features? Those are integrated into the kernel as a user library?
Could you explain more about the hardware acceleration?
I am curious about the the compression/decompression and deduplication features in BlueField-2.
Do you provide some APIs for those features? If so, where can I find it?
Are there any updates for this question?
I wonder whether I can enable the storage accelerator (deduplication, compression/decompression) in a form of API.
