By Lincoln Lavoie, UNH-IOL, Principal Engineer
A little more than 3 years ago, we set out to improve the qualities and capabilities of the continuous integration (CI) testing of the DPDK source code and new patches. Now, it’s not uncommon for testing to come last in the process, and sometimes it doesn’t receive it’s fair share of attention. Which of us hasn’t worked on a project, with the boss pushing off the testing efforts or milestones, in order to allow “just that bit more” focus or expansion on the features or customer wishes for the project? However, as DPDK has matured and is a key foundation for many applications and deployments, we need to ensure the stability and robustness of the codebase and incoming patches.
Enter the Community CI team, a virtual team of DPDK community members who meet regularly and help maintain all of the tooling that enables the testing activities. We’re working to better organize and support the testing resources available to the DPDK development teams. Today, there are 3 primary testing sources, running as TravisCI, Intel’s hosted lab, and the community lab hosted at the UNH-IOL. Each of the “labs” provides results back to the DPDK project’s patchwork tracking system, and tests different aspects of the “code,” ranging from compile and unit tests, to performance and functionality tests on bare-metal with real NICs and drivers.
We thought it might be worthwhile to spend a few words on a blog to highlight what testing is already running, what testing is under development, and finally, where do we see things going from here. Starting with the Travis CI systems, which are triggered from a github robot that is tracking commits on the main branch, running compile and unit testing across both x86 and aarch64 architectures, with cross compiles to 32-bit, all on top of the Ubuntu OS. The robot and CI jobs for this integration have been developed and maintained largely by our Redhat peers, so, shoutouts where they’re due, “great job guys!” You can check out the Travis CI builds here: https://travis-ci.com/github/DPDK
Next up, Intel’s hosted lab, where they are running compile testing across a number of different operating systems on x86 hardware. The operating systems range from Linux distros, to FreeBSD, and even Windows. Adding another dimension, the compilation is also checked using both Makefile and Meson. The system is tracking both the main branch and as well as providing results for patches, before their merge onto the main, and provides early feedback to the patch submitter if issues pop up from a patch, afterall, “who knew” that approach didn’t work over all of DPDK’s supported OSes (well, now you know). Intel’s support in providing both the hosted lab environment and keeping things running smoothly is a great help, “thanks Intel team!” You can see a sample of the Intel hosted lab output in patchworks here: https://patchwork.dpdk.org/patch/75413/
Lastly, for the currently operating testing, is the community lab, hosted at the UNH-IOL. Testing in the community lab is also tracking both the main branch and submitted patches, providing a combination of compile, unit testing, functional testing, and performance testing. The compile and unit testing also add one additional dimension, with some cooperation to downstream projects, such as Open vSwitch and SPDK, providing early detection of a patch breaking outside of DPDK directly. For the functional and performance testing, the lab hosts a number of “bare-metal” systems with NICs provided by DPDK member companies, including Broadcom, Intel, Mellanox, and NXP currently. The performance testing runs a throughput test on each NIC (in some cases there are multiple speeds or hardware variants included), checking for a drop in forwarding performance from a well known reference. Gold and Silver DPDK member companies are eligible to submit hardware for hosting in the community lab and are strongly encouraged to do so by the governing and tech boards of DPDK, as this directly expands the testing coverages. Functional testing is the newest and growing aspect within the community lab output, adding more tests running on “bare-metal”, based on the DTS (DPDK Test Suite) project. Aiming to provide majority coverage for the NIC features list. The community lab results also feed directly into the patchwork system, with additional information available on the lab portal as well: https://lab.dpdk.org/results/dashboard/
Looking to the future, there are two efforts currently underway. First, the robot that is tracking the github main branch is being extended to also drive builds on the OpenSUSE Build Service (OBS), further extending that coverage to more OS / build variants. Second is the expansion of the feature testing coverage, which is focused on extending and updating the DTS coverage, with that work driving back into the functional testing run in the community lab.
The Community CI team meets every other week, on Thursdays, at 1pm UTC. All DPDK community members and maintainers are welcome to join and participate in the discussions, just drop a line to ci@dpdk.org. We hope to see you there!