User Stories

Introduction

The open-source community thrives on collective efforts to improve and augment software products. In this post, we take a closer look at the operations of the Data Plane Development Kit (DPDK) community lab and its contributions towards testing and refining the DPDK software project.

DPDK, a set of libraries and drivers for fast packet processing, is embraced by major technology players including Red Hat, NVIDIA, Intel, Arm, and more. However, a crucial element that bolsters the confidence of these companies in DPDK is the dedicated testing Community Lab hosted by the University of New Hampshire Interoperability Lab. Their primary service to the DPDK community involves rigorous testing of code contributions and providing valuable feedback to the developers.

The DPDK Community Lab: An Overview

Initiated around 2017-2018, the DPDK Community Lab was conceived to ensure that incoming code contributions didn’t introduce performance regressions across various vendor channels. Initial testing focused primarily on performance testing and gradually expanded to encompass a broader range of checks.

The lab executes multiple categories of tests on incoming source code contributions, including performance, functional testing, unit and compile testing, and Application Binary Interface (ABI) stability checks. ABI testing provides stability for development across different versions within that cycle. For instance, releases 22.11, 23.03, and 23.07 are/will be ABI compliant with the current major abi version. A new major ABI version will be released with the 23.11 LTS release which may break ABI compliance with releases from the current cycle. But then the subsequent 24.03 and 24.07 releases will be ABI compliant to the ABI major version introduced at 23.11.

The Testing Suite: A Developer’s First Encounter

One of the first steps for developers contributing to DPDK involves navigating the project’s mailing list, where they submit patches for review. Any code change or patch is subsequently passed through the community lab’s testing suite. Should a patch trigger an issue or fail the automated tests, the developers are notified through email as well as the ‘patchwork’ system, an automated web portal that tracks incoming source code patches.

Understanding the testing suite is an essential part of a new developer’s journey. Once developers familiarize themselves with the codebase, the next stage involves learning how to submit patches and navigate the unique mailing list patch archive approach, which can initially be somewhat alien to developers more accustomed to using the major code collaboration and version control tools like GitLab, GitHub, or Bitbucket’s direct patch submission process.

Extending Test Coverage: Evolution Over Time

Over the years, the DPDK community lab has significantly extended its test coverage, adding new types of hardware, architectures, and operating systems to its testing suite. In 2022 and 2023 the Community Lab has worked to update our internal container build system to make it OCI compliant. That has allowed the lab to seamlessly offer the same test coverage for arm64 platforms which was previously offered only for x86 systems. The lab has also made other expansions for ARM, like adding arm32 unit tests to CI runs, and is proud to say there is now parity between x86 and ARM test coverage in the Community Lab.

The lab operates on the principle of relative comparison; it does not publish absolute performance numbers. Instead, it compares the performance of a patch against the mainline version of the codebase. This approach ensures that no individual patch introduces significant performance degradation, which could negatively impact end users and integrators.

Policies and Test Cases

The lab derives its test cases from several sources, primarily the DPDK core developer community and the DPDK Test Suite (DTS). The latter focuses on functional testing and performance testing of the DPDK system as a whole, while the former is focused on individual building blocks within the codebase. There’s an ongoing effort within the community to merge DTS into the DPDK main repository to align its release cycles more closely with those of DPDK itself.

The third category of tests comes from purpose-built sample applications, such as the Federal Information Processing Standards (FIPS) test cases. These cases test the implementation of cryptographic algorithms built into DPDK. With recent advancements from NIST, the lab has transitioned from a manual vector request and testing approach to an automated API-driven process, which greatly facilitates the automation of cryptographic implementation testing.

Testing Methods

At the heart of the lab’s testing approach are three key methods: unit testing, functional testing, and performance testing. Each of these contributes a different perspective, ensuring a comprehensive review of the application. For instance, unit testing focuses on the smallest parts of the application, while functional and performance testing assess the application’s functionality and speed under various conditions.

Compiling is an important step in the process, which they undertake before running ABI (Application Binary Interface) tests. Ensuring that the interface remains consistent is vital, as it confirms the binary formatting of the interfaces isn’t altered. This ensures that applications depending on the DPDK stack don’t face unexpected failures.

The lab’s Continuous Integration (CI) platform, Jenkins, is instrumental in organizing the testing workflow. A graphical representation of our Jenkins Testing Tree can offer a clearer understanding of the testing process, starting from patch application, branching into various forms of testing, and then cycling back for reiteration.

As patches flow into the system, Jenkins starts the process by applying these patches to the DPDK Mainline. If the patches apply correctly, the updated DPDK enters the testing pipeline. The system then breaks down into functional and performance testing using the DPDK Test Suite (DTS) across different servers and network interface cards (NICs). Concurrently, ABI testing ensures consistency at the driver kernel level. Simultaneously, they conduct compile testing on both x86 and ARM architectures. Finally, DPDK’s inbuilt unit tests confirm the robustness of each small part of the software.

Expanding Partner Coverage

In terms of expanding the lab’s test coverage and partnering with member companies, they have historically relied on organic growth within the community. However, recent efforts have focused on DPDK Gold members, targeting the integration of their latest generation hardware into the Lab’s testing suite. This direct engagement ensures their systems are among the first tested by DPDK’s community infrastructure, providing a clear benefit for their participation.

The DPDK Community lab does not eliminate the need for in-house testing, instead it complements it, providing an additional layer of assurance and reliability. Being an open-source project, the lab’s testing capabilities offer wide-ranging coverage, ensuring that the project is robust and dependable. The ongoing expansion of the lab test coverage is testament to the DPDK community’s commitment to delivering consistently robust code. 

Learn more about the DPDK Community Lab testing and for how to get involved visit: https://lab.dpdk.org/results/dashboard/about/

Author: Benjamin Thomas – bthomas@linuxfoundation.org

Introduction

In the fast-paced world of telecommunications, companies are constantly seeking solutions to address evolving challenges and meet the demands of their customers. Ericsson, a global leader in the industry, has been at the forefront of incorporating new technologies into its product portfolio. One such technology is the Data Plane Development Kit (DPDK), which has proven instrumental in revolutionizing packet processing for Ericsson’s network infrastructure. This user story delves into Ericsson’s utilization of DPDK, the benefits it has brought, and the challenges associated with transitioning to a cloud-native environment.

Ericsson’s Shifting Landscape 

Ericsson, a prominent vendor in the telecommunications domain, has a rich history of innovation and adaptability. With over 100,000 employees and a diverse range of products, Ericsson has witnessed a significant shift from traditional infrastructure to cloud-native solutions. As the industry embraces cloud-native architectures, Ericsson recognizes the importance of incorporating new technologies that align with this paradigm shift. DPDK, though not entirely new, has emerged as a critical component within Ericsson’s product portfolio, facilitating efficient packet processing and enabling the company to remain competitive in an evolving market.

Exploring DPDK’s Role

Niklas Widell – Standardization Manager at Ericsson AB, and Maria Lingemark – Senior Software Engineer at Ericsson shed light on the company’s adoption of DPDK. Maria, who has been involved in evaluating and utilizing DPDK since 2016, emphasizes the need for high-speed packet processing and the ability to split packet flows into multiple parallel streams. DPDK’s Event Dev implementation has been instrumental in achieving these goals, enabling Ericsson to process a large number of packets per second while maintaining the flexibility to distribute packet processing across multiple steps.

Transitioning from Specialized Hardware 

Before incorporating DPDK, Ericsson relied on proprietary ASIC hardware to handle packet processing. However, the company recognized the need to shift toward more readily deployable commercial off-the-shelf (COTS) hardware solutions. DPDK played a crucial role in enabling Ericsson to transition from specialized hardware to a more versatile and scalable environment, reducing the reliance on custom solutions and increasing the reach of their offerings to a broader customer base.

Flexibility and Cost Efficiency

DPDK offers Ericsson the flexibility to deploy their packet processing solutions across a range of hardware configurations, both on ASIC hardware and on common x86-based platforms. By leveraging DPDK’s capabilities, Ericsson can scale their applications and efficiently utilize the available CPU resources. Moreover, the compatibility of DPDK with multiple drivers allows Ericsson to leverage hardware-specific features where available, enhancing performance and optimizing resource utilization.

Challenges of Observability and Cloud-Native Adoption 

As Ericsson embraces cloud-native architectures, they encounter challenges related to observability, performance monitoring, and troubleshooting. Observing and comprehending the behavior of a complex system that processes packets in parallel across multiple CPUs and threads can be daunting. Balancing observability with performance optimization becomes crucial, requiring continuous refinement and adaptation. Additionally, the shift to cloud-based deployments necessitates rethinking observability strategies and ensuring seamless performance monitoring in these environments.

We needed to shift from doing packet processing on special purpose hardware, to doing it on cloud-based general compute hardware. DPDK enabled this – it created versatility and broadened external access. It significantly helped Ericsson meet our customers’ needs and demands as those changed in scale, and gave our team greater portability as well. And the ability to be able to reuse it across different departments without having to rewrite code was, and is, a significant benefit. – Maria Lingemark, Senior Software Engineer – Ericsson

To tackle the observability challenges, Ericsson leverages the eBPF (extended Berkeley Packet Filter) integration in DPDK. By deploying eBPF programs within the DPDK framework, they have achieved efficient packet processing, improved throughput, and enhanced network visibility. The flexibility offered by eBPF allows Ericsson to tailor their networking solutions to specific use cases, ensuring optimal performance and resource utilization. 

“Ericsson uses the included eBPF support in DPDK to simplify observability in complex cloud environments.” Anders Hansen, Cloud RAN System Developer – Ericsson

DPDK BBDev (Baseband Device)

DPDK BBDev (Baseband Device) plays a critical role in Ericsson’s ability to develop a portable and efficient Radio Access Network (RAN) implementation that seamlessly integrates with hardware acceleration from leading silicon vendors. This integration enables Ericsson to leverage the full potential of specialized hardware acceleration features offered by these vendors, enhancing the performance and efficiency of their RAN solutions.

By utilizing DPDK BBDev, Ericsson gains access to a standardized programming interface that abstracts the complexities of hardware-specific optimizations. This allows them to focus on developing high-performance RAN software while ensuring compatibility with various hardware platforms. The portability provided by DPDK BBDev enables Ericsson to deploy their RAN solutions across a wide range of hardware architectures, offering flexibility to their customers, while cultivating a heath ORAN eco-system in the industry.

“DPDK BBDev enables Ericsson to create a portable and efficient RAN implementation that is well integrated with HW acceleration from major silicon vendors” – Michael Lundkvist,
Principal Developer, RAN Application Architect – Ericsson

The integration of HW acceleration from major silicon vendors further boosts Ericsson’s RAN implementation. These hardware accelerators are specifically designed to offload computationally intensive tasks, such as FEC processing, resulting in improved throughput, lower latency, and reduced power consumption. By effectively utilizing these acceleration capabilities through DPDK BBDev, Ericsson delivers efficient and high-performing RAN solutions to their customers.

For more in-depth information on how DPDK BBDev enables Ericsson’s portable and efficient RAN implementation, you can refer to the white paper provided here. This white paper will delve into the technical details and showcase the advantages of integrating DPDK BBDev with hardware acceleration from major silicon vendors, offering valuable insights into Ericsson’s innovative RAN solutions.

DPDK and the Open Source Linux Foundation Community

————————————————–

Ericsson derives substantial benefits from its active involvement in both the open-source DPDK (Data Plane Development Kit) community and the larger Linux Foundation. By being an integral part of these communities, Ericsson experiences several advantages that contribute to their success and technological advancements.

First and foremost, being part of the DPDK community grants Ericsson access to a thriving ecosystem of contributors and developers focused on advancing high-performance packet processing. This access enables Ericsson to stay at the forefront of technological developments, leverage new features, and benefit from ongoing enhancements to DPDK. The collaborative nature of the open-source community encourages continuous innovation, allowing Ericsson to deliver cutting-edge solutions to their customers.

Engaging in the DPDK community also fosters collaboration and knowledge sharing with industry peers and experts. Ericsson can exchange ideas, best practices, and insights, benefitting from the collective expertise of the community. This collaboration helps Ericsson overcome challenges, improve their solutions, and accelerate their development cycles, all while contributing to the growth and success of the DPDK project.

Furthermore, Ericsson experiences a faster time to market by utilizing DPDK and collaborating within the community. By leveraging the work done by the DPDK community, Ericsson can capitalize on existing libraries, APIs, and optimizations, saving valuable development effort and resources. This efficiency enables Ericsson to bring their solutions to market more rapidly, meeting customer demands, gaining a competitive edge, and seizing market opportunities promptly.

Interoperability and compatibility are additional advantages that Ericsson enjoys through their involvement in the DPDK community and the larger Linux Foundation. DPDK’s emphasis on interoperability and common standards allows Ericsson to seamlessly integrate their solutions with other systems and platforms. This compatibility fosters a broader ecosystem, enabling Ericsson to collaborate effectively with other vendors and organizations, further expanding their market reach.

Participating in these open-source communities also positions Ericsson as an influential player and thought leader in high-performance networking and packet processing. Their contributions to the DPDK project not only enhance the framework’s functionality but also demonstrate their technical expertise and commitment to open-source initiatives. Ericsson’s influence and leadership within the community allow them to shape the direction and evolution of DPDK, driving the adoption of industry standards and best practices.

Lastly, being part of the larger Linux Foundation ecosystem offers Ericsson access to a vast network of organizations, developers, and industry leaders. This ecosystem provides collaboration opportunities, potential partnerships, and access to a network of expertise. By leveraging these connections, Ericsson can foster innovation, explore joint development efforts, and stay at the forefront of technological advancements in networking and telecommunications.

Author: Michel Machado – michel@digirati.com.br

Overview 

Originally developed at Boston University, Gatekeeper is the brainchild of researchers who looked at the state of distributed denial-of-service (DDoS) attacks and realized that the community lacked an affordable, performant, and deployable solution to defending against such attacks. On one hand, cloud companies offer DDoS protection as a service, but this can be costly. On the other hand, many research proposals have been developed to allow Internet operators to protect their own networks, but none of these ideas have yielded production-quality software systems. Gatekeeper puts theory into practice by providing network operators with an instantly deployable and affordable solution for defending against DDoS attacks, and does so without sacrificing performance by leveraging DPDK as a key enabling technology.

The Challenge

Part of the challenge in defending against DDoS attacks is differentiating good traffic from bad traffic in seconds. To do so most effectively requires capturing the qualities of individual flows as they pass through the DDoS mitigation system — this allows the system to rate limit flows, apply policies based on the traffic features, and punish flows that misbehave by blocking them completely. Capturing these qualities for each packet that passes through the system requires an extreme amount of CPU and memory resources, especially during attacks that nowadays stretch beyond 1 Tbps. To withstand attacks of this magnitude, DDoS mitigation systems either need to be widely deployed in parallel (which is expensive), or need to be especially careful in how they process packets. The latter is where Gatekeeper utilizes DPDK to be able to work efficiently and affordably.

The Solution

To be able to process packets at this scale, kernel bypass is absolutely necessary. We chose DPDK as a kernel bypass solution because of its stability and support from industry, as well as the feature set that it supports. In fact, the feature set of DPDK is so rich that it significantly reduced our time to market since we did not have to write everything from scratch.

Gatekeeper heavily relies on three key features in DPDK: (1) NUMA-aware memory management, (2) burst packet I/O, and (3) eBPF. These features allow Gatekeeper to enforce policies as programs instead of firewall rules, and to do so efficiently. This gives operators a lot of flexibility in determining how flows are processed by Gatekeeper without having to sacrifice performance.

On occasion, we found some shortcomings in DPDK libraries. For example, the LPM6/FIB6/RIB6 libraries that perform longest prefix matching on IPv6 were not a good fit, and we had to implement our own. But for each issue we have come across, we’ve found huge success with other libraries as described below. Furthermore, the community is hard at work to address production demands such as dynamically setting memory zones (see rte_memzone_max_set() for details), which previously required patching DPDK to change.

The Results

With DPDK, Gatekeeper achieves the following benefits:

• NUMA-aware memory management allows Gatekeeper to reduce memory access latency by enabling CPU cores to access local memory instead of remote memory.
• Burst packet I/O reduces the per-packet cost of accessing and updating queues, enabling Gatekeeper to keep up with volumetric attacks.
• eBPF (integrated in DPDK) enables deployers to write policies that are impossible to express in other solutions such as requiring TCP friendliness, bandwidth per flow, and quotas for auxiliary packets (e.g. ICMP, TCP SYN, fragments) per flow. Thanks to the guarantee of termination of eBPF programs, Gatekeeper can gracefully continue processing packets even when an eBPF program is buggy.

Many other DPDK features, including prefetching, the kernel-NIC interface, and packet filters play key supporting roles. With DPDK’s help, a modest Gatekeeper server can track 2 billion flows while processing at the very least 10 Mpps through eBPF program policies to decide how to allow, rate limit, or drop traffic.

Gatekeeper puts DDoS defense back on the hands of network operators, administrators and the general Open Source community. What was until recently only available via opaque and expensive third-party services can now be deployed by anyone with the appropriate infrastructure, with levels of flexibility and control that simply did not formerly exist. Andre Nathan – Digirati

The Benefits

DDoS attacks cause great financial, political, and social damage, and are only increasing in magnitude, complexity, and frequency. With Gatekeeper, network operators have a production-quality, open source choice in the market to defend their infrastructure and services. With the aid of technologies like DPDK, Gatekeeper is able to flexibly and efficiently defend against attacks, lowering the cost of deployment and enabling many stakeholders to protect themselves. To date, Gatekeeper has been deployed by Internet service providers, data centers, and gaming companies, and hopes to reach new deployers to eventually eradicate DDoS attacks.

Check out their GitHub here

White paper link 

Have a user story of your own that you would like to share across the DPDK and Linux foundation communities? Submit one here.