Event organizers can apply for an Event D&I Badge for reasons of leadership, self-reflection, and self-improvement on issues critical to fostering more diverse and inclusive events. For us, this means taking conscious steps to make it easier for people of all backgrounds to participate in our events — from submitting speaking sessions to attending.
The awarding of the CHAOSS D&I badge is an acknowledgement of DPDK’s commitment to implementing healthy Diversity and Inclusion (D&I) practices. These efforts are complementary to community-driven efforts to engage in using more inclusive language within the DPDK codebase. For this event, we were measured on our efforts related to Speaker and Attendee Diversity & Inclusion, commitment to a specific code of conduct for all attendees, and measurement of access to the event, specifically measuring what, if any, barriers to access attendees must overcome to attend the event. Because all recent DPDK Summit events have been virtual, it has been an especially important initiative to welcome attendees from all over the world. One way we have been able to accomplish this is to offer all virtual events free of cost.
While we are still a long way off from a fully diverse crowd, we are thrilled to have been recognized for the efforts we are making. It is so important –especially in a community as traditionally homogeneous as DPDK — that all voices be heard and welcomed.
Big thanks to Rachel Braun and the Linux Foundation Events team for help making this a reality.
We look forward to making even more strides towards greater diversity and wider inclusion among DPDK events in the future.
DPDK is widely used across the technology industry to accelerate packet processing on a varied mix of platforms and architectures. However, all architectures are not created the same way and they all have differing characteristics. For frameworks such as DPDK, it is a challenge to efficiently use specific architecture features without impacting performance of other supported architectures.
One such characteristic is the memory model, which describes the behavior of accesses to shared memory by multi-processor systems. The Arm and PowerPC architectures support a weakly ordered memory model whereas x86 supports a strongly ordered memory model. Consider the following table that shows the ordering guarantees provided by these architectures for various sequences of memory operations.
As shown above, in the Arm architecture, all four possibilities of sequences of memory operations can be reordered. If an algorithm requires these memory operations to be executed (completed) in the program order, memory barriers are required to enforce the ordering. On x86, only store – load sequence can be reordered and requires a barrier to enforce the program order. The other sequences are guaranteed to execute in the program order.
However, not all algorithms need the stronger ordering guarantees. For example, take the case of a simple spinlock protecting a critical section.
The operations inside the critical section are not allowed to hoist above ‘spinlock-lock’ and sink below ‘spinlock-unlock’. But the operations above are allowed to sink below ‘spinlock-lock’ and the operations below are allowed to hoist above ‘spinlock-unlock’. In this case, protecting the critical section using locks does not require that lock and unlock functions provide full barriers. This gives more flexibility to the CPU to execute the instructions efficiently during run time. Note that this is just one example and many other algorithms have similar behaviors. The Arm architecture ‘provides load/store instructions, atomic instructions and barriers that support sinking and hoisting of memory operations in one direction .
In order to support such architectural differences DPDK uses abstract APIs. The rte_smp_mb/wmb/rmb APIs provide support for full memory barrier, store memory barrier and load memory barrier. APIs for atomic operations such as rte_atomic_load/store/add/sub are also available. These APIs use full barriers and do not take any memory ordering parameter that help implement one-way barriers. Hence, it is not possible for the DPDK internal algorithms and applications to make use of the underlying CPU architecture effectively.
The DPDK community explored several options to solve this issue. The first option looked at relaxing the memory barriers used in the implementation of the above-mentioned APIs. It was soon realized that the type of memory ordering required depends on the algorithm and cannot be generalized. i.e., different algorithms may call the same API with different memory ordering. The second option looked at extending the existing APIs to take the memory ordering as one of the parameters while providing backward compatibility to existing APIs. This would however have resulted in writing large number of APIs to conform with the existing APIs.
Enter C11 Memory Model
The C language introduced the C11 memory model to address the above discussed differences in CPU architectures. It provides a memory model, that encompasses the behavior of widely used architectures, for multiple threads of execution to communicate with each other using shared memory. The C language supports this memory model through the atomic_xxx APIs provided in stdatomic.h. GCC and Clang compilers also provide __atomic_xxx built-ins. These APIs and built-ins allow the programmers to express the weak memory ordering inherent in their algorithms.
Several algorithms in DPDK were modified to use the C11 memory model on Arm. These changes didn’t affect the performance on x86 and at the same time, it improved the performance on weakly ordered memory model architectures like Arm. Thorough testing was done on several more algorithms to prove this further. Based on these results, the DPDK Tech Board voted unanimously to adopt the C11 memory model as the default memory model starting in the 20.08 release . This means that, all the patches submitted by the DPDK community for subsequent releases should use C11 memory model. In order to support this adoption, Arm has agreed to change the existing code to use C11 memory model.
The community discussed whether to use the __atomic_xxx built-ins or the atomic_xxx APIs provided via stdatomic.h. The __atomic_xxx built-ins require the memory order parameter in every built-in, requiring the programmer to make a conscious decision on the right memory order to use. They are supported by GCC, ICC and Clang compilers. The atomic_xxx APIs are not supported by compilers packaged with older versions of Linux distributions. After some debate it was decided to use the built-ins as there was no advantage in using the atomic APIs.
For x86 architecture, the __atomic_thread_fence(__ATOMIC_SEQ_CST) generates mfence instruction. DPDK implements the same barrier semantics with less overhead. Hence a decision was taken to introduce a wrapper rte_atomic_thread_fence. For x86 architecture, it calls the optimized implementation for __ATOMIC_SEQ_CST and calls __atomic_thread_fence for the rest. For rest of the architectures, it calls __atomic_thread_fence.
The community also decided to enforce this adoption using the check patch script. The script is updated to throw a warning if the patch uses the rte_smp_mb/rmb/wmb and rte_atomic_xxx APIs. All patch owners must fix such warning by using the __atomic_xxx built-ins. It is also the maintainers’ responsibility to look out for these warnings.
Even though Arm has agreed to change the existing code to use C11 memory model, contributions from the community are highly welcome. So far 12 libraries have been updated with several more still to be updated. Several race conditions and atomicity related bugs have been identified and fixed during this process.
The adoption of the C11 memory model has helped DPDK community to develop robust code that performs best on all supported architectures. The community has developed a very good understanding of the C11 memory model.
If anyone needs help in designing/coding their next algorithm using C11 memory model, please ask at firstname.lastname@example.org.
Special thanks to Ola Liljedahl, Philippe Robin, Ananyev Konstantin, David Christensen and David Marchand for reviewing this blog.
There are 34 new contributors (including authors, reviewers and testers).
Welcome to Andrew Boyer, Andrii Pypchenko, Ashish Sadanandan, Barry Cao, Dana Vardi, Dapeng Yu, Fabio Pricoco, Fei Chen, Francis Kelly, Fredrik A Lindgren, George Prekas, Jacek Bułatek, Jiawei Zhu, Kiran KN, Lingyu Liu, Louis Peens, Mateusz Pacuszka, Meir Levi, Milena Olech, Murphy Yang, Nalla Pradeep, Neel Patel, Odi Assli, Paolo Valerio, Peng He, Samik Gupta, Simon Ellmann, Somalapuram Amaranath, Subhi Masri, Szymon T Cudzilo, Tyler Retzlaff, Viacheslav Galaktionov, Wenjun Wu and Yongxin Liu.
Below is the percentage of commits per employer (with authors count):
Based on Reviewed-by and Acked-by tags, the top non-PMD reviewers are:
31 Ruifeng Wang <email@example.com> 29 Ferruh Yigit <firstname.lastname@example.org> 25 David Marchand <email@example.com> 23 Maxime Coquelin <firstname.lastname@example.org> 17 Honnappa Nagarahalli <email@example.com> 16 Ori Kam <firstname.lastname@example.org> 15 Jerin Jacob <email@example.com> 13 Konstantin Ananyev <firstname.lastname@example.org> 11 Viacheslav Ovsiienko <email@example.com>
The new features for 21.05 may be submitted until mid-March, in order to be reviewed and integrated before mid-April.
In DPDK 20.11, the DPDK community adopted ABI stable releases with the aim of easing the adoption of new features for DPDK users. If ABI stable releases is a new concept for you, or to simply refresh your memory, please read the previous post on Why is ABI Stability Important? The DPDK community put the policies and mechanisms in place to make ABI stability a reality from DPDK 19.11 onwards with a commitment to reviewing progress after a year. A year has now passed, so how did things go? The good news is that 100% backward compatibility was preserved through all the quarterly releases since DPDK 19.11.
100% compatibility means that a packet processing application built with DPDK 19.11, works just as well when upgraded to DPDK 20.08. This is crucially without necessitating an application rebuild, making for seamless upgrades. This is during a period when significant new features were added to DPDK, such as Cryptodev support for ChaCha20-Poly1305 AEAD for example. To really understand what DPDK ABI Stability means for a DPDK user, watch the demo …
The DPDK 20.11 release declares a new major ABI version v21 and begins a new period of backward compatibly with the DPDK 20.11 release. Kudos and thanks to all the maintainers and contributors whose hard work have made ABI stability a success.
To learn more about Microsoft, DPDK’s newest Gold member, and how they are working within the DPDK community, we sat down with Omar Cardona, Doug Stamper and Harini Ramakrishnan from the Core Networking team for a Q&A session. Read below to see how Microsoft collaborates with DPDK.
Can you tell us a little about your organization/department?
We are the Core Networking team in the Azure Edge + Platform (Azure E+P) group at Microsoft. Our organization has the charter to deliver platform innovation to support Microsoft Core OSes and bring services to modernize customer infrastructure. Our team is responsible for the L1-L7 networking stack including offloads, driver platform, transports, and HTTP across the cloud and edge. We are also responsible for SDN and container/Kubernetes networking. Key to our mission is a predictable, high bandwidth, low latency networking data plane that serves as the foundation for cloud and edge infrastructure and services.
Why is your organization adopting an open source approach?
Microsoft strongly believes in working with the open source communities on technological innovations to deliver transformational experiences. Multiple teams in our organization are contributing to and working with open source communities and standard bodies to innovate in the open, accelerating adoption and standardization. In fact, we have a sister team contributing to the Linux Kernel, and general Linux OS and tooling. They work across Microsoft Cloud and On-Prem deployments, having a long-standing partnership with the DPDK community.
We have heard from our ecosystem partners and customers a desire in aligning Windows with Linux and Open Source on networking architecture and implementation. Towards this end, it was a natural progression for us to collaborate with the DPDK community and align our data plane investments.
Why did you join DPDK and what sort of impact do you think DPDK has on the data plane?
Workloads today require an insatiable amount of network bandwidth with low latency performance. NICs are getting faster with 200GbE in the horizon, while single threaded performance of a CPU seems to be stagnating. Customers do not want to compromise their experience and are seamlessly moving workloads between the cloud and the edge deployments. The high packet rates of a cloud and a container world pose a significant challenge to the OS Kernel network stack, specifically for Virtual Networking and Network Function Virtualization. While the Windows network stack has evolved tremendously improving the cycles/per byte performance over the years, we are nearing the point of diminishing returns.
DPDK successfully by-passes the limitations of the in-kernel network stack, while also providing a path to user mode processing. The ecosystem and ISVs are observing that DPDK allows them to enter a different realm of network performance, even at the cost of dedicating the hardware to the appliance.
We believe that through our partnership with the DPDK community, we can bring this to more Windows users.
What do you see as the top benefits of being part of the DPDK community?
Significant effort has been underway over the last couple of years, to add Windows platform support to DPDK. Today, with v20.11 release, we have ~10 critical core libraries and drivers working on Windows platform. This would not have been possible without the support of our partners at Intel and Mellanox, under the guidance of the Techboard lead Thomas Monjalon. Not to forget critical contributions from DPDK end users and maintainers on memory management and stabilization. We would have likely not gotten the reach that we have today, but for the community.
There is a long journey ahead to integrate Windows Platform support to ensure that each new version of DPDK libraries and APIs will build and run successfully on Windows (without forking) as validated by testing performed by the community lab. Monumental efforts are required to bring these solutions to work seamlessly with the Windows ecosystem. We hope that by working with the DPDK community, we can learn from the collective expertise and leverage the performance standardization that DPDK brings. We are deeply interested in enabling a kernel-bypass technology on Windows for devices and appliances ranging from IOT, to Client, to Cloud Based Servers.
Aligning with the community on a high-performance packet processing model allows us to contribute and leverage mindshare, and co-engineer towards a common goal.
What sort of contributions has your team made –or plans to make — to the community,ecosystem through DPDK participation?
One of the primary contributions from Microsoft has been bringing the ecosystem together to collectively consider DPDK as a viable kernel bypass technology on Windows. In addition, we have begun our contributions with the generic NetUIO driver for Windows. We have contributions in the pipeline which will bring support for applications such as testpmd and tools. This is just the beginning; we are looking forward to working with the UNH lab in standardizing testing on Windows SKUs and bringing our ability to get the most out of Windows to the community. We are experimenting with Network Virtualization for VM and Container performance, and increasingly leveraging DPDK tooling for Network performance profiling.
What do you think sets DPDK apart from other industry alliances or organizations?
DPDK brings unparalleled and extreme focus on network performance. DPDK is very well organized, with good code structure and great documentation to simplify onboarding. It has a sensible multi-platform strategy, feasible for Windows inclusion. Specifically, the modularity and cohesion of the components and ease of combining the minimal necessary libraries to achieve application performance goals.
How will DPDK help your business?
Our team contributes to the multi-OS platform, foundational to operate and scale Microsoft’s businesses across Azure Cloud and Edge portfolio of products. Increasing application density and bandwidth requirements on our data plane, across Windows and Linux, necessitates an investment in a promising network stack bypass technology such as DPDK. DPDK’s user space poll-based, run-to-completion model allows for significant scaling of network performance in the face of stalled CPU speeds.
Working with the community on how DPDK lands on Windows Platform allows modernization of the network data path for future workloads. Additionally, a common architecture increases the overall quality of the drivers and consumers, benefiting all DPDK high volume platforms.
What advice would you give to someone considering joining DPDK or getting involved in open source?
The first step would be to subscribe to the firstname.lastname@example.org email list and observe how the community functions. It’s good to start small and identify an area of expertise to contribute to. Something as simple as reviewing and testing incoming patches adds value. Beyond that, participating by submitting patches, contributing to tool chains, adding CI tests are all valuable ways to contribute.
We are looking for contributions and expertise to bring DPDK Windows forward. Contribute patches under the guidelines, reference “dpdk-windows” in the email or write to email@example.com. We have a Windows working group that you can also join to learn how to help.
What do you want to tell those members of the community that are interested in Windows Platform support?
Test the DPDK libraries on Windows and share your feedback! Head over to the getting started guide. A general roadmap has been published, however, we want to hear from the community to further influence our roadmap. Encourage everyone to participate in the survey below:
The DPDK Project has had an active third quarter thus far with the DPDK Userspace Summit in September, release of 20.11 release (the biggest DPDK release yet!), and the Governing Board focused on 2021 planning and budgeting during an October 13 virtual meeting.
A release candidate (RC3) for 20.11 went out last week, and turned out to be one of the biggest releases to date for DPDK. Notable updates include:
We continue to seek additional DPDK reviewers and are looking for volunteers. If you are an active contributor to DPDK, we invite you to help review your fellow contributors’ patches; help us collectively improve DPDK releases.
DPDK Userspace Summit
Capping off a tumultuous 2020, the DPDK community came together virtually for the DPDK Userspace Summit, September 22-23rd. With over 550 registrants and 250 active participants from over 46 countries, the ecosystem came together for what was one of the largest DPDK Userspace Summits to date. We would like to extend a thank you to all the speakers, planning committee members, as well as and Arm for sponsoring the event. Technical Presentations and videos are available in the event archive: https://www.dpdk.org/event/dpdk-userspace-summit/
Happy 10 years, DPDK!
DPDK project celebrated it’s 10 year anniversary during the virtual summit, marking the occasion with a new celebratory webpage, with excerpts and pictures from across the community: https://www.dpdk.org/10th-anniversary/
Upcoming Events: With the prevailing impact of the global pandemic, DPDK continues to monitor and assess the viability of hosting an in person gathering; as of today, we have moved to a virtual event calendar and anticipate our next virtual gathering towards the end of Q1, 2021. Look forward to an event and CFP announcement soon.
Budget: The Board has updated the budget to reflect our real-time changes as a result of the impact of COVID-19 and is currently discussing 2021 budget priorities and alignment. We will provide an update once the preliminary budget has been made.
Lab: A sub-team has been working with the DPDK Community Lab team, including UNH-IOL, to expand the testing coverage provided by the Community Lab. We are happy to report that under the guidance of the Technical Steering Committee, the Testing Group has exceeded their targeted testing coverage goals of 40%, with over 56% Functional test coverage in the community lab. The Community Lab team continues to refine its practices and plans to expand in the coming months.
Member Outreach: No changes to membership this quarter. If you have any suggestions for companies or individuals that we should reach out to, please let us know.
Thank you for your continued engagement with the DPDK community! We hope everyone is staying healthy and keeping busy during these uncertain times.
There are 64 new contributors (including authors, reviewers and testers). Welcome to Aidan Goddard, Amit Bernstein, Andrey Vesnovaty, Artur Rojek, Benoît Ganne, Brandon Lo, Brian Johnson, Brian Poole, Christophe Grosse, Churchill Khangar, Conor Walsh, David Liu, Dawid Lukwinski, Diogo Behrens, Dongdong Liu, Franck Lenormand, Galazka Krzysztof, Guoyang Zhou, Haggai Eran, Harshitha Ramamurthy, Ibtisam Tariq, Ido Segev, Jay Jayatheerthan, Jiawen Wu, Jie Zhou, John Alexander, Julien Massonneau, Jørgen Østergaard Sloth, Khoa To, Li Zhang, Lingli Chen, Liu Tianjiao, Maciej Rabeda, Marcel Cornu, Mike Ximing Chen, Muthurajan Jayakumar, Nan Chen, Nick Connolly, Norbert Ciosek, Omkar Maslekar, Padraig Connolly, Piotr Bronowski, Przemyslaw Ciesielski, Qin Sun, Radha Mohan Chintakuntla, Rani Sharoni, Raveendra Padasalagi, Robin Zhang, RongQing Li, Shay Amir, Steve Yang, Steven Lariau, Tom Rix, Venkata Suresh Kumar P, Vijay Kumar Srivastava, Vikas Gupta, Vimal Chungath, Vipul Ashri, Wei Huang, Wei Ling, Weqaar Janjua, Yi Yang, Yogesh Jangra and Zhenghua Zhou.
Below is the breakout of commits by employer:
Based on Reviewed-by and Acked-by tags, the top non-PMD reviewers are:
128 Ferruh Yigit <firstname.lastname@example.org> 68 Bruce Richardson <email@example.com> 63 Andrew Rybchenko <firstname.lastname@example.org> 62 David Marchand <email@example.com> 53 Ruifeng Wang <firstname.lastname@example.org> 40 Konstantin Ananyev <email@example.com> 38 Ajit Khaparde <firstname.lastname@example.org> 37 Ori Kam <email@example.com> 33 Honnappa Nagarahalli <firstname.lastname@example.org>
New features for 21.02 may be submitted during the next 3 weeks, in order to be reviewed and integrated before mid-January. DPDK 21.02 should be small in order to release in early February: https://core.dpdk.org/roadmap#dates
Please share your roadmap.
Thanks to everyone who helped make this release happen – what a great way to wrap up 2020!
For the past three years, the DPDK developer community has honored some of the most outstanding contributions across the project over the past year. While things are quite different this year, we still wanted to take some time to call out some of the amazing success of our developers. While normally, the community meets together in person for the DPDK Userspace Summit (previous locations include Bordeaux, France and Dublin, Ireland) to share knowledge, collaborate on best practices and community alignment, this year’s event was held virtually due to the COVID-19 pandemic.
Not to break tradition, 2020 winners were still recognized during the event on September 22’s Opening Remarks by DPDK Board Chair Jim St. Leger. We were pleased to welcome even more attendees to this year’s event, given folks were able to join from their own homes.
Please join us in congratulating all of our nominees and winners!
A little more than 3 years ago, we set out to improve the qualities and capabilities of the continuous integration (CI) testing of the DPDK source code and new patches. Now, it’s not uncommon for testing to come last in the process, and sometimes it doesn’t receive it’s fair share of attention. Which of us hasn’t worked on a project, with the boss pushing off the testing efforts or milestones, in order to allow “just that bit more” focus or expansion on the features or customer wishes for the project? However, as DPDK has matured and is a key foundation for many applications and deployments, we need to ensure the stability and robustness of the codebase and incoming patches.
Enter the Community CI team, a virtual team of DPDK community members who meet regularly and help maintain all of the tooling that enables the testing activities. We’re working to better organize and support the testing resources available to the DPDK development teams. Today, there are 3 primary testing sources, running as TravisCI, Intel’s hosted lab, and the community lab hosted at the UNH-IOL. Each of the “labs” provides results back to the DPDK project’s patchwork tracking system, and tests different aspects of the “code,” ranging from compile and unit tests, to performance and functionality tests on bare-metal with real NICs and drivers.
We thought it might be worthwhile to spend a few words on a blog to highlight what testing is already running, what testing is under development, and finally, where do we see things going from here. Starting with the Travis CI systems, which are triggered from a github robot that is tracking commits on the main branch, running compile and unit testing across both x86 and aarch64 architectures, with cross compiles to 32-bit, all on top of the Ubuntu OS. The robot and CI jobs for this integration have been developed and maintained largely by our Redhat peers, so, shoutouts where they’re due, “great job guys!” You can check out the Travis CI builds here: https://travis-ci.com/github/DPDK
Next up, Intel’s hosted lab, where they are running compile testing across a number of different operating systems on x86 hardware. The operating systems range from Linux distros, to FreeBSD, and even Windows. Adding another dimension, the compilation is also checked using both Makefile and Meson. The system is tracking both the main branch and as well as providing results for patches, before their merge onto the main, and provides early feedback to the patch submitter if issues pop up from a patch, afterall, “who knew” that approach didn’t work over all of DPDK’s supported OSes (well, now you know). Intel’s support in providing both the hosted lab environment and keeping things running smoothly is a great help, “thanks Intel team!” You can see a sample of the Intel hosted lab output in patchworks here: https://patchwork.dpdk.org/patch/75413/
Lastly, for the currently operating testing, is the community lab, hosted at the UNH-IOL. Testing in the community lab is also tracking both the main branch and submitted patches, providing a combination of compile, unit testing, functional testing, and performance testing. The compile and unit testing also add one additional dimension, with some cooperation to downstream projects, such as Open vSwitch and SPDK, providing early detection of a patch breaking outside of DPDK directly. For the functional and performance testing, the lab hosts a number of “bare-metal” systems with NICs provided by DPDK member companies, including Broadcom, Intel, Mellanox, and NXP currently. The performance testing runs a throughput test on each NIC (in some cases there are multiple speeds or hardware variants included), checking for a drop in forwarding performance from a well known reference. Gold and Silver DPDK member companies are eligible to submit hardware for hosting in the community lab and are strongly encouraged to do so by the governing and tech boards of DPDK, as this directly expands the testing coverages. Functional testing is the newest and growing aspect within the community lab output, adding more tests running on “bare-metal”, based on the DTS (DPDK Test Suite) project. Aiming to provide majority coverage for the NIC features list. The community lab results also feed directly into the patchwork system, with additional information available on the lab portal as well: https://lab.dpdk.org/results/dashboard/
Looking to the future, there are two efforts currently underway. First, the robot that is tracking the github main branch is being extended to also drive builds on the OpenSUSE Build Service (OBS), further extending that coverage to more OS / build variants. Second is the expansion of the feature testing coverage, which is focused on extending and updating the DTS coverage, with that work driving back into the functional testing run in the community lab.
The Community CI team meets every other week, on Thursdays, at 1pm UTC. All DPDK community members and maintainers are welcome to join and participate in the discussions, just drop a line to email@example.com. We hope to see you there!