DPDK Governing board Update – July 2020

By Blog

The DPDK Project Governing board held a “virtual” meeting on July 13th, welcoming both Doug Stamper from Microsoft and Marvell’s new representative, Prasun Kapoor, to the board. Bruce Richardson (of Intel) rotated into the Technical Board representative seat, previously held by Thomas Monjalon (of Mellanox/NVIDIA), in Q3. Note, the Technical Board seat is rotated among Technical Board members on a quarterly basis.

It was noted that 20.08 RC2 went out this week, and it’s shaping up to be a big release! The DPDK project is looking for more code reviewers. If you are an active contributor to DPDK, we invite you to help review your fellow contributors’ patches; help us collectively improve DPDK releases (team work makes the dream work). We would like to thank the entire DPDK developer community who does the hard work that makes our project what it is.

The board recognized the ongoing efforts by Stephen Hemminger (Microsoft) and project maintainers to adopt more inclusive code language, with the replacement of terms like “Blacklist-Whitelist” and “Master-Slave.” Inclusive language aims to avoid expressions and terms that could be perceived as racist, sexist, ableist, or generally biased, prejudiced or demeaning to any particular group of people. DPDK joins other open source groups, including the Linux Kernel community, in working towards improving the use of inclusive language blocks. We know these changes require significant work. We are appreciative to the commitment of our entire community to support the transition to more inclusive terminology in the upcoming 20.11 release.

Additional updates from the Governing Board:

Updated DPDK Charter: Following up on our earlier summer update, the Governing Board, in conjunction with the Technical Board, concluded its months-long review and revision of the DPDK Charter, presenting it for ratification to the Governing Board at the meeting this week. The Governing Board unanimously approved the charter revisions in their entirety.

The edits bring the Charter up-to-date and serve as a current reflection of the project as it is today. The latest charter is available at https://www.dpdk.org/charter/, with changes noted in the table. Please take a look.

Events: We are moving solely to virtual events this year. The DPDK Summit Userspace event will take place virtually September 22-23. Registration is now live, and we encourage you to register early. Be on the lookout for agenda updates.

Budget: The Board has updated the budget to reflect our real-time changes as a result of the impact of COVID-19.

Lab: A sub-team has been working with the DPDK Community Lab team, including UNH-IOL. We are making good progress to properly resource, fund, and improve our lab’s community benefit.

Member Outreach: As mentioned earlier this summer, the board has also started a sub-committee that is looking to grow the DPDK Project’s membership. If you have any suggestions for companies or individuals that we should reach out to, please let us know.

Thank you for your continued engagement with the DPDK community! We hope everyone is staying healthy and keeping busy during these uncertain times

DPDK Community Update – Summer 2020

By Blog

We hope you are keeping safe and healthy, as we all adjust to the new normal we find ourselves in, with coronavirus and social unrest around us.  We’re proud of the leadership of our technical community to tackle our own issues and moving to more inclusive language, removing divisive terminology (e.g master/slave) that does not reflect our community values. Thank you for your leadership and work. It is greatly appreciated.

The DPDK project Governing Board, Technical Board, Marketing and other working groups have been busy over the past couple months. 

The Governing Board has held two meetings since the lock-downs went into effect, in April and early June, reassessing the project budget, marketing and event plans, while the Technical Board continued to focus on the technical priorities of DPDK, with the recent release of 20.05. Our community continues their stellar reputation of delivering our releases on time. Well done!

As we will not be able to meet in person for some time, and unable to provide the usual Governing Board readout on stage at one of the DPDK Summits, we wanted to provide a short written update, that we will continue on a regular basis moving forward. And we do apologize for not providing more timely governing board activity updates in the past. We will strive to improve and keep the communication open and timely. Do not hesitate to reach out should you have any questions. 

DPDK Welcomes New Members

Our community thrives through the work of the community and the financial support enabled by our members. We are pleased to share recent additions to the DPDK Project membership. The DPDK Project welcomes Microsoft as our newest Gold member, with Mr. Douglas Stamper joining the Governing Board, bringing his expertise with DPDK and the Windows ecosystem. Additionally, AMD joined as a Silver member. Welcome to both companies!

         

 

DPDK Userspace Summit 2020 Moves to a Virtual Format

Due to the prevailing coronavirus situation across the globe and related travel restrictions, we made the difficult decision to transition the DPDK Userspace Summit to a virtual experience, happening September 22-23. The Call for proposals and registration are now open. Note that the CFP closes July 12. Please consider submitting an abstract! We hope you will join us! 

DPDK will continue to monitor the prevailing situation and assess the feasibility of safely holding physical community gatherings. Please bear with us. 

  • China and Japan events are on hold. It is currently looking unlikely we can make in-person events happen in 2020. 
  • The US DPDK Summit event is pushed out to Q1 of 2021.

Strategic and Financial Update

It is of note that the DPDK project is in a financially sound position to see 2020 through, under the watchful gaze of the Governing Board’s Financial committee, chaired by Rashid Khan from Red Hat. 

The Governing Board reviewed and approved an updated 2020 budget including our anticipated 2020 expenses of $316k, deferring unused event allocations to our reserve fund, in anticipation of 2021 events and the Community Lab expansion. 

The Governing Board has noted the importance of member retention and recruitment, particularly in these uncertain times, with the formation of a Strategic Outreach working group, comprised of a sub-group of Governing Board members, tasked with ensuring the identity, focus, and charter of DPDK is clearly captured, published and communicated, along with a clear and compelling membership value proposition. 

Towards this end, the Governing Board in conjunction with the Technical Board, has spent the last month reviewing and updating the DPDK Charter, which has not seen any significant changes over the past 3 years. The edits will bring the Charter up to date and serve as a current reflection of the project as it is today.    

The DPDK Project has also launched a new project analytics dashboard, providing insights into development activity and contributions. We continue to be an active healthy community. Please try out the new tool. Let us know if you feel the data is accurate and reflects the community breadth of contributors.

Community Lab Expansion 

As Jim and Thomas shared in an earlier message to the community, we have been evaluating the value and role of the DPDK Community Lab, hosted at University of New Hampshire’s InterOperability Lab, (UNH IoL).  Upon the recommendations of the Technical Board, the lab continues to play a key role in regression testing. The Governing Board has approved up to $35,000 in additional investment to expand the test coverage and support provided to the community through the Community Lab. We anticipate increasing the depth and breadth of test cases available to the community.  

The Community Lab meets on a bi-weekly basis and is open to the DPDK community. We would encourage you to participate –  reach out to ci@dpdk.org to be added to the meeting calendar invite.

DPDK Marketing Activities

The marketing working group has a new project whitepaper and companion video series, to be published in the coming month.  Keep an eye out for it!

In 2020, DPDK celebrates 10 years as an active open source project from its formative years at Intel. We have come a long way since then, becoming the world’s leading open source packet processing tool kit.  Stay tuned for a fun look back across the past 10 years. 

Expect to see some exciting things happening over the next couple months. Stay tuned for more! And please let us know if you would like more information on any of these topics. 

 

DPDK’s 20.05 Release is Here!

By Blog

A new DPDK release, 20.05,  is available here: https://fast.dpdk.org/rel/dpdk-20.05.tar.xz

It was quite a big release cycle, with:

  •         1304 commits from 189 authors
  •         1983 files changed, 145825 insertions(+), 29147 deletions(-)

There are no plans (yet) to start a maintenance branch for 20.05.

This version, as the previous one (20.02), is ABI-compatible with 19.11.

Below are some new features, grouped by category.

  • General
    • packet processing graph
    • ring synchronisation modes for VMs and containers
    • RCU API for deferred resource reclamation
    • telemetry rework
    • low overhead tracing framework
    • GCC 10 support
  • Networking
    •  flowaging API
    • driver for Intel Foxville I225
  • Cryptography
    • ChaCha20-Poly1305 crypto algorithm
    • event mode in the example application ipsec-secgw
  •  Baseband
    • 5G driver for Intel FPGA-based N3000

More details in the release notes: http://doc.dpdk.org/guides/rel_notes/release_20_05.html

There are 69 new contributors (including authors, reviewers and testers):

Welcome to Andrea Arcangeli, Asim Jamshed, Cheng Peng, Christos Ricudis, Dave Burley, Dong Zhou, Dongsheng Rong, Eugeny Parshutin, Evan Swanson, Fady Bader, Farah Smith, Guy Tzalik, Hailin Xu, Igor Russkikh, JP Lee, Jakub Neruda, James Fox, Jianwei Mei, Jiawei Wang, Jun W Zhou, Juraj Linkeš, Karra Satwik, Kishore Padmanabha, Lihong Ma, Lijian Zhang, Linsi Yuan, Louise Kilheeney, Lukasz Wojciechowski, Mairtin o Loingsigh, Martin Spinler, Matteo Croce, Michael Haeuptle, Mike Baucom, Mit Matelske, Mohsin Shaikh, Muhammad Ahmad, Muhammad Bilal, Nannan Lu, Narcisa Vasile, Niall Power, Peter Spreadborough, Przemyslaw Patynowski, Qi Fu, Real Valiquette, Rohit Raj, Roland Qi, Sarosh Arif, Satheesh Paul, Shahaji Bhosle, Sharon Haroni, Sivaprasad Tummala, Souvik Dey, Steven Webster, Tal Shnaiderman, Tasnim Bashar, Vadim Podovinnikov, Venky Venkatesh, Vijaya Mohan Guvva, Vu Pham, Wentao Cui, Xi Zhang, Xiaoxiao Zeng, Xinfeng Zhao, Yash Sharma, Yu Jiang, Zalfresso-Jundzillo, Zhihong Peng, Zhimin Huang, and Zhiwei He.

Here is a breakout of patches per company contribution:

The new features for 20.08 may be submitted during the next 17 days.  DPDK 20.08 should be released in early August, in a tight schedule:  http://core.dpdk.org/roadmap#dates

Thanks to the entire DPDK community, and happy birthday to our RTE_MAGIC!

Community Update from the DPDK Board & Tech Chairs

By Blog

Greetings DPDK Project Community:

 We hope you are managing okay and remaining safe and healthy as the coronavirus crisis has thrown the world into a frenzy.  While many of us are used to working remotely and using online collaborative tools, we are having to adapt to new circumstances at home. It’s hard on everyone.  

 The DPDK Governing Board, Technical Board, and Marketing Committee have been hard at work locking down our final 2020 budget, crafting plans around worldwide DPDK Summit events, and exploring hosting a broader data plane projects developer event.  However, upcoming public gatherings and events have been cancelled or postponed and are no longer an option in the near term. DPDK has not yet migrated to virtual events. For now, our plans around DPDK Summit events are in flux:

  •  China and Japan events are on hold. As things develop further we will see if we can make something happen, perhaps in Q3 or Q4. 
  • The US DPDK Summit event is  pushed out to Q1 of 2021. The team made this decision before the pandemic emerged in order to create some  calendar space between the Summit and Userspace events. We have targeted the second half of September for the Userspace event, continuing to host it in Bordeaux.  As of now this is still our intent. But as we’re sure you can understand, the situation remains fluid until the pandemic subsides. Please bear with us.

 We have also continued to explore options to increase the value and role of the DPDK Community Lab that is currently hosted at the University of New Hampshire, InterOperability Lab, also known as UNH/IoL. We’d like to acknowledge Lincoln Lavoie and team’s continued efforts here, especially as he remains in the lab to triage hardware issues. However, we’ve struggled to find community volunteers to step up and lead the lab effort.  We are now looking at alternatives including a minimum set of activities and regression tests that the Technical Board recommends we maintain. Other efforts to expand the use of the lab will likely require additional budget. We are trying to converge on a solution. Help is still welcome should anyone want to get more involved.

The community continues to be amazing.  This year we’ve delivered our 20.02 release with few time accommodations.  Work is underway for the 20.05 release with several new features and improvements. https://core.dpdk.org/roadmap/ 20.08 and 20.11 will follow.  We are confident the community collaboration and execution capability will serve us well during these times.  We are not expecting too much delay in releases. Of note, while most git maintainers do have a back-up in place, please let us know if you are able to help fill in any gaps. 

Overall we want to ensure you prioritize yourself and your family; your personal health and safety are paramount to us. Rest assured the DPDK community is here for you – whether it be  help on a features, code, or guidance on how to handle a challenging home dilemma, please do reach out. The DPDK family is strong and tremendously supportive. We are glad to be part of it.

Stay safe and healthy.

All the best,

Thomas and Jim

 

DPDK 20.02 Release Now Available

By Blog

The latest DPDK release, 20.02, is now available:  https://fast.dpdk.org/rel/dpdk-20.02.tar.xz

The statistics show  a post-LTS ABI-compatible release:

        820 commits from 156 authors

        876 files changed, 59941 insertions(+), 22565 deletions(-)

There are currently no plans (yet) to start a maintenance branch for 20.02. This release and the next ones (20.05, 20.08) are ABI-compatible with 19.11.

Below are some new features, grouped by category.

General:

  •         ABI check tooling
  •         FreeBSD 13 support
  •         Linux kernel modules disabled in default build
  •         API wait until equal
  •         use CPU lcore with ID > total max
  •         ring with configurable element size
  •         pool of mbufs with pinned external buffers

Networking:

  •         Pensando ionic driver
  •         Marvell driver for OCTEON TX2 end point
  •         Mellanox vDPA driver
  •         rte_flow API for DSCP
  •         rte_flow API for L2TPv3 over IP

Cryptography:

  •          Marvell OCTEON TX2 inline IPsec
  •          CPU crypto based on new libraries (intel-ipsec-mb and AArch64cryptolib)
  •          synchronous CPU crypto API
  •         more algorithms: ECDSA, ECPM

Applications:

  •         event mode in l3fwd example

More details are available in the  technical release notes: http://doc.dpdk.org/guides/rel_notes/release_20_02.html

There are 69 new contributors (including authors, reviewers and testers):

Welcome to Abed Kamaluddin, Abhishek Marathe, Adam Ludkiewicz, Adrian Podlawski, Aleksandr Loktionov, Alexander Kozyrev, Alfredo Cardigliano, Andrew Pinski, Apeksha Gupta, Balakrishna Bhamidipati, Carolyn Wyborny, Chandu Babu N, Cheng Jiang, Chengchang Tang, Damian Milosek, Dariusz Chaberski, Dariusz Jagus, Dexuan Cui, Dmitry Kozlyuk, Donald Lee, Dzmitry Sautsa, Eugenio Pérez, Fang TongHao, Gal Cohen, Gargi Sau, Girish Nandibasappa, Ilja Van Sprundel, Itsuro Oda, Jaroslaw Gawin, Jörg Thalheim, Kiran Patil, Krzysztof Galazka, Kumar Amber, Li Feng, Lijun Ou, Mahipal Challa, Manish Chopra, Marcin Formela, Martyna Szapar, Mateusz Rusinski, Michael Baum, Michal Litwicki, Michal Swiatkowski, Narcisa Vasile, Niclas Storm, Pandi Kumar Maharajan, Piotr Azarewicz, Piotr Kwapulinski, Piotr Pietruszewski, Prateek Agarwal, Praveen Shetty, Rafael Ávila de Espíndola, Ricardo Roldan, Robert Konklewski, Satananda Burla, Savinay Dharmappa, Scott Wasson, Selwin Sebastian, Shannon Nelson, Shiri Kuzin, Sunil Pai G, Sylwia Wnuczko, Thomas Faivre, Tomasz Konieczny, Vitaliy Mysak, Xuan Ding, Xuan Li, Xueming Zhang, and Yisen Zhuang.

Below is the percentage of patches per company:

Based on Reviewed-by and Acked-by tags, the top reviewers are:

     76     Viacheslav Ovsiienko <viacheslavo@mellanox.com>

     56     Qi Zhang <qi.z.zhang@intel.com>

     47     Xiaolong Ye <xiaolong.ye@intel.com>

     46     Akhil Goyal <akhil.goyal@nxp.com>

     44     Ferruh Yigit <ferruh.yigit@intel.com>

     43     Matan Azrad <matan@mellanox.com>

     43     Jerin Jacob <jerinj@marvell.com>

     38     Beilei Xing <beilei.xing@intel.com>

     35     Qiming Yang <qiming.yang@intel.com>

     34     Ajit Khaparde <ajit.khaparde@broadcom.com>

     33     Maxime Coquelin <maxime.coquelin@redhat.com>

     31     Ori Kam <orika@mellanox.com>

     25     Gavin Hu <gavin.hu@arm.com>

     24     David Marchand <david.marchand@redhat.com>

     23     Konstantin Ananyev <konstantin.ananyev@intel.com>

The new features for 20.05 may be submitted during the next 21 days, in order to be reviewed and integrated before mid-April. DPDK 20.05 should be released on 20-05-20.  http://core.dpdk.org/roadmap#dates The schedule is pushed because of the special worldwide context. Some future release cycles will have to be shorter.

Thanks to the entire DPDK community!

 

DPDK 2020 Events Update

By Blog

Our 2020 event planning is taking longer than usual, given the current situation with the Novel CoronaVirus. We are working diligently to solidify plans for events throughout 2020 and will have information soon.

In the interim, please read the below excerpt from the Linux Foundation’s official response regarding the situation and potential impact across all Linux Foundation events:

The Linux Foundation is continuously monitoring the Novel Coronavirus situation and is committed to ensuring the safety of our event participants and staff at our upcoming events. We will be following all recommended guidelines from the Centers for Disease Control (CDC) and the World Health Organization (WHO) as the situation progresses.

The Linux Foundation wants to assure our community and all event participants that their health and well being are of the utmost importance to us. We will continue to regularly check the latest official information and updates from all major health agencies leading up to our events and will make any necessary changes based on that information. We recognize that much is unknown at this time, and will strive to make decisions based on facts rather than fear. We encourage our event participants to do the same. To all those who are being affected, please accept our sincerest sympathies.

Please read the full update, which includes specific recommended action for onsite events, here: https://events.linuxfoundation.org/2020/02/12/novel-coronavirus-update/

DPDK 19.11, with new ABI Policy and API rework, is Now Available

By Blog

A new major release is available:
https://fast.dpdk.org/rel/dpdk-19.11.tar.xz

The statistics are quite impressive:
1611 commits from 199 authors
1914 files changed, 164270 insertions(+), 44609 deletions(-)

The branch 19.11 should be supported for at least two years,
making it recommended for system integration and deployment.
The maintainer of this new LTS is Luca Boccassi.

The new major ABI version is 20.
The next releases 20.02, 20.05 and 20.08 will be ABI compatible with 19.11.

Below are some new features, grouped by category.
General:
– new ABI policy and versioning
– API reworks in preparation of 1 year without ABI breakage
– mempool objects not crossing pages
– dynamic mbuf layout
– eBPF arm64 JIT
– LTO build
Networking:
– Hisilicon hns3 driver
– NXP PFE driver
– virtio packed ring optimizations
– ethdev hairpin API
– ethdev flow tag API
– ethdev packet type range API
– ethdev LRO packet size API
– RIB/FIB libraries
– KNI with VA
Cryptography:
– Marvell Nitrox driver
– Marvell OCTEON TX2 crypto driver
– session-less asymmetric crypto
– IPsec SAD API
Applications:
– IOAT example
– l2fwd-event example
– several examples are removed

More details in the release notes:
http://doc.dpdk.org/guides/rel_notes/release_19_11.html

There are 70 new contributors (including authors, reviewers and testers).
Welcome to Abhishek Sachan, Adrian Moreno, Alvin Zhang,
Amaranath Somalapuram, Anand Sunkad, Andy Gospodarek, Anna Lukin,
Antara Ganesh Kolar, Bing Zhao, Bo Chen, Chengchang Tang,
Chengwen Feng, Chenxu Di, Chunsong Feng, Cristian Bidea, Doug Dziggel,
Feifei Wang, Guinan Sun, Hao Chen, Heinrich Kuhn, Hongbo Zheng,
Huisong Li, Igor Chauskin, Ivan Ilchenko, Jeremy Plsek, Jerry Hao OS,
Jiaqi Min, Jin Yu, Junfeng Guo, Junyu Jiang, Kommula Shiva Shankar,
Lin Li, Lu Qiuwen, Lunyuan Cui, Manish Tomar, Marcin Baran,
Md Fahad Iqbal Polash, Michael Pfeiffer, Michael Shamis,
Min Hu (Connor), Min Wang (Jushui), Mitch Williams, Nagadheeraj Rottela,
Paul Atkins, Pawel Modrak, Phanendra Vukkisala, Priyanka Jain,
Rahul Shah, Rajesh Ravi, Scott W Taylor, Shougang Wang,
Shuki Katzenelson, Subrahmanyam Nilla, Sucharitha Sarananaga,
Sunila Sahu, Sylvain Rodon, Tony Nguyen, Vakul Garg, Venkat Duvvuru,
Venkateshwarlu Nalla, Wangyu (Eric), Wei Hu (Xavier), Xiaobing Zhang,
Xiaofeng Deng, Xiaoyun Wang, Xun Ni, Yahui Cao,  Yilong Lv, Yu Zhang,
and Zengmo Gao.

Below is the number of patches per company (with authors count):
559     Intel (73)
209     Mellanox (16)
173     Marvell (23)
170     NXP (10)
119     Broadcom (7)
77     Red Hat (8)
61     Huawei (9)
49     OKTET Labs (4)
41     ARM (7)
35     Microsoft (3)
21     6WIND (6)
17     Solarflare (1)
13     Chelsio (1)
12     Cisco (3)
9     IBM (1)
5     AMD (1)

Based on Reviewed-by and Acked-by tags, the top reviewers are:
154     Xiaolong Ye, Intel
118     Ferruh Yigit, Intel
113     Akhil Goyal, NXP
85     Matan Azrad, Mellanox
78     Maxime Coquelin, Redhat
63     Ajit Khaparde, Broadcom
62     Viacheslav Ovsiienko, Mellanox
58     Qi Zhang, Intel
58     Jerin Jacob, Marvel
52     Qiming Yang, Intel
48     Somnath Kotur, Broadcom
44     Thomas Monjalon, Mellanox
39     Hemant Agrawal, NXP
36     Nipun Gupta, NXP
33     David Marchand, Redhat
32     Bruce Richardson, Intel
32     Andrew Rybchenko, SolarFlare
31     Gavin Hu, Arm
29     Luca Boccassi, Debian
29     Konstantin Ananyev, Intel
25     Ori Kam Mellanox
24     Anatoly Burakov, Intel
22     Olivier Matz, 6 Wind

The new features for 20.02 may be submitted during the next 12 days,
in order to be reviewed and integrated before mid-January.
DPDK 20.02 should be released on Valentine’s Day.
http://core.dpdk.org/roadmap#dates

Thanks everyone, and happy Thanksgiving to our American friends!

How Red Hat is Using DPDK

By Blog

 

A Q&A with Rashid Khan, DPDK Board Member and Director, Networking at Red Hat

Rashid Khan is the Director of Networking development at Red Hat. His team’s responsibilities include upstream and downstream development of Linux kernel networking, DPDK, OVS OVN, NFV, MPTCP, ebpf XDP. Before joining Red Hat, Rashid managed the Linux team at Broadcom consumer products division. There he was in charge of board support drivers team, silicon bring up, and application deployment. Prior to that Rashid has approximately 20 years of networking, telco, voice over IP, signal processing experience at various  startups and huge conglomerates. 

What is Red Hat’s history with DPDK? How long have you been involved in the community?

Red Hat is an open-source Linux company that participates in the development of the Linux operating system, and provides support to its customers via Red Hat’s Linux distribution (RHEL). About four years ago, Red Hat noticed a lot of traction for the Data Plane Development Kit (DPDK) project to get traffic from network interface cards into virtual machines (VMs) and containers, with better packet speeds and development, resulting in faster time-to-market. Allowing new features and new NICs to work with Linux kernel is a bit hard, because Linux maintains the principle of not breaking things that are already working. DPDK offered a new approach; it did not come with the same constraints, and it offered an easier way to move network traffic, add new drivers for new cards, and add more features quickly.

DPDK provided a method for getting packets faster to VMs, and by helping accelerate networking for its virtualization products.  

What was the primary use case that got you started with DPDK?

Red Hat  primarily uses DPDK with Open vSwitch to get packets routed within servers from Network Interface Cards (NIC), to VMs (we leverage DPDK as the foundation for our NFV stack).

Can you talk a little more about some of these (and other) use cases?

Sure, there are a few I can touch on:

  • Telcos are used to custom SW/HW builds from vendors,  but because of increasing consumer demand for new features without a willingness to pay more, there is a strong need to go with commodity hardware and open-source software, and to put applications on top through virtual machines and containers. So right now, we’re investing heavily in solving those issues and getting network traffic to VMs, containers and pods as efficiently as possible. 
  • We’re also investing time and resources into the hardening of DPDK itself, taking it to the next level if you will. We’ve worked really hard to get our CI tests working at the University of New Hampshire’s Interoperability Lab (UNH-IOL). We can detect regressions in DPDK-OVS and alert the submitters of patches. HW vendors collaborating very closely with us on this include Intel, Mellanox, Netronome, and Broadcom. 

So what are some of the challenges DPDK is experiencing, and how do you think they can be addressed?

  • DPDK is being used extensively everywhere, and that is great. Now we need to collaborate to take it to the next level of maturity. For example, the community needs to collaborate to find a way to continue adding advanced features without compromising the stability of existing HW and existing features. Stability of Long Term Releases (LTS) has to be solidified.  DPDK will have to adapt to cater to that — number of releases, duration of releases, integration with other products and projects will have to be evaluated. But the community is up for the challenge! There are enough senior/experienced people who are putting the right steps in place to solve for this. 
  • Another challenge is that there are a lot of consumers of DPDK, but not very many active participants. Red Hat, Mellanox, and Intel are really stepping up and putting in resources, but we’d love to see more organizations participating. There are folks working full time on LTSs, and Red Hat has actually hired head count just for DPDK: we have about a half-dozen people focused on DPDK upstream, plus 8-9 working with DPDK for product integration.
  • There’s also the challenge of debuggability.  The more we add to DPDK the harder it is to debug systems running DPDK.  And larger customers demand bug fixes plus an absolute root cause for each bug. That requires debuggability of every layer and complete predictability. In some cases adding debug code moves the problem, or may even alleviate the problem. Unfortunately, that is not enough for large telco customers. They need to know the root cause of the problem, and how a patch fixes that problem directly. More work is needed to get DPDK to that level.  As products mature, more of these “Day 2” problems need to be addressed. The same is the case with DPDK; the time has come to get the project to the next level.
  • The next challenge for DPDK is to integrate with smart(er) NICs to enable HW offloads so main motherboard CPUs inside the servers are not doing the heavy lifting of the packet movement. We are working with multiple HW vendors to address that challenge. 

How can someone get involved in DPDK? 

There are so many ways to get involved! There are no impediments in joining — it’s easy to participate. You can start by making DPDK work for your specific hardware, enhance CI activities, collaborate on UNH lab, patch review, or just start by participating in online meetings. There are no barriers to entry – joining mailing lists and conference calls is a good first step. Does not have to be a large commitment in the beginning, you can start with just a few hours a week. 

And again –  we see a lot more consumers of DPDK than we do participants. Some of the biggest benefits of participation are to (selfishly) safeguard your own hardware and products, but to also have a voice in the overall direction of the project. And the more diverse the community, the better – that’s the spirit of open source. 

Why is ABI Stability Important?

By Blog

Ray Kinsella & Thomas Monjalon

Most open-source software projects follow distinct life cycle patterns as they evolve from the genesis of a idea, all the way through to a mature stable well-defined project. You can see this evolution reflected in all aspects of the project. Upstreaming rules for instance, are usually permissive in the early days and then become gradually more conservative and risk adverse over time. Similarly, you expect lots of bugs in the early days, then things becomes more tested and stable over time, and so on.

This life-cycle is also evident in a project’s management of it’s application binary interface, usually shortened to ABI. Before continuing, two things should be pointed out;

  1. If you are bit fuzzy on what I mean by ABI, also know as “binary compatibility”, you should read through the wikipedia entry on it.
  2. We have made quite a bit of use of the dataset’s and tools from the ABI Laboratory project. It’s worth following the links to that project below to review the data-sets, we annotated those links with a †.

In the early days of a project, ABI changes are rarely given any consideration as you are usually way too busy trying to change the world! Then time goes on and the project becomes popular. The more users join the project, and become dependent on it, the more time you spend making sure that you don’t break their software.

At the extreme end of this cycle are very mature projects like the Linux Kernel. Linus Torvalds explains Linux’s commitment to maintaining a stable ABI in his own words.

“We care about user-space interfaces to an insane degree. We go to extreme lengths to maintain even badly designed or unintentional interfaces. Breaking user programs simply isn’t acceptable.” – Linus Torvalds, 2005

There are a few distinct patterns of ABI management, between fledgling and very mature project’s like Linux.

Patterns of ABI management

In some software libraries, an evolutionary pattern is very clear, that is they follow the common pattern of an unstable ABI in their early day’s and then after some period of settling they declare a 1.0 release and their ABI is more or less set in stone from that point on-wards. The GStreamer (†) project is good example of this form evolution.

Some software projects, particularly programming languages and operating systems, by virtue of being governed either by a strict set of standards and/or the requirement to offer very strong guarantees about backward compatibility, change very rarely. That is, they have a well-defined ABI from the start and it very rarely changes thereafter, with is no period of stabilization as such. LibC++ (†) and GlibC (†) are good examples of these sorts of projects.

Other software projects will support a stable ABI version for some period of time, usually months or more often years, with planned periodic ABI breakages to introduce new features or to facilitate re-factoring. These breakage’s are often timed to coordinate with the lifecycle of consuming software such as operating system distributions (Debian etc) or higher level applications. LibAV (†) and ffmpeg (†) are clear examples of this kind of project.

Finally, some software project’s by virtue of a design philosophy or simply because they are that bit earlier in their lifecycle, choose to offer fewer guarantees of ABI compatibility. DPDK and Boost projects are both good examples of this kind of project.

Why are ABI Breakages considered bad?

Modern software ecosystems are built on a hard commitment that binary interfaces will be carefully managed. When this commitment does not hold things fall apart rapidly, with applications failing to start or randomly crashing.

Imagine a world in which there was no guarantee that applications installed from an ‘app’ store or repository would just work, imagine how frustrating that might be for users? Today this all just works and we take for granted that behind the scenes, engineers are working hard to ensure that updates don’t break ABIs and therefore do not break applications. However many will remember a time when such guarantees either didn’t exist or were hard to enforce.

And the consequences? Naturally defensive behaviours will follow, developers will start to statically link with their dependencies and become slow about picking up the latest version of those dependencies as being too risky. In the worst case, some developers might start looking for another ecosystem that doesn’t break their code and their application quite so much.

And this worst case, happens more often than you might think …

Miguel De Icaza is one the fathers’ of the GNOME Project, one of the best desktop environments for Linux. For a few years in the late 90s and early 00s, it looked like Linux desktop distributions based on GNOME had a real shot with competing with Microsoft Windows to become a popular desktop operating system. However despite all the excitement, huge community effort, and commercial support from major Linux vendors, it never really happened. Miguel explains why in his blog post What Killed the Linux Desktop (worth a read).

“Backwards compatibility is not a sexy problem. It is not even remotely an interesting problem to solve. Nobody wants to do that work, everyone wants to innovate, and be responsible for the next big feature in Linux.

So Linux was left with idealists that wanted to design the best possible system without having to worry about boring details like support and backwards compatibility.

Meanwhile, you can still run the 2001 Photoshop that came when XP was launched on Windows 8. And you can still run your old OSX apps on Mountain Lion…” – Miguel De Icaza, 2012

It’s a sobering message, ABI stability done right helped contribute to Linux’s vast success as an operating system and done wrong, it hurt it’s popularity as desktop operating system. The risk is for project’s with an unstable ABI is clear, eventually your consumers will start looking for something else that doesn’t break their code quite so much.

DPDK’s has had an ABI policy committing the community to preserving the DPDK ABI since 2015.

Note that the above process for ABI deprecation should not be undertaken lightly. ABI stability is extremely important for downstream consumers of the DPDK, especially when distributed in shared object form. Every effort should be made to preserve the ABI whenever possible. The ABI should only be changed for significant reasons, such as performance enhancements. ABI breakage due to changes such as reorganizing public structure fields for aesthetic or readability purposes should be avoided. – DPDK ABI Policy, 19.08

The DPDK ABI policy encourages contributors to be mindful of consumers when making ABI changes. What is changing in DPDK, is that this policy is now evolving to offer consumers more guarantees of future compatibility.

How we are changing DPDK?

Recently the 6th revision of a new ABI policy was posted to the community, intended to start the process of moving DPDK out of the last category of projects described above and providing it’s consumers with more certainty around future ABI compatibility. This policy has been approved in principle by the DPDK Technical Board and will become the new policy following the DPDK 19.11 LTS release.

The intention is to continue to provide DPDK’s consumers the best possible features and performance for building dataplane applications, now with the addition of clearer upgrade paths and a stronger commitment to backward compatibility.

The change will mean that DPDK will now follow a pattern similar to that described for the LibAV and FFMpeg projects above. A pattern that is characterized by periods of ABI stability with periodic ABI breakages to facilitate change. In this way, a DPDK “major” ABI version will be declared aligned with the DPDK LTS release, and then supported in all the quarterly release over the year following the LTS release.

What does this mean for Contributors?

At a high-level, it means that the community will become more deliberate about how the DPDK ABI is managed. Any new features will be required to maintain existing interfaces between LTS releases, and in general ABI changes will receive more scrutiny than has been the case in the past.

To be absolutely clear, the DPDK ABI can change while ABI compatibility is being maintained.

This means that the DPDK community will guarantee, that applications built and dynamically linked against the most recent LTS release will continue to work, without requiring a rebuild, through the quarterly releases for the year following the LTS release. The DPDK ABI can and will continue to evolve during this period, adding great new features and improvements, so long as ABI compatibility with the LTS release is preserved.

Changes that are so dramatic as to require an ABI compatibility breakage will now need to wait until the next ABI breakage window at the next LTS release.

How do we prepare for this change?

The initial period of ABI stability will run for one year following the v19.11 release. This was designed to minimize disruption to the community, as most contributors are targeting the LTS release with their changes. Currently ABI breakage windows are aligned with LTS releases, meaning that even in the worst case event of an unavoidable ABI breaking change, the impact of the new policy will be minimal.

This has been designed to start to familiarize the community with the requirements of ABI compatibility, while still permitting ABI breakages for the next LTS release. The ABI policy will then be reviewed after this initial year, with the intention of lengthening the stability period and period between ABI breakages to two years.

If you are interested in the next level of detail of how the new policy will work, can review the patch.

Second-Annual DPDK Community Awards Recognize Hard Work & Collaboration

By Blog

 

The DPDK developer community convenes each year at the DPDK “Userspace” event to share knowledge, discuss best practices, and further align the community. During the event we take some time to reflect upon successes of the past year and recognize some of the amazing contributions from across the project – DPDK Community Awards.

 

 

Winners were recognized September 19 at the DPDK Userspace event in Bordeaux, France Details about each award category and its winners appear below. 

Please join us in congratulating all of our nominees and winners!

DPDK Top Ambassador:  Tim O’Driscoll
Tim has been involved with DPDK for many years on the management and marketing side, and will be recognized by most from his previous  “world tours” attending DPDK events in China, India, Europe and the States. He’s very approachable and always has a friendly word, which when coupled with his knowledge of DPDK and packet processing, makes him an excellent DPDK ambassador.

Innovation: Arm team
Congratulations to the Arm team for their innovative work on RCU, MCS Lock, and Ticket Lock. They spend a lot of time working to improve DPDK’s performance and find new alternatives. In just two years, they have improved performance and introduced new alternatives.

Contribution (Code): David Marchand
A long-time contributor to DPDK, David  is meticulous and accurate, taking the time to ensure the results of his work are perfect. His recent contributions DPDK (log fixes, Environment Abstraction Layer (EAL)  fixes, test fixes) show his love for a job well done. He’s also very friendly and funny guy, and it’s a real pleasure to work with him.

Contribution (Maintainer): Andrew Rybchenko
Andrew does very valuable work in DPDK; he’s of a great help for reviews in the mempool and mbuf areas. He’s rigorous and always brings a constructive and positive perspective.  He has the maintainer mindset, and ensures the APIs fit every case now and in the future.

Contribution (Operations):  Ferruh Yigit
Ferruh spends a lot of time ensuring things are progressing. He’s the engine  powering a fast=paced release and improvement cycle. In addition to his numerous contributions (code, reviews, doc, etc.) , Ferruh does an essential job in pinging people to ensure that everything is getting done on time.

Contribution (Testing): Aaron Conole
Aaron has been a regular DPDK contributor since 2015. In the  few past years, he has invested a lot of effort in setting up Travis CI for the DPDK project, working closely with the UNH Interop Lab team. He is especially good with testing powers in Open Source workflows. On top of that, Aaron is always available to provide constructive feedback, be it about technical topics, or about beer brewing!