All Posts By

DPDK

DPDK Community Update – Summer 2020

By Blog

We hope you are keeping safe and healthy, as we all adjust to the new normal we find ourselves in, with coronavirus and social unrest around us.  We’re proud of the leadership of our technical community to tackle our own issues and moving to more inclusive language, removing divisive terminology (e.g master/slave) that does not reflect our community values. Thank you for your leadership and work. It is greatly appreciated.

The DPDK project Governing Board, Technical Board, Marketing and other working groups have been busy over the past couple months. 

The Governing Board has held two meetings since the lock-downs went into effect, in April and early June, reassessing the project budget, marketing and event plans, while the Technical Board continued to focus on the technical priorities of DPDK, with the recent release of 20.05. Our community continues their stellar reputation of delivering our releases on time. Well done!

As we will not be able to meet in person for some time, and unable to provide the usual Governing Board readout on stage at one of the DPDK Summits, we wanted to provide a short written update, that we will continue on a regular basis moving forward. And we do apologize for not providing more timely governing board activity updates in the past. We will strive to improve and keep the communication open and timely. Do not hesitate to reach out should you have any questions. 

DPDK Welcomes New Members

Our community thrives through the work of the community and the financial support enabled by our members. We are pleased to share recent additions to the DPDK Project membership. The DPDK Project welcomes Microsoft as our newest Gold member, with Mr. Douglas Stamper joining the Governing Board, bringing his expertise with DPDK and the Windows ecosystem. Additionally, AMD joined as a Silver member. Welcome to both companies!

         

 

DPDK Userspace Summit 2020 Moves to a Virtual Format

Due to the prevailing coronavirus situation across the globe and related travel restrictions, we made the difficult decision to transition the DPDK Userspace Summit to a virtual experience, happening September 22-23. The Call for proposals and registration are now open. Note that the CFP closes July 12. Please consider submitting an abstract! We hope you will join us! 

DPDK will continue to monitor the prevailing situation and assess the feasibility of safely holding physical community gatherings. Please bear with us. 

  • China and Japan events are on hold. It is currently looking unlikely we can make in-person events happen in 2020. 
  • The US DPDK Summit event is pushed out to Q1 of 2021.

Strategic and Financial Update

It is of note that the DPDK project is in a financially sound position to see 2020 through, under the watchful gaze of the Governing Board’s Financial committee, chaired by Rashid Khan from Red Hat. 

The Governing Board reviewed and approved an updated 2020 budget including our anticipated 2020 expenses of $316k, deferring unused event allocations to our reserve fund, in anticipation of 2021 events and the Community Lab expansion. 

The Governing Board has noted the importance of member retention and recruitment, particularly in these uncertain times, with the formation of a Strategic Outreach working group, comprised of a sub-group of Governing Board members, tasked with ensuring the identity, focus, and charter of DPDK is clearly captured, published and communicated, along with a clear and compelling membership value proposition. 

Towards this end, the Governing Board in conjunction with the Technical Board, has spent the last month reviewing and updating the DPDK Charter, which has not seen any significant changes over the past 3 years. The edits will bring the Charter up to date and serve as a current reflection of the project as it is today.    

The DPDK Project has also launched a new project analytics dashboard, providing insights into development activity and contributions. We continue to be an active healthy community. Please try out the new tool. Let us know if you feel the data is accurate and reflects the community breadth of contributors.

Community Lab Expansion 

As Jim and Thomas shared in an earlier message to the community, we have been evaluating the value and role of the DPDK Community Lab, hosted at University of New Hampshire’s InterOperability Lab, (UNH IoL).  Upon the recommendations of the Technical Board, the lab continues to play a key role in regression testing. The Governing Board has approved up to $35,000 in additional investment to expand the test coverage and support provided to the community through the Community Lab. We anticipate increasing the depth and breadth of test cases available to the community.  

The Community Lab meets on a bi-weekly basis and is open to the DPDK community. We would encourage you to participate –  reach out to ci@dpdk.org to be added to the meeting calendar invite.

DPDK Marketing Activities

The marketing working group has a new project whitepaper and companion video series, to be published in the coming month.  Keep an eye out for it!

In 2020, DPDK celebrates 10 years as an active open source project from its formative years at Intel. We have come a long way since then, becoming the world’s leading open source packet processing tool kit.  Stay tuned for a fun look back across the past 10 years. 

Expect to see some exciting things happening over the next couple months. Stay tuned for more! And please let us know if you would like more information on any of these topics. 

 

DPDK’s 20.05 Release is Here!

By Blog

A new DPDK release, 20.05,  is available here: https://fast.dpdk.org/rel/dpdk-20.05.tar.xz

It was quite a big release cycle, with:

  •         1304 commits from 189 authors
  •         1983 files changed, 145825 insertions(+), 29147 deletions(-)

There are no plans (yet) to start a maintenance branch for 20.05.

This version, as the previous one (20.02), is ABI-compatible with 19.11.

Below are some new features, grouped by category.

  • General
    • packet processing graph
    • ring synchronisation modes for VMs and containers
    • RCU API for deferred resource reclamation
    • telemetry rework
    • low overhead tracing framework
    • GCC 10 support
  • Networking
    •  flowaging API
    • driver for Intel Foxville I225
  • Cryptography
    • ChaCha20-Poly1305 crypto algorithm
    • event mode in the example application ipsec-secgw
  •  Baseband
    • 5G driver for Intel FPGA-based N3000

More details in the release notes: http://doc.dpdk.org/guides/rel_notes/release_20_05.html

There are 69 new contributors (including authors, reviewers and testers):

Welcome to Andrea Arcangeli, Asim Jamshed, Cheng Peng, Christos Ricudis, Dave Burley, Dong Zhou, Dongsheng Rong, Eugeny Parshutin, Evan Swanson, Fady Bader, Farah Smith, Guy Tzalik, Hailin Xu, Igor Russkikh, JP Lee, Jakub Neruda, James Fox, Jianwei Mei, Jiawei Wang, Jun W Zhou, Juraj Linkeš, Karra Satwik, Kishore Padmanabha, Lihong Ma, Lijian Zhang, Linsi Yuan, Louise Kilheeney, Lukasz Wojciechowski, Mairtin o Loingsigh, Martin Spinler, Matteo Croce, Michael Haeuptle, Mike Baucom, Mit Matelske, Mohsin Shaikh, Muhammad Ahmad, Muhammad Bilal, Nannan Lu, Narcisa Vasile, Niall Power, Peter Spreadborough, Przemyslaw Patynowski, Qi Fu, Real Valiquette, Rohit Raj, Roland Qi, Sarosh Arif, Satheesh Paul, Shahaji Bhosle, Sharon Haroni, Sivaprasad Tummala, Souvik Dey, Steven Webster, Tal Shnaiderman, Tasnim Bashar, Vadim Podovinnikov, Venky Venkatesh, Vijaya Mohan Guvva, Vu Pham, Wentao Cui, Xi Zhang, Xiaoxiao Zeng, Xinfeng Zhao, Yash Sharma, Yu Jiang, Zalfresso-Jundzillo, Zhihong Peng, Zhimin Huang, and Zhiwei He.

Here is a breakout of patches per company contribution:

The new features for 20.08 may be submitted during the next 17 days.  DPDK 20.08 should be released in early August, in a tight schedule:  http://core.dpdk.org/roadmap#dates

Thanks to the entire DPDK community, and happy birthday to our RTE_MAGIC!

Community Update from the DPDK Board & Tech Chairs

By Blog

Greetings DPDK Project Community:

 We hope you are managing okay and remaining safe and healthy as the coronavirus crisis has thrown the world into a frenzy.  While many of us are used to working remotely and using online collaborative tools, we are having to adapt to new circumstances at home. It’s hard on everyone.  

 The DPDK Governing Board, Technical Board, and Marketing Committee have been hard at work locking down our final 2020 budget, crafting plans around worldwide DPDK Summit events, and exploring hosting a broader data plane projects developer event.  However, upcoming public gatherings and events have been cancelled or postponed and are no longer an option in the near term. DPDK has not yet migrated to virtual events. For now, our plans around DPDK Summit events are in flux:

  •  China and Japan events are on hold. As things develop further we will see if we can make something happen, perhaps in Q3 or Q4. 
  • The US DPDK Summit event is  pushed out to Q1 of 2021. The team made this decision before the pandemic emerged in order to create some  calendar space between the Summit and Userspace events. We have targeted the second half of September for the Userspace event, continuing to host it in Bordeaux.  As of now this is still our intent. But as we’re sure you can understand, the situation remains fluid until the pandemic subsides. Please bear with us.

 We have also continued to explore options to increase the value and role of the DPDK Community Lab that is currently hosted at the University of New Hampshire, InterOperability Lab, also known as UNH/IoL. We’d like to acknowledge Lincoln Lavoie and team’s continued efforts here, especially as he remains in the lab to triage hardware issues. However, we’ve struggled to find community volunteers to step up and lead the lab effort.  We are now looking at alternatives including a minimum set of activities and regression tests that the Technical Board recommends we maintain. Other efforts to expand the use of the lab will likely require additional budget. We are trying to converge on a solution. Help is still welcome should anyone want to get more involved.

The community continues to be amazing.  This year we’ve delivered our 20.02 release with few time accommodations.  Work is underway for the 20.05 release with several new features and improvements. https://core.dpdk.org/roadmap/ 20.08 and 20.11 will follow.  We are confident the community collaboration and execution capability will serve us well during these times.  We are not expecting too much delay in releases. Of note, while most git maintainers do have a back-up in place, please let us know if you are able to help fill in any gaps. 

Overall we want to ensure you prioritize yourself and your family; your personal health and safety are paramount to us. Rest assured the DPDK community is here for you – whether it be  help on a features, code, or guidance on how to handle a challenging home dilemma, please do reach out. The DPDK family is strong and tremendously supportive. We are glad to be part of it.

Stay safe and healthy.

All the best,

Thomas and Jim

 

DPDK 20.02 Release Now Available

By Blog

The latest DPDK release, 20.02, is now available:  https://fast.dpdk.org/rel/dpdk-20.02.tar.xz

The statistics show  a post-LTS ABI-compatible release:

        820 commits from 156 authors

        876 files changed, 59941 insertions(+), 22565 deletions(-)

There are currently no plans (yet) to start a maintenance branch for 20.02. This release and the next ones (20.05, 20.08) are ABI-compatible with 19.11.

Below are some new features, grouped by category.

General:

  •         ABI check tooling
  •         FreeBSD 13 support
  •         Linux kernel modules disabled in default build
  •         API wait until equal
  •         use CPU lcore with ID > total max
  •         ring with configurable element size
  •         pool of mbufs with pinned external buffers

Networking:

  •         Pensando ionic driver
  •         Marvell driver for OCTEON TX2 end point
  •         Mellanox vDPA driver
  •         rte_flow API for DSCP
  •         rte_flow API for L2TPv3 over IP

Cryptography:

  •          Marvell OCTEON TX2 inline IPsec
  •          CPU crypto based on new libraries (intel-ipsec-mb and AArch64cryptolib)
  •          synchronous CPU crypto API
  •         more algorithms: ECDSA, ECPM

Applications:

  •         event mode in l3fwd example

More details are available in the  technical release notes: http://doc.dpdk.org/guides/rel_notes/release_20_02.html

There are 69 new contributors (including authors, reviewers and testers):

Welcome to Abed Kamaluddin, Abhishek Marathe, Adam Ludkiewicz, Adrian Podlawski, Aleksandr Loktionov, Alexander Kozyrev, Alfredo Cardigliano, Andrew Pinski, Apeksha Gupta, Balakrishna Bhamidipati, Carolyn Wyborny, Chandu Babu N, Cheng Jiang, Chengchang Tang, Damian Milosek, Dariusz Chaberski, Dariusz Jagus, Dexuan Cui, Dmitry Kozlyuk, Donald Lee, Dzmitry Sautsa, Eugenio Pérez, Fang TongHao, Gal Cohen, Gargi Sau, Girish Nandibasappa, Ilja Van Sprundel, Itsuro Oda, Jaroslaw Gawin, Jörg Thalheim, Kiran Patil, Krzysztof Galazka, Kumar Amber, Li Feng, Lijun Ou, Mahipal Challa, Manish Chopra, Marcin Formela, Martyna Szapar, Mateusz Rusinski, Michael Baum, Michal Litwicki, Michal Swiatkowski, Narcisa Vasile, Niclas Storm, Pandi Kumar Maharajan, Piotr Azarewicz, Piotr Kwapulinski, Piotr Pietruszewski, Prateek Agarwal, Praveen Shetty, Rafael Ávila de Espíndola, Ricardo Roldan, Robert Konklewski, Satananda Burla, Savinay Dharmappa, Scott Wasson, Selwin Sebastian, Shannon Nelson, Shiri Kuzin, Sunil Pai G, Sylwia Wnuczko, Thomas Faivre, Tomasz Konieczny, Vitaliy Mysak, Xuan Ding, Xuan Li, Xueming Zhang, and Yisen Zhuang.

Below is the percentage of patches per company:

Based on Reviewed-by and Acked-by tags, the top reviewers are:

     76     Viacheslav Ovsiienko <viacheslavo@mellanox.com>

     56     Qi Zhang <qi.z.zhang@intel.com>

     47     Xiaolong Ye <xiaolong.ye@intel.com>

     46     Akhil Goyal <akhil.goyal@nxp.com>

     44     Ferruh Yigit <ferruh.yigit@intel.com>

     43     Matan Azrad <matan@mellanox.com>

     43     Jerin Jacob <jerinj@marvell.com>

     38     Beilei Xing <beilei.xing@intel.com>

     35     Qiming Yang <qiming.yang@intel.com>

     34     Ajit Khaparde <ajit.khaparde@broadcom.com>

     33     Maxime Coquelin <maxime.coquelin@redhat.com>

     31     Ori Kam <orika@mellanox.com>

     25     Gavin Hu <gavin.hu@arm.com>

     24     David Marchand <david.marchand@redhat.com>

     23     Konstantin Ananyev <konstantin.ananyev@intel.com>

The new features for 20.05 may be submitted during the next 21 days, in order to be reviewed and integrated before mid-April. DPDK 20.05 should be released on 20-05-20.  http://core.dpdk.org/roadmap#dates The schedule is pushed because of the special worldwide context. Some future release cycles will have to be shorter.

Thanks to the entire DPDK community!

 

DPDK 2020 Events Update

By Blog

Our 2020 event planning is taking longer than usual, given the current situation with the Novel CoronaVirus. We are working diligently to solidify plans for events throughout 2020 and will have information soon.

In the interim, please read the below excerpt from the Linux Foundation’s official response regarding the situation and potential impact across all Linux Foundation events:

The Linux Foundation is continuously monitoring the Novel Coronavirus situation and is committed to ensuring the safety of our event participants and staff at our upcoming events. We will be following all recommended guidelines from the Centers for Disease Control (CDC) and the World Health Organization (WHO) as the situation progresses.

The Linux Foundation wants to assure our community and all event participants that their health and well being are of the utmost importance to us. We will continue to regularly check the latest official information and updates from all major health agencies leading up to our events and will make any necessary changes based on that information. We recognize that much is unknown at this time, and will strive to make decisions based on facts rather than fear. We encourage our event participants to do the same. To all those who are being affected, please accept our sincerest sympathies.

Please read the full update, which includes specific recommended action for onsite events, here: https://events.linuxfoundation.org/2020/02/12/novel-coronavirus-update/

DPDK 19.11, with new ABI Policy and API rework, is Now Available

By Blog

A new major release is available:
https://fast.dpdk.org/rel/dpdk-19.11.tar.xz

The statistics are quite impressive:
1611 commits from 199 authors
1914 files changed, 164270 insertions(+), 44609 deletions(-)

The branch 19.11 should be supported for at least two years,
making it recommended for system integration and deployment.
The maintainer of this new LTS is Luca Boccassi.

The new major ABI version is 20.
The next releases 20.02, 20.05 and 20.08 will be ABI compatible with 19.11.

Below are some new features, grouped by category.
General:
– new ABI policy and versioning
– API reworks in preparation of 1 year without ABI breakage
– mempool objects not crossing pages
– dynamic mbuf layout
– eBPF arm64 JIT
– LTO build
Networking:
– Hisilicon hns3 driver
– NXP PFE driver
– virtio packed ring optimizations
– ethdev hairpin API
– ethdev flow tag API
– ethdev packet type range API
– ethdev LRO packet size API
– RIB/FIB libraries
– KNI with VA
Cryptography:
– Marvell Nitrox driver
– Marvell OCTEON TX2 crypto driver
– session-less asymmetric crypto
– IPsec SAD API
Applications:
– IOAT example
– l2fwd-event example
– several examples are removed

More details in the release notes:
http://doc.dpdk.org/guides/rel_notes/release_19_11.html

There are 70 new contributors (including authors, reviewers and testers).
Welcome to Abhishek Sachan, Adrian Moreno, Alvin Zhang,
Amaranath Somalapuram, Anand Sunkad, Andy Gospodarek, Anna Lukin,
Antara Ganesh Kolar, Bing Zhao, Bo Chen, Chengchang Tang,
Chengwen Feng, Chenxu Di, Chunsong Feng, Cristian Bidea, Doug Dziggel,
Feifei Wang, Guinan Sun, Hao Chen, Heinrich Kuhn, Hongbo Zheng,
Huisong Li, Igor Chauskin, Ivan Ilchenko, Jeremy Plsek, Jerry Hao OS,
Jiaqi Min, Jin Yu, Junfeng Guo, Junyu Jiang, Kommula Shiva Shankar,
Lin Li, Lu Qiuwen, Lunyuan Cui, Manish Tomar, Marcin Baran,
Md Fahad Iqbal Polash, Michael Pfeiffer, Michael Shamis,
Min Hu (Connor), Min Wang (Jushui), Mitch Williams, Nagadheeraj Rottela,
Paul Atkins, Pawel Modrak, Phanendra Vukkisala, Priyanka Jain,
Rahul Shah, Rajesh Ravi, Scott W Taylor, Shougang Wang,
Shuki Katzenelson, Subrahmanyam Nilla, Sucharitha Sarananaga,
Sunila Sahu, Sylvain Rodon, Tony Nguyen, Vakul Garg, Venkat Duvvuru,
Venkateshwarlu Nalla, Wangyu (Eric), Wei Hu (Xavier), Xiaobing Zhang,
Xiaofeng Deng, Xiaoyun Wang, Xun Ni, Yahui Cao,  Yilong Lv, Yu Zhang,
and Zengmo Gao.

Below is the number of patches per company (with authors count):
559     Intel (73)
209     Mellanox (16)
173     Marvell (23)
170     NXP (10)
119     Broadcom (7)
77     Red Hat (8)
61     Huawei (9)
49     OKTET Labs (4)
41     ARM (7)
35     Microsoft (3)
21     6WIND (6)
17     Solarflare (1)
13     Chelsio (1)
12     Cisco (3)
9     IBM (1)
5     AMD (1)

Based on Reviewed-by and Acked-by tags, the top reviewers are:
154     Xiaolong Ye, Intel
118     Ferruh Yigit, Intel
113     Akhil Goyal, NXP
85     Matan Azrad, Mellanox
78     Maxime Coquelin, Redhat
63     Ajit Khaparde, Broadcom
62     Viacheslav Ovsiienko, Mellanox
58     Qi Zhang, Intel
58     Jerin Jacob, Marvel
52     Qiming Yang, Intel
48     Somnath Kotur, Broadcom
44     Thomas Monjalon, Mellanox
39     Hemant Agrawal, NXP
36     Nipun Gupta, NXP
33     David Marchand, Redhat
32     Bruce Richardson, Intel
32     Andrew Rybchenko, SolarFlare
31     Gavin Hu, Arm
29     Luca Boccassi, Debian
29     Konstantin Ananyev, Intel
25     Ori Kam Mellanox
24     Anatoly Burakov, Intel
22     Olivier Matz, 6 Wind

The new features for 20.02 may be submitted during the next 12 days,
in order to be reviewed and integrated before mid-January.
DPDK 20.02 should be released on Valentine’s Day.
http://core.dpdk.org/roadmap#dates

Thanks everyone, and happy Thanksgiving to our American friends!

How Red Hat is Using DPDK

By Blog

 

A Q&A with Rashid Khan, DPDK Board Member and Director, Networking at Red Hat

Rashid Khan is the Director of Networking development at Red Hat. His team’s responsibilities include upstream and downstream development of Linux kernel networking, DPDK, OVS OVN, NFV, MPTCP, ebpf XDP. Before joining Red Hat, Rashid managed the Linux team at Broadcom consumer products division. There he was in charge of board support drivers team, silicon bring up, and application deployment. Prior to that Rashid has approximately 20 years of networking, telco, voice over IP, signal processing experience at various  startups and huge conglomerates. 

What is Red Hat’s history with DPDK? How long have you been involved in the community?

Red Hat is an open-source Linux company that participates in the development of the Linux operating system, and provides support to its customers via Red Hat’s Linux distribution (RHEL). About four years ago, Red Hat noticed a lot of traction for the Data Plane Development Kit (DPDK) project to get traffic from network interface cards into virtual machines (VMs) and containers, with better packet speeds and development, resulting in faster time-to-market. Allowing new features and new NICs to work with Linux kernel is a bit hard, because Linux maintains the principle of not breaking things that are already working. DPDK offered a new approach; it did not come with the same constraints, and it offered an easier way to move network traffic, add new drivers for new cards, and add more features quickly.

DPDK provided a method for getting packets faster to VMs, and by helping accelerate networking for its virtualization products.  

What was the primary use case that got you started with DPDK?

Red Hat  primarily uses DPDK with Open vSwitch to get packets routed within servers from Network Interface Cards (NIC), to VMs (we leverage DPDK as the foundation for our NFV stack).

Can you talk a little more about some of these (and other) use cases?

Sure, there are a few I can touch on:

  • Telcos are used to custom SW/HW builds from vendors,  but because of increasing consumer demand for new features without a willingness to pay more, there is a strong need to go with commodity hardware and open-source software, and to put applications on top through virtual machines and containers. So right now, we’re investing heavily in solving those issues and getting network traffic to VMs, containers and pods as efficiently as possible. 
  • We’re also investing time and resources into the hardening of DPDK itself, taking it to the next level if you will. We’ve worked really hard to get our CI tests working at the University of New Hampshire’s Interoperability Lab (UNH-IOL). We can detect regressions in DPDK-OVS and alert the submitters of patches. HW vendors collaborating very closely with us on this include Intel, Mellanox, Netronome, and Broadcom. 

So what are some of the challenges DPDK is experiencing, and how do you think they can be addressed?

  • DPDK is being used extensively everywhere, and that is great. Now we need to collaborate to take it to the next level of maturity. For example, the community needs to collaborate to find a way to continue adding advanced features without compromising the stability of existing HW and existing features. Stability of Long Term Releases (LTS) has to be solidified.  DPDK will have to adapt to cater to that — number of releases, duration of releases, integration with other products and projects will have to be evaluated. But the community is up for the challenge! There are enough senior/experienced people who are putting the right steps in place to solve for this. 
  • Another challenge is that there are a lot of consumers of DPDK, but not very many active participants. Red Hat, Mellanox, and Intel are really stepping up and putting in resources, but we’d love to see more organizations participating. There are folks working full time on LTSs, and Red Hat has actually hired head count just for DPDK: we have about a half-dozen people focused on DPDK upstream, plus 8-9 working with DPDK for product integration.
  • There’s also the challenge of debuggability.  The more we add to DPDK the harder it is to debug systems running DPDK.  And larger customers demand bug fixes plus an absolute root cause for each bug. That requires debuggability of every layer and complete predictability. In some cases adding debug code moves the problem, or may even alleviate the problem. Unfortunately, that is not enough for large telco customers. They need to know the root cause of the problem, and how a patch fixes that problem directly. More work is needed to get DPDK to that level.  As products mature, more of these “Day 2” problems need to be addressed. The same is the case with DPDK; the time has come to get the project to the next level.
  • The next challenge for DPDK is to integrate with smart(er) NICs to enable HW offloads so main motherboard CPUs inside the servers are not doing the heavy lifting of the packet movement. We are working with multiple HW vendors to address that challenge. 

How can someone get involved in DPDK? 

There are so many ways to get involved! There are no impediments in joining — it’s easy to participate. You can start by making DPDK work for your specific hardware, enhance CI activities, collaborate on UNH lab, patch review, or just start by participating in online meetings. There are no barriers to entry – joining mailing lists and conference calls is a good first step. Does not have to be a large commitment in the beginning, you can start with just a few hours a week. 

And again –  we see a lot more consumers of DPDK than we do participants. Some of the biggest benefits of participation are to (selfishly) safeguard your own hardware and products, but to also have a voice in the overall direction of the project. And the more diverse the community, the better – that’s the spirit of open source. 

Why is ABI Stability Important?

By Blog

Ray Kinsella & Thomas Monjalon

Most open-source software projects follow distinct life cycle patterns as they evolve from the genesis of a idea, all the way through to a mature stable well-defined project. You can see this evolution reflected in all aspects of the project. Upstreaming rules for instance, are usually permissive in the early days and then become gradually more conservative and risk adverse over time. Similarly, you expect lots of bugs in the early days, then things becomes more tested and stable over time, and so on.

This life-cycle is also evident in a project’s management of it’s application binary interface, usually shortened to ABI. Before continuing, two things should be pointed out;

  1. If you are bit fuzzy on what I mean by ABI, also know as “binary compatibility”, you should read through the wikipedia entry on it.
  2. We have made quite a bit of use of the dataset’s and tools from the ABI Laboratory project. It’s worth following the links to that project below to review the data-sets, we annotated those links with a †.

In the early days of a project, ABI changes are rarely given any consideration as you are usually way too busy trying to change the world! Then time goes on and the project becomes popular. The more users join the project, and become dependent on it, the more time you spend making sure that you don’t break their software.

At the extreme end of this cycle are very mature projects like the Linux Kernel. Linus Torvalds explains Linux’s commitment to maintaining a stable ABI in his own words.

“We care about user-space interfaces to an insane degree. We go to extreme lengths to maintain even badly designed or unintentional interfaces. Breaking user programs simply isn’t acceptable.” – Linus Torvalds, 2005

There are a few distinct patterns of ABI management, between fledgling and very mature project’s like Linux.

Patterns of ABI management

In some software libraries, an evolutionary pattern is very clear, that is they follow the common pattern of an unstable ABI in their early day’s and then after some period of settling they declare a 1.0 release and their ABI is more or less set in stone from that point on-wards. The GStreamer (†) project is good example of this form evolution.

Some software projects, particularly programming languages and operating systems, by virtue of being governed either by a strict set of standards and/or the requirement to offer very strong guarantees about backward compatibility, change very rarely. That is, they have a well-defined ABI from the start and it very rarely changes thereafter, with is no period of stabilization as such. LibC++ (†) and GlibC (†) are good examples of these sorts of projects.

Other software projects will support a stable ABI version for some period of time, usually months or more often years, with planned periodic ABI breakages to introduce new features or to facilitate re-factoring. These breakage’s are often timed to coordinate with the lifecycle of consuming software such as operating system distributions (Debian etc) or higher level applications. LibAV (†) and ffmpeg (†) are clear examples of this kind of project.

Finally, some software project’s by virtue of a design philosophy or simply because they are that bit earlier in their lifecycle, choose to offer fewer guarantees of ABI compatibility. DPDK and Boost projects are both good examples of this kind of project.

Why are ABI Breakages considered bad?

Modern software ecosystems are built on a hard commitment that binary interfaces will be carefully managed. When this commitment does not hold things fall apart rapidly, with applications failing to start or randomly crashing.

Imagine a world in which there was no guarantee that applications installed from an ‘app’ store or repository would just work, imagine how frustrating that might be for users? Today this all just works and we take for granted that behind the scenes, engineers are working hard to ensure that updates don’t break ABIs and therefore do not break applications. However many will remember a time when such guarantees either didn’t exist or were hard to enforce.

And the consequences? Naturally defensive behaviours will follow, developers will start to statically link with their dependencies and become slow about picking up the latest version of those dependencies as being too risky. In the worst case, some developers might start looking for another ecosystem that doesn’t break their code and their application quite so much.

And this worst case, happens more often than you might think …

Miguel De Icaza is one the fathers’ of the GNOME Project, one of the best desktop environments for Linux. For a few years in the late 90s and early 00s, it looked like Linux desktop distributions based on GNOME had a real shot with competing with Microsoft Windows to become a popular desktop operating system. However despite all the excitement, huge community effort, and commercial support from major Linux vendors, it never really happened. Miguel explains why in his blog post What Killed the Linux Desktop (worth a read).

“Backwards compatibility is not a sexy problem. It is not even remotely an interesting problem to solve. Nobody wants to do that work, everyone wants to innovate, and be responsible for the next big feature in Linux.

So Linux was left with idealists that wanted to design the best possible system without having to worry about boring details like support and backwards compatibility.

Meanwhile, you can still run the 2001 Photoshop that came when XP was launched on Windows 8. And you can still run your old OSX apps on Mountain Lion…” – Miguel De Icaza, 2012

It’s a sobering message, ABI stability done right helped contribute to Linux’s vast success as an operating system and done wrong, it hurt it’s popularity as desktop operating system. The risk is for project’s with an unstable ABI is clear, eventually your consumers will start looking for something else that doesn’t break their code quite so much.

DPDK’s has had an ABI policy committing the community to preserving the DPDK ABI since 2015.

Note that the above process for ABI deprecation should not be undertaken lightly. ABI stability is extremely important for downstream consumers of the DPDK, especially when distributed in shared object form. Every effort should be made to preserve the ABI whenever possible. The ABI should only be changed for significant reasons, such as performance enhancements. ABI breakage due to changes such as reorganizing public structure fields for aesthetic or readability purposes should be avoided. – DPDK ABI Policy, 19.08

The DPDK ABI policy encourages contributors to be mindful of consumers when making ABI changes. What is changing in DPDK, is that this policy is now evolving to offer consumers more guarantees of future compatibility.

How we are changing DPDK?

Recently the 6th revision of a new ABI policy was posted to the community, intended to start the process of moving DPDK out of the last category of projects described above and providing it’s consumers with more certainty around future ABI compatibility. This policy has been approved in principle by the DPDK Technical Board and will become the new policy following the DPDK 19.11 LTS release.

The intention is to continue to provide DPDK’s consumers the best possible features and performance for building dataplane applications, now with the addition of clearer upgrade paths and a stronger commitment to backward compatibility.

The change will mean that DPDK will now follow a pattern similar to that described for the LibAV and FFMpeg projects above. A pattern that is characterized by periods of ABI stability with periodic ABI breakages to facilitate change. In this way, a DPDK “major” ABI version will be declared aligned with the DPDK LTS release, and then supported in all the quarterly release over the year following the LTS release.

What does this mean for Contributors?

At a high-level, it means that the community will become more deliberate about how the DPDK ABI is managed. Any new features will be required to maintain existing interfaces between LTS releases, and in general ABI changes will receive more scrutiny than has been the case in the past.

To be absolutely clear, the DPDK ABI can change while ABI compatibility is being maintained.

This means that the DPDK community will guarantee, that applications built and dynamically linked against the most recent LTS release will continue to work, without requiring a rebuild, through the quarterly releases for the year following the LTS release. The DPDK ABI can and will continue to evolve during this period, adding great new features and improvements, so long as ABI compatibility with the LTS release is preserved.

Changes that are so dramatic as to require an ABI compatibility breakage will now need to wait until the next ABI breakage window at the next LTS release.

How do we prepare for this change?

The initial period of ABI stability will run for one year following the v19.11 release. This was designed to minimize disruption to the community, as most contributors are targeting the LTS release with their changes. Currently ABI breakage windows are aligned with LTS releases, meaning that even in the worst case event of an unavoidable ABI breaking change, the impact of the new policy will be minimal.

This has been designed to start to familiarize the community with the requirements of ABI compatibility, while still permitting ABI breakages for the next LTS release. The ABI policy will then be reviewed after this initial year, with the intention of lengthening the stability period and period between ABI breakages to two years.

If you are interested in the next level of detail of how the new policy will work, can review the patch.

Second-Annual DPDK Community Awards Recognize Hard Work & Collaboration

By Blog

 

The DPDK developer community convenes each year at the DPDK “Userspace” event to share knowledge, discuss best practices, and further align the community. During the event we take some time to reflect upon successes of the past year and recognize some of the amazing contributions from across the project – DPDK Community Awards.

 

 

Winners were recognized September 19 at the DPDK Userspace event in Bordeaux, France Details about each award category and its winners appear below. 

Please join us in congratulating all of our nominees and winners!

DPDK Top Ambassador:  Tim O’Driscoll
Tim has been involved with DPDK for many years on the management and marketing side, and will be recognized by most from his previous  “world tours” attending DPDK events in China, India, Europe and the States. He’s very approachable and always has a friendly word, which when coupled with his knowledge of DPDK and packet processing, makes him an excellent DPDK ambassador.

Innovation: Arm team
Congratulations to the Arm team for their innovative work on RCU, MCS Lock, and Ticket Lock. They spend a lot of time working to improve DPDK’s performance and find new alternatives. In just two years, they have improved performance and introduced new alternatives.

Contribution (Code): David Marchand
A long-time contributor to DPDK, David  is meticulous and accurate, taking the time to ensure the results of his work are perfect. His recent contributions DPDK (log fixes, Environment Abstraction Layer (EAL)  fixes, test fixes) show his love for a job well done. He’s also very friendly and funny guy, and it’s a real pleasure to work with him.

Contribution (Maintainer): Andrew Rybchenko
Andrew does very valuable work in DPDK; he’s of a great help for reviews in the mempool and mbuf areas. He’s rigorous and always brings a constructive and positive perspective.  He has the maintainer mindset, and ensures the APIs fit every case now and in the future.

Contribution (Operations):  Ferruh Yigit
Ferruh spends a lot of time ensuring things are progressing. He’s the engine  powering a fast=paced release and improvement cycle. In addition to his numerous contributions (code, reviews, doc, etc.) , Ferruh does an essential job in pinging people to ensure that everything is getting done on time.

Contribution (Testing): Aaron Conole
Aaron has been a regular DPDK contributor since 2015. In the  few past years, he has invested a lot of effort in setting up Travis CI for the DPDK project, working closely with the UNH Interop Lab team. He is especially good with testing powers in Open Source workflows. On top of that, Aaron is always available to provide constructive feedback, be it about technical topics, or about beer brewing!

Memory in DPDK Part 2: Deep Dive into IOVA

By Blog

By Antanoly Burakov

This post is Part 2 of a 4-part blog series that was originally published on the Intel Developer Zone blog.

Introduction

In the previous article, we covered the main concepts and principles behind Data Plane Development Kit (DPDK) memory management and how they contribute to DPDK’s unparalleled performance. However, DPDK is a complex beast that needs to be configured correctly to make the most out of it. In particular, picking the right kernel driver and IOVA mode may be crucial, depending on the application, as well as the environment in which said application is intended to run. This article discusses various options available and makes recommendations on what should be used.

Environment Abstraction Layer (EAL) Parameters

At the heart of DPDK lies the Environment Abstraction Layer (EAL). The EAL is a DPDK library that, as its name suggests, abstracts away the environment (hardware, OS, and so on) and presents a unified interface to software. EAL handles a great many things and is easily the single most complex part of DPDK. Some of the things EAL is responsible for include:

  • Managing CPU cores and non-uniform memory access (NUMA) nodes
  • Making hardware devices available to DPDK poll-mode drivers (PMDs) by mapping their registers into memory
  • Managing hardware and software interrupts
  • Abstracting away platform differences such as endianness, cache line size, and so on
  • Managing memory and multiprocess synchronization
  • Providing platform- and OS-independent ways of working with atomics, memory barriers, and other synchronization primitives
  • Loading and enumerating hardware buses, devices, and PMDs

The above list is by no means exhaustive, but it gives an idea of how vital the EAL is to DPDK. It is therefore no surprise that a lot of configuration in DPDK has to do with configuring the EAL. Currently, this is (directly or indirectly) done through specifying command-line parameters to the DPDK initialization routine. Usually, a DPDK application command-line would look like the following:

Some applications using DPDK may hide this step from the user (such as OvS-DPDK), so there may be no need to specify EAL command-line parameters explicitly, but it is nevertheless always happening in the background.

IO Virtual Addresses (IOVA) Modes

DPDK is a user space application framework, so software using DPDK works with regular virtual addresses, like any other software. However, DPDK also provides user space PMDs and a set of APIs to perform IO operations entirely from user space. As was discussed in the previous article in this series, the hardware does not understand user space virtual addresses; instead, it uses IO addresses—either physical addresses (PA), or IO virtual addresses (IOVA).

The DPDK API does not distinguish between physical and IO virtual addresses, and always refers to either as IOVA, even if no IO memory management unit (IOMMU) is involved to provide the VA part. However, DPDK does distinguish between cases where physical addresses are used as IOVA, and cases where IOVA matches user space virtual addresses. These cases are referred to as IOVA modes in the DPDK API, and there are two of them: IOVA as PA, and IOVA as VA.

IOVA as Physical Addresses (PA) Mode

When IOVA as PA mode is used, the IOVA addresses assigned to all DPDK memory areas are actual physical addresses, and virtual memory layout matches the physical memory layout. The good thing about this approach is that it is simple: it works with all hardware (that is, does not require IOMMU), and it works well with kernel space (it is trivial to convert a real physical address to a kernel space address). This is in fact how DPDK has worked for a long time, and it is in many ways considered the default.

There are certain disadvantages associated with using IOVA as PA mode, however. One of them is that it requires privileges—DPDK cannot get a memory region’s real physical address without having access to the system’s page map. Thus, it is not possible to run in IOVA as PA mode without root privileges on the system.

Another notable limitation of IOVA as PA mode is that virtual memory layout follows physical memory layout. This means that if physical memory space is fragmented (that is, there are lots of small segments instead of a few large ones), the virtual memory space follows that fragmentation. In extreme cases, the fragmentation can be so severe that the number of standalone, physically contiguous segments exhausts DPDK’s internal data structures used to store information about those segments, and DPDK initialization simply fails.

The DPDK community has come up with workarounds to address these issues. For example, one way to reduce the impact of fragmentation is to use bigger page sizes—the problem is not fixed, but a standalone 1-gigabyte (GB) segment is way more useful than a standalone 2-megabyte (MB) segment. Rebooting the system and reserving huge pages at boot time instead of at run time is another widely used workaround. None of the above workarounds fix the underlying problem though, and the DPDK community is so used to dealing with it that every DPDK user (knowingly or unknowingly) ends up following the same thought process of “I need X MB of memory, but I’ll reserve X+Y MB just in case!”

IOVA as Virtual Addresses (VA) Mode

IOVA as VA mode, in contrast, is a mode in which the underlying physical memory layout is not followed. Instead, the physical memory is reshuffled in such a way as to match the virtual memory layout. DPDK EAL does so by relying on kernel infrastructure, which in turn uses IOMMU to remap physical memory.

The advantage of this approach is obvious: in the case of IOVA as VA mode, all memory is both VA- and IOVA-contiguous. This means that any memory allocation that requires lots of IOVA-contiguous memory is more likely to succeed because the memory looks IOVA-contiguous to the hardware, even though the underlying physical memory may not be. Because of the remapping, the problem of fragmented IOVA space becomes irrelevant; however heavily fragmented the physical memory can be, it is always remapped to appear as an IOVA-contiguous chunk of memory.

Another advantage of using IOVA as VA mode is that it does not require any privileges, because it does not need access to the system page map. This allows running DPDK as a non-root user, and makes it easier to use DPDK in environments where privileged access is undesirable, such as cloud-native environments.

There is of course one disadvantage to using IOVA as VA mode. For various reasons, using the IOMMU may not be an option. Such circumstances may include:

  • Hardware that does not support using IOMMU
  • Platform may not have an IOMMU in the first place (for example, a VM without IOMMU emulation)
  • Software devices (for example, DPDK’s Kernel Network Interface (KNI) PMD) will not support IOVA as VA mode
  • Some IOMMUs (generally emulated ones) may have a limited address width, which, while not preventing the use of IOVA as VA mode, limits its usefulness
  • Using DPDK on an OS other than Linux*

However, such cases are relatively rare, and in the great majority of scenarios, IOVA as VA mode will work just fine.

Which IOVA Mode to Use

In many cases, DPDK chooses IOVA as PA mode as the default, as it is the most safe mode to use from the hardware perspective. Any given hardware (or software) PMD is all but guaranteed to support at least IOVA as PA mode. Nevertheless, all DPDK users are highly encouraged to use IOVA as VA mode whenever possible, as there are undeniable advantages to using this mode.

The user, however, does not have to pick one over the other. The most suitable IOVA mode is detected automatically, and the default value most definitely works for the majority of cases, so no user interaction is required to make this choice. If the default is not suitable, the user can attempt to override the IOVA mode with an EAL flag (applicable to DPDK 17.11 and later) by using the –iova-mode EAL command-line parameter:

In most cases, VA and PA modes do not exclude each other and either one can be used, but there are some circumstances where IOVA as PA mode will be the only available option. If using IOVA as VA mode is not available, DPDK automatically switches over to IOVA as PA mode, even if it was requested to use IOVA as VA mode through an EAL parameter.

DPDK also provides an API to query which particular IOVA mode is in use at run time, but generally it is not used in user applications, as such information is usually only required by entities like DPDK PMDs and bus drivers.

IOVA Mode and DPDK PCI Drivers

DPDK does not do all hardware device register and interrupt mapping by itself; it needs a little help from the kernel. To accomplish that, all hardware devices that are to be used by DPDK need to be bound to a generic Peripheral Component Interconnect (PCI) kernel driver. The generic part means that this driver is not locked to a specific set of PCI IDs like regular drivers, but can instead be used with any PCI device.

To bind a device to a generic driver, DPDK users are encouraged to refer to DPDK documentation, which describes this process for all supported OSes. However, a few words need to be said about various user space IO drivers supported by DPDK, and which IOVA modes they support. It may seem like there would be a 1:1 correspondence between a kernel driver and supported IOVA modes, but that is not actually the case. The following section discusses available drivers on Linux.

User Space IO (UIO) Drivers

The oldest kernel driver in the DPDK codebase is the igb_uio driver. It has been there pretty much since the beginning of DPDK, and it is thus the most widely used and the most familiar driver to DPDK developers.

This driver relies on kernel user space IO (UIO) infrastructure to work, and provides support for all interrupt types (legacy, message signaled interrupts (MSI), and MSI-X), as well as creating virtual functions. It also exposes hardware devices’ registers and interrupt handles through the /dev/uio file system, which DPDK EAL then uses to map them into user space and make them available for DPDK PMDs.

The igb_uio driver is very simple and does not do very much. It is therefore no surprise that it does not support using IOMMU. Or, to be more precise, it does support IOMMU, but only in pass-through mode, which sets up a 1:1 mapping between IOVA and the physical address. Using full IOMMU mode is not supported by igb_uio. As a consequence, the igb_uio driver only supports IOVA as PA mode and cannot work in IOVA as VA mode at all.

A driver similar to igb_uio is available in the kernel: uio_pci_generic. It works pretty much the same way as igb_uio, except that it is more limited in what it can do. For example, igb_uio supports all interrupt types (legacy, MSI, and MSI-X), while uio_pci_generic only supports legacy interrupts. Perhaps more importantly, igb_uio can also create virtual functions, while uio_pci_generic cannot; so, if creating virtual functions while using a DPDK physical function driver is a requirement, igb_uio is the only option.

Thus, in most cases, igb_uio would be either equivalent or preferable to uio_pci_generic. All of the limitations with regard to using IOMMU apply equally to both igb_uio and uio_pci_generic drivers—they cannot use full IOMMU functionality, and thus only support IOVA as PA mode.

VFIO Kernel Driver

An alternative to the above drivers is a vfio-pci driver. It is part of Virtual Function I/O (VFIO) kernel infrastructure and was introduced in Linux version 3.6. The VFIO infrastructure makes both device registers and device interrupts available to user space applications, and can use the IOMMU to set up IOVA mappings to perform IO from user space. The latter part is crucial—this driver was developed specifically for use with IOMMU and, on older kernels, will not even work without IOMMU enabled.

Contrary to what might seem intuitive, using the VFIO driver allows using both IOVA as PA and IOVA as VA modes. This is because, while it is recommended to use IOVA as VA mode to avail all of the benefits of that mode, nothing stops DPDK’s EAL from setting up IOMMU maps in such a way as to follow the physical memory layout 1:1; the IOVA mappings are arbitrary, after all. In that case, even though the IOMMU is used, DPDK will work in IOVA as PA mode, thereby allowing things like DPDK KNI to work. It does, however, still require root privileges to use IOVA as PA mode.

On more recent kernels (4.5+, backported to some older versions), there is an enable_unsafe_noiommu_mode option available that allows using VFIO without IOMMU. This mode is for all intents and purposes identical to UIO-based drivers, and shares all of the same advantages and limitations they have.

Which Kernel Driver to Use

Generally speaking, it is not a choice that has to be made. More often than not, the situation dictates the appropriate driver to use. The following flowchart is helpful in deciding which driver can be used in a particular circumstance:

As is clear from Figure 5, it is highly recommended to use the VFIO driver in just about all cases, especially in production environments. Using IOMMU provides device isolation at a hardware level, which makes applications using DPDK more secure, and using IOVA as VA mode allows better use of memory through remapping, as well as not requiring root privileges to run DPDK applications. However, certain use cases will require either igb_uio or uio_pci_generic drivers.

Software Poll Mode Drivers (PMD)

In addition to the above, DPDK also comes with a range of software PMDs that do not require a generic kernel PCI driver, and instead rely on standard kernel infrastructure to provide hardware support. This enables DPDK to work with almost any hardware, even if it is not natively supported by DPDK.

Currently, DPDK has PMDs for the PCAP library, which is a widely used and supported packet capture library for network hardware. DPDK also supports Linux networking with an AF_PACKET PMD, and there is also ongoing work to support AF_XDP natively in DPDK. Using these PMDs comes with a (sometimes considerable) performance cost, but the flipside is that the setup is easy, and these PMDs usually do not care about IOVA mode at all.

Summary

This article provided an in-depth view of how DPDK deals with physical memory, as well as outlined physical addressing features available in DPDK when using various Linux* kernel drivers.

This is the second article in the series of articles about memory management in DPDK. The first article outlined key principles that lie at the foundation of DPDK’s memory management subsystem. The following articles in this series provide a historical perspective on memory management features available in DPDK long term support (LTS) releases 17.11 and earlier, as well as describe the changes and new features available in 18.11 and later DPDK versions.