[dpdk-dev] maintainers: claim responsability for xen

  As some users are still using xen as the hypervisor, I suggest to
continue support for xen in DPDK. And from 16.11, I will be the
maintainer of all xen-related files.

Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
---
 MAINTAINERS | 1 +
 1 file changed, 1 insertion(+)
  

2016-11-07 07:38, Jianfeng Tan:
> As some users are still using xen as the hypervisor, I suggest to
> continue support for xen in DPDK. And from 16.11, I will be the
> maintainer of all xen-related files.
> 
> Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>

Applied

Please Jianfeng, could you start your new role by sending a status
of Xen dom0 support in DPDK? I think there are some issues in latest releases.
  

On Wed, Nov 9, 2016 at 5:03 PM, Thomas Monjalon
<thomas.monjalon@6wind.com> wrote:
> 2016-11-07 07:38, Jianfeng Tan:
>> As some users are still using xen as the hypervisor, I suggest to
>> continue support for xen in DPDK. And from 16.11, I will be the
>> maintainer of all xen-related files.
>>
>> Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
>
> Applied
>
> Please Jianfeng, could you start your new role by sending a status
> of Xen dom0 support in DPDK? I think there are some issues in latest releases.
>

Pls also CC me and xen-devel@lists.xenproject.org if possible.

Thanks!
  

Hi Thomas and Konrad,


On 11/11/2016 2:59 AM, Konrad Rzeszutek Wilk wrote:
> On Wed, Nov 9, 2016 at 5:03 PM, Thomas Monjalon
> <thomas.monjalon@6wind.com> wrote:
>> 2016-11-07 07:38, Jianfeng Tan:
>>> As some users are still using xen as the hypervisor, I suggest to
>>> continue support for xen in DPDK. And from 16.11, I will be the
>>> maintainer of all xen-related files.
>>>
>>> Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
>> Applied
>>
>> Please Jianfeng, could you start your new role by sending a status
>> of Xen dom0 support in DPDK? I think there are some issues in latest releases.

Yes, I'm on this. And hope to send out update in the next week.

>>
> Pls also CC me and xen-devel@lists.xenproject.org if possible.

Glad to do that.

Thanks,
Jianfeng

>
> Thanks!
  

2016-11-11 04:49, Tan, Jianfeng:
> Hi Thomas and Konrad,
> 
> On 11/11/2016 2:59 AM, Konrad Rzeszutek Wilk wrote:
> > On Wed, Nov 9, 2016 at 5:03 PM, Thomas Monjalon wrote:
> >> 2016-11-07 07:38, Jianfeng Tan:
> >>> As some users are still using xen as the hypervisor, I suggest to
> >>> continue support for xen in DPDK. And from 16.11, I will be the
> >>> maintainer of all xen-related files.
> >>>
> >>> Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
> >> Applied
> >>
> >> Please Jianfeng, could you start your new role by sending a status
> >> of Xen dom0 support in DPDK? I think there are some issues in latest releases.
> 
> Yes, I'm on this. And hope to send out update in the next week.
> 
> > Pls also CC me and xen-devel@lists.xenproject.org if possible.
> 
> Glad to do that.
> 
> Thanks,
> Jianfeng

Any news about Xen dom0 status in DPDK?

I think there is not enough interest for Xen dom0 in the DPDK community.
Last time we talked about, it was because of a severe bug which is not
fixed yet:
	http://dpdk.org/ml/archives/dev/2016-July/044207.html
	http://dpdk.org/ml/archives/dev/2016-July/044376.html

The request (6 month ago) was to give more time for feedbacks:
	http://dpdk.org/ml/archives/dev/2016-July/044847.html

Is it time now to officially remove Dom0 support?
  

Le 16/02/2017 à 12:06, Thomas Monjalon a écrit :
> The request (6 month ago) was to give more time for feedbacks:
> 	http://dpdk.org/ml/archives/dev/2016-July/044847.html
>
> Is it time now to officially remove Dom0 support?
yes
  

On Thu, Feb 16, 2017 at 12:06:18PM +0100, Thomas Monjalon wrote:
> 2016-11-11 04:49, Tan, Jianfeng:
> > Hi Thomas and Konrad,
> > 
> > On 11/11/2016 2:59 AM, Konrad Rzeszutek Wilk wrote:
> > > On Wed, Nov 9, 2016 at 5:03 PM, Thomas Monjalon wrote:
> > >> 2016-11-07 07:38, Jianfeng Tan:
> > >>> As some users are still using xen as the hypervisor, I suggest to
> > >>> continue support for xen in DPDK. And from 16.11, I will be the
> > >>> maintainer of all xen-related files.
> > >>>
> > >>> Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
> > >> Applied
> > >>
> > >> Please Jianfeng, could you start your new role by sending a status
> > >> of Xen dom0 support in DPDK? I think there are some issues in latest releases.
> > 
> > Yes, I'm on this. And hope to send out update in the next week.
> > 
> > > Pls also CC me and xen-devel@lists.xenproject.org if possible.
> > 
> > Glad to do that.
> > 
> > Thanks,
> > Jianfeng
> 
> Any news about Xen dom0 status in DPDK?
> 
> I think there is not enough interest for Xen dom0 in the DPDK community.
> Last time we talked about, it was because of a severe bug which is not
> fixed yet:
> 	http://dpdk.org/ml/archives/dev/2016-July/044207.html
> 	http://dpdk.org/ml/archives/dev/2016-July/044376.html
> 
> The request (6 month ago) was to give more time for feedbacks:
> 	http://dpdk.org/ml/archives/dev/2016-July/044847.html
> 
> Is it time now to officially remove Dom0 support?

So we do have an prototype implementation of netback but it is waiting
for review of xen-devel to the spec.

And I believe the implementation does utilize some of the dom0
parts of code in DPDK.
  

On Thu, Feb 16, 2017 at 2:36 PM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> On Thu, Feb 16, 2017 at 12:06:18PM +0100, Thomas Monjalon wrote:
>>
>> Any news about Xen dom0 status in DPDK?
>>
>> I think there is not enough interest for Xen dom0 in the DPDK community.
>> Last time we talked about, it was because of a severe bug which is not
>> fixed yet:
>>       http://dpdk.org/ml/archives/dev/2016-July/044207.html
>>       http://dpdk.org/ml/archives/dev/2016-July/044376.html
>>
>> The request (6 month ago) was to give more time for feedbacks:
>>       http://dpdk.org/ml/archives/dev/2016-July/044847.html
>>
>> Is it time now to officially remove Dom0 support?
>
> So we do have an prototype implementation of netback but it is waiting
> for review of xen-devel to the spec.
>
> And I believe the implementation does utilize some of the dom0
> parts of code in DPDK.

FWIW I've also tested the 16.11 code in a Xen PV domU with uses the
xen_dom0 module due to the lack of hugepages.
  

Le 16/02/2017 à 14:36, Konrad Rzeszutek Wilk a écrit :
>> Is it time now to officially remove Dom0 support?
> So we do have an prototype implementation of netback but it is waiting
> for review of xen-devel to the spec.
>
> And I believe the implementation does utilize some of the dom0
> parts of code in DPDK.

Please, do you have URLs/pointers about it? It would be interesting to 
share it with DPDK community too.

Best regards,
   Vincent
  

On Thu, Feb 16, 2017 at 10:51:44PM +0100, Vincent JARDIN wrote:
> Le 16/02/2017 à 14:36, Konrad Rzeszutek Wilk a écrit :
> > > Is it time now to officially remove Dom0 support?
> > So we do have an prototype implementation of netback but it is waiting
> > for review of xen-devel to the spec.
> > 
> > And I believe the implementation does utilize some of the dom0
> > parts of code in DPDK.
> 
> Please, do you have URLs/pointers about it? It would be interesting to share
> it with DPDK community too.

Joao, would it be possible to include an tarball of the patches? I know
they are no in the right state with the review of the staging
grants API - they are incompatible, but it may help folks to get
a feel for what DPDK APIs you used?

Staging grants API:
https://lists.xenproject.org/archives/html/xen-devel/2016-12/msg01878.html

> 
> Best regards,
>   Vincent
  

On Fri, Feb 17, 2017 at 5:07 PM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> On Thu, Feb 16, 2017 at 10:51:44PM +0100, Vincent JARDIN wrote:
>> Le 16/02/2017 à 14:36, Konrad Rzeszutek Wilk a écrit :
>> > > Is it time now to officially remove Dom0 support?
>> > So we do have an prototype implementation of netback but it is waiting
>> > for review of xen-devel to the spec.
>> >
>> > And I believe the implementation does utilize some of the dom0
>> > parts of code in DPDK.
>>
>> Please, do you have URLs/pointers about it? It would be interesting to share
>> it with DPDK community too.
>
> Joao, would it be possible to include an tarball of the patches? I know
> they are no in the right state with the review of the staging
> grants API - they are incompatible, but it may help folks to get
> a feel for what DPDK APIs you used?
>
> Staging grants API:
> https://lists.xenproject.org/archives/html/xen-devel/2016-12/msg01878.html
>

The topic of the grants API is unrelated to the dom0 memory pool. The
memory pool which uses xen_create_contiguous_region() is used in cases
we know that there are no hugepages available.

Joao and I met in Dublin and I whined about not being able to call
into the grants API from userspace and instead need to kick a kernel
driver to do the work for every burst. It would be great if that could
change in the future.
  

On 02/17/2017 04:07 PM, Konrad Rzeszutek Wilk wrote:
> On Thu, Feb 16, 2017 at 10:51:44PM +0100, Vincent JARDIN wrote:
>> Le 16/02/2017 à 14:36, Konrad Rzeszutek Wilk a écrit :
>>>> Is it time now to officially remove Dom0 support?
>>> So we do have an prototype implementation of netback but it is waiting
>>> for review of xen-devel to the spec.
>>>
>>> And I believe the implementation does utilize some of the dom0
>>> parts of code in DPDK.
>>
>> Please, do you have URLs/pointers about it? It would be interesting to share
>> it with DPDK community too.
> 
> Joao, would it be possible to include an tarball of the patches? I know
> they are no in the right state with the review of the staging
> grants API - they are incompatible, but it may help folks to get
> a feel for what DPDK APIs you used?
OK, see attached - I should note that its a WIP as Konrad noted, but once the
staging grants work is finished, the code would be improved to have it in better
shape (as well as in feature parity) for a proper RFC [and adhering to the
project coding style].

Joao
  

On 02/20/2017 09:56 AM, Jan Blunck wrote:
> On Fri, Feb 17, 2017 at 5:07 PM, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com> wrote:
>> On Thu, Feb 16, 2017 at 10:51:44PM +0100, Vincent JARDIN wrote:
>>> Le 16/02/2017 à 14:36, Konrad Rzeszutek Wilk a écrit :
>>>>> Is it time now to officially remove Dom0 support?
>>>> So we do have an prototype implementation of netback but it is waiting
>>>> for review of xen-devel to the spec.
>>>>
>>>> And I believe the implementation does utilize some of the dom0
>>>> parts of code in DPDK.
>>>
>>> Please, do you have URLs/pointers about it? It would be interesting to share
>>> it with DPDK community too.
>>
>> Joao, would it be possible to include an tarball of the patches? I know
>> they are no in the right state with the review of the staging
>> grants API - they are incompatible, but it may help folks to get
>> a feel for what DPDK APIs you used?
>>
>> Staging grants API:
>> https://lists.xenproject.org/archives/html/xen-devel/2016-12/msg01878.html
> 
> The topic of the grants API is unrelated to the dom0 memory pool. The
> memory pool which uses xen_create_contiguous_region() is used in cases
> we know that there are no hugepages available.
Correct, I think what Konrad was trying to say was that xen-netback normally
lives in a PV domain which doesn't have superpages, therefore such driver would
need that memory pool part in order to work. The mentioned spec are additions to
xen netif ABI for backend to safely map a fixed set of grant references
(recycled overtime, provided by frontend) with the purpose of avoiding grant ops
- DPDK would be one of the users.

> Joao and I met in Dublin and I whined about not being able to call
> into the grants API from userspace and instead need to kick a kernel
> driver to do the work for every burst. It would be great if that could
> change in the future.
Hm, I recall about that discussion. AFAIK you can do both grant alloc/revoke of
pages through xengntshr_share_pages(...) and xengntshr_unshare(...) APIs
provided by libxengnttab[0] starting 4.7 or, libxc on older versions with
xc_gntshr_share_pages/xc_gntshr_munmap[2]. For the notification (or kicks) you
can allocate the event channel in the guest with libevtchn[1] starting 4.7, with
xenevtchn_bind_unbound_port(...) or libxc on older versions with
xc_evtchn_bind_unbound_port(...)[2]. And kick the guest with xenevtchn_notify or
xc_evtchn_notify(...) [latter on older versions]. In short these APIs are ioctls
to /dev/gntdev and /dev/evtchn. xenstore operations can also be done in
userspace with libxenstore[3].

To have the (similar) behavior of VRING_AVAIL_F_NO_INTERRUPT (i.e. avoiding the
kicks) you "just" don't set rsp_event in ring (e.g. no calls to
RING_FINAL_CHECK_FOR_RESPONSES), and keep checking for unconsumed Rx/Tx
responses. For guest request notification (to wake up the backend for new Tx/Rx
requests), you're dependent on whether backend requests it since it's the one
setting req_event index. If it indeed sets it then you gotta use the evtchn
notify that I depicted in the previous paragraph.

Hope that helps!

Joao

[0]
https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=tools/libs/gnttab/include/xengnttab.h;hb=HEAD
[1]
https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=tools/libs/evtchn/include/xenevtchn.h;hb=HEAD
[2]
https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=tools/libxc/include/xenctrl_compat.h;hb=HEAD
[3]
https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=tools/xenstore/include/xenstore.h;hb=HEAD
  

2017-02-20 15:33, Joao Martins:
> On 02/17/2017 04:07 PM, Konrad Rzeszutek Wilk wrote:
> > On Thu, Feb 16, 2017 at 10:51:44PM +0100, Vincent JARDIN wrote:
> >> Le 16/02/2017 à 14:36, Konrad Rzeszutek Wilk a écrit :
> >>>> Is it time now to officially remove Dom0 support?
> >>> So we do have an prototype implementation of netback but it is waiting
> >>> for review of xen-devel to the spec.
> >>>
> >>> And I believe the implementation does utilize some of the dom0
> >>> parts of code in DPDK.
> >>
> >> Please, do you have URLs/pointers about it? It would be interesting to share
> >> it with DPDK community too.
> > 
> > Joao, would it be possible to include an tarball of the patches? I know
> > they are no in the right state with the review of the staging
> > grants API - they are incompatible, but it may help folks to get
> > a feel for what DPDK APIs you used?
> OK, see attached - I should note that its a WIP as Konrad noted, but once the
> staging grants work is finished, the code would be improved to have it in better
> shape (as well as in feature parity) for a proper RFC [and adhering to the
> project coding style].

Excuse my ignorance on Xen.
Is xen-netback for Dom0?
Is the DPDK Dom0 support working and useful?
  

On 23 February 2017 at 03:49, Thomas Monjalon <thomas.monjalon@6wind.com>
wrote:

> 2017-02-20 15:33, Joao Martins:
> > On 02/17/2017 04:07 PM, Konrad Rzeszutek Wilk wrote:
> > > On Thu, Feb 16, 2017 at 10:51:44PM +0100, Vincent JARDIN wrote:
> > >> Le 16/02/2017 à 14:36, Konrad Rzeszutek Wilk a écrit :
> > >>>> Is it time now to officially remove Dom0 support?
> > >>> So we do have an prototype implementation of netback but it is
> waiting
> > >>> for review of xen-devel to the spec.
> > >>>
> > >>> And I believe the implementation does utilize some of the dom0
> > >>> parts of code in DPDK.
> > >>
> > >> Please, do you have URLs/pointers about it? It would be interesting
> to share
> > >> it with DPDK community too.
> > >
> > > Joao, would it be possible to include an tarball of the patches? I know
> > > they are no in the right state with the review of the staging
> > > grants API - they are incompatible, but it may help folks to get
> > > a feel for what DPDK APIs you used?
> > OK, see attached - I should note that its a WIP as Konrad noted, but
> once the
> > staging grants work is finished, the code would be improved to have it
> in better
> > shape (as well as in feature parity) for a proper RFC [and adhering to
> the
> > project coding style].
>
> Excuse my ignorance on Xen.
> Is xen-netback for Dom0?
> Is the DPDK Dom0 support working and useful?
>

I'm also curious about Xen support on DPDK, mostly because I'm planning to
run DPDK Apps on Amazon EC2 and they are Xen-based...

So, sorry to ask this here but, how to use DPDK on Amazon EC2 (i.e., on top
of Xen)? Which EC2 flavor / DPDK drivers to use?

Thanks!
Thiago
  

Ping

The Xen dom0 support in DPDK seems dead.

Reminder:
Last time we talked about, it was because of a severe bug which is not
fixed yet:
        http://dpdk.org/ml/archives/dev/2016-July/044207.html
        http://dpdk.org/ml/archives/dev/2016-July/044376.html
The request (9 months ago) was to give more time for feedbacks:
        http://dpdk.org/ml/archives/dev/2016-July/044847.html


23/02/2017 09:49, Thomas Monjalon:
> 2017-02-20 15:33, Joao Martins:
> > On 02/17/2017 04:07 PM, Konrad Rzeszutek Wilk wrote:
> > > On Thu, Feb 16, 2017 at 10:51:44PM +0100, Vincent JARDIN wrote:
> > >> Le 16/02/2017 à 14:36, Konrad Rzeszutek Wilk a écrit :
> > >>>> Is it time now to officially remove Dom0 support?
> > >>> So we do have an prototype implementation of netback but it is waiting
> > >>> for review of xen-devel to the spec.
> > >>>
> > >>> And I believe the implementation does utilize some of the dom0
> > >>> parts of code in DPDK.
> > >>
> > >> Please, do you have URLs/pointers about it? It would be interesting to share
> > >> it with DPDK community too.
> > > 
> > > Joao, would it be possible to include an tarball of the patches? I know
> > > they are no in the right state with the review of the staging
> > > grants API - they are incompatible, but it may help folks to get
> > > a feel for what DPDK APIs you used?
> > OK, see attached - I should note that its a WIP as Konrad noted, but once the
> > staging grants work is finished, the code would be improved to have it in better
> > shape (as well as in feature parity) for a proper RFC [and adhering to the
> > project coding style].
> 
> Excuse my ignorance on Xen.
> Is xen-netback for Dom0?
> Is the DPDK Dom0 support working and useful?
  

Hi  Thomas and all,

Apologize for being an unqualified maintainer.

> -----Original Message-----
> From: Thomas Monjalon [mailto:thomas@monjalon.net]
> Sent: Friday, May 5, 2017 6:04 AM
> To: Joao Martins; Konrad Rzeszutek Wilk; Tan, Jianfeng
> Cc: Konrad Rzeszutek Wilk; dev@dpdk.org; Xen-devel
> Subject: Re: [dpdk-dev] [PATCH] maintainers: claim responsability for xen
> 
> Ping
> 
> The Xen dom0 support in DPDK seems dead.
> 
> Reminder:
> Last time we talked about, it was because of a severe bug which is not
> fixed yet:
>         http://dpdk.org/ml/archives/dev/2016-July/044207.html

For this bug, we removed the userspace memset(0) and suppose it has been done by kernel, however, xen0 uses __get_free_pages() kernel API to map hugepages and reseve memseg, I think it makes sense to zero the hugepage for xen0 in rte_dom0_mm kernel module (instead of some special code for xen0 in userspace) to keep aligned behavior.

>         http://dpdk.org/ml/archives/dev/2016-July/044376.html

It does not make any sense to upstream a netfront PMD before we have a netback PMD, as the legacy netback driver would be the bottleneck. Anyone has plan on this? And a question mark keeps in my mind that is it a must to implement netback in dom0?

From another perspective, instead of using netfront/netback, we can also use virtio/vhost as the device model; however, xl tool in xen only supports vhost-kernel backend instead of vhost-user backend. So anyone has plan to enhance xl tool so that we can accelerate dom0 just using vswitch like OVS-DPDK?

A third solution is to use xenvirtio as the frontend, and vhost_xen as the backend. This solution is to use virtio ring on grant table mechanism of xen. Honestly, I don't even know if it still work now. And to make it more usable, better to upstream vhost_xen inside popular vswitch like OVS-DPDK.

> The request (9 months ago) was to give more time for feedbacks:
>         http://dpdk.org/ml/archives/dev/2016-July/044847.html

Apologize again that I volunteer to maintain these files, but spend very few time on this.

Thanks,
Jianfeng
  

Thanks Jianfeng for giving new ideas.

There is not much activity on Xen side.
Is there someone working on DPDK+Xen? Any news?

The technical board requested to re-consider Xen support in DPDK.
It will be discussed in the next techboard meeting:
	https://annuel.framapad.org/p/r.0c3cc4d1e011214183872a98f6b5c7db


11/05/2017 13:41, Tan, Jianfeng:
> Hi  Thomas and all,
> 
> Apologize for being an unqualified maintainer.
> 
> > -----Original Message-----
> From: Thomas Monjalon [mailto:thomas@monjalon.net]
> 
> > Ping
> > 
> > The Xen dom0 support in DPDK seems dead.
> > 
> > Reminder:
> > Last time we talked about, it was because of a severe bug which is not
> > fixed yet:
> >         http://dpdk.org/ml/archives/dev/2016-July/044207.html
> 
> For this bug, we removed the userspace memset(0) and suppose it has been done by kernel, however, xen0 uses __get_free_pages() kernel API to map hugepages and reseve memseg, I think it makes sense to zero the hugepage for xen0 in rte_dom0_mm kernel module (instead of some special code for xen0 in userspace) to keep aligned behavior.
> 
> >         http://dpdk.org/ml/archives/dev/2016-July/044376.html
> 
> It does not make any sense to upstream a netfront PMD before we have a netback PMD, as the legacy netback driver would be the bottleneck. Anyone has plan on this? And a question mark keeps in my mind that is it a must to implement netback in dom0?
> 
> From another perspective, instead of using netfront/netback, we can also use virtio/vhost as the device model; however, xl tool in xen only supports vhost-kernel backend instead of vhost-user backend. So anyone has plan to enhance xl tool so that we can accelerate dom0 just using vswitch like OVS-DPDK?
> 
> A third solution is to use xenvirtio as the frontend, and vhost_xen as the backend. This solution is to use virtio ring on grant table mechanism of xen. Honestly, I don't even know if it still work now. And to make it more usable, better to upstream vhost_xen inside popular vswitch like OVS-DPDK.
> 
> > The request (9 months ago) was to give more time for feedbacks:
> >         http://dpdk.org/ml/archives/dev/2016-July/044847.html
> 
> Apologize again that I volunteer to maintain these files, but spend very few time on this.
> 
> Thanks,
> Jianfeng