[dpdk-dev] log: deprecate history dump

  The log history uses rte_mempool. In order to remove the mempool
dependency in EAL (and improve the build), this feature is deprecated.
The ABI is kept but the behaviour is now voided because it seems this
function was not used. The history can be read from syslog.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 app/test-pmd/cmdline.c                       |   3 -
 app/test/autotest_test_funcs.py              |   5 --
 app/test/commands.c                          |   4 +-
 app/test/test_logs.c                         |   3 -
 doc/guides/rel_notes/deprecation.rst         |   3 +
 lib/librte_eal/bsdapp/eal/eal_debug.c        |   6 --
 lib/librte_eal/common/eal_common_log.c       | 128 +--------------------------
 lib/librte_eal/common/include/rte_log.h      |   8 ++
 lib/librte_eal/linuxapp/eal/eal_debug.c      |   6 --
 lib/librte_eal/linuxapp/eal/eal_interrupts.c |   1 -
 lib/librte_eal/linuxapp/eal/eal_ivshmem.c    |   1 -
 lib/librte_eal/linuxapp/eal/eal_log.c        |   3 -
 lib/librte_mempool/rte_mempool.c             |   4 -
 13 files changed, 16 insertions(+), 159 deletions(-)
  

Thomas,

On Thu, Jun 9, 2016 at 4:09 PM, Thomas Monjalon
<thomas.monjalon@6wind.com> wrote:
> The log history uses rte_mempool. In order to remove the mempool
> dependency in EAL (and improve the build), this feature is deprecated.
> The ABI is kept but the behaviour is now voided because it seems this
> function was not used. The history can be read from syslog.

It does look like it is not really used.
I am for this change unless someone complains.

Comments below.

> Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
> ---
>  app/test-pmd/cmdline.c                       |   3 -
>  app/test/autotest_test_funcs.py              |   5 --
>  app/test/commands.c                          |   4 +-
>  app/test/test_logs.c                         |   3 -
>  doc/guides/rel_notes/deprecation.rst         |   3 +
>  lib/librte_eal/bsdapp/eal/eal_debug.c        |   6 --
>  lib/librte_eal/common/eal_common_log.c       | 128 +--------------------------
>  lib/librte_eal/common/include/rte_log.h      |   8 ++
>  lib/librte_eal/linuxapp/eal/eal_debug.c      |   6 --
>  lib/librte_eal/linuxapp/eal/eal_interrupts.c |   1 -
>  lib/librte_eal/linuxapp/eal/eal_ivshmem.c    |   1 -
>  lib/librte_eal/linuxapp/eal/eal_log.c        |   3 -
>  lib/librte_mempool/rte_mempool.c             |   4 -
>  13 files changed, 16 insertions(+), 159 deletions(-)

- You missed autotest_data.py.

$ git grep dump_log_history
app/test/autotest_data.py:               "Command" :    "dump_log_history",

- eal Makefile still refers to librte_mempool.

- Since you are looking at this, what keeps us from removing the
dependency on librte_ring ?
I would say it was mainly because of mempool.
Maybe ivshmem ?


[snip]

> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
> index ad05eba..f11df93 100644
> --- a/doc/guides/rel_notes/deprecation.rst
> +++ b/doc/guides/rel_notes/deprecation.rst
> @@ -8,6 +8,9 @@ API and ABI deprecation notices are to be posted here.
>  Deprecation Notices
>  -------------------
>
> +* The log history is deprecated in release 16.07.
> +  It is voided and will be completely removed in release 16.11.
> +

It is voided in 16.07 and will be completely removed in release 16.11.


>  * The ethdev hotplug API is going to be moved to EAL with a notification
>    mechanism added to crypto and ethdev libraries so that hotplug is now
>    available to both of them. This API will be stripped of the device arguments

> diff --git a/lib/librte_eal/common/eal_common_log.c b/lib/librte_eal/common/eal_common_log.c
> index b5e37bb..94ecdd2 100644
> --- a/lib/librte_eal/common/eal_common_log.c
> +++ b/lib/librte_eal/common/eal_common_log.c
> @@ -56,29 +56,11 @@
>  #include <rte_spinlock.h>
>  #include <rte_branch_prediction.h>
>  #include <rte_ring.h>
> -#include <rte_mempool.h>
>
>  #include "eal_private.h"
>
>  #define LOG_ELT_SIZE     2048

We don't need LOG_ELT_SIZE.

>
> -#define LOG_HISTORY_MP_NAME "log_history"
> -
> -STAILQ_HEAD(log_history_list, log_history);
> -
> -/**
> - * The structure of a message log in the log history.
> - */
> -struct log_history {
> -       STAILQ_ENTRY(log_history) next;
> -       unsigned size;
> -       char buf[0];
> -};
> -
> -static struct rte_mempool *log_history_mp = NULL;
> -static unsigned log_history_size = 0;
> -static struct log_history_list log_history;
> -
>  /* global log structure */
>  struct rte_logs rte_logs = {
>         .type = ~0,
> @@ -86,10 +68,7 @@ struct rte_logs rte_logs = {
>         .file = NULL,
>  };
>
> -static rte_spinlock_t log_dump_lock = RTE_SPINLOCK_INITIALIZER;
> -static rte_spinlock_t log_list_lock = RTE_SPINLOCK_INITIALIZER;
>  static FILE *default_log_stream;
> -static int history_enabled = 1;
>
>  /**
>   * This global structure stores some informations about the message
> @@ -106,59 +85,14 @@ static RTE_DEFINE_PER_LCORE(struct log_cur_msg, log_cur_msg);
>  /* default logs */
>
>  int
> -rte_log_add_in_history(const char *buf, size_t size)
> +rte_log_add_in_history(const char *buf __rte_unused, size_t size __rte_unused)
>  {
> -       struct log_history *hist_buf = NULL;
> -       static const unsigned hist_buf_size = LOG_ELT_SIZE - sizeof(*hist_buf);
> -       void *obj;
> -
> -       if (history_enabled == 0)
> -               return 0;
> -
> -       rte_spinlock_lock(&log_list_lock);
> -
> -       /* get a buffer for adding in history */
> -       if (log_history_size > RTE_LOG_HISTORY) {
> -               hist_buf = STAILQ_FIRST(&log_history);
> -               if (hist_buf) {
> -                       STAILQ_REMOVE_HEAD(&log_history, next);
> -                       log_history_size--;
> -               }
> -       }
> -       else {
> -               if (rte_mempool_mc_get(log_history_mp, &obj) < 0)
> -                       obj = NULL;
> -               hist_buf = obj;
> -       }
> -
> -       /* no buffer */
> -       if (hist_buf == NULL) {
> -               rte_spinlock_unlock(&log_list_lock);
> -               return -ENOBUFS;
> -       }
> -
> -       /* not enough room for msg, buffer go back in mempool */
> -       if (size >= hist_buf_size) {
> -               rte_mempool_mp_put(log_history_mp, hist_buf);
> -               rte_spinlock_unlock(&log_list_lock);
> -               return -ENOBUFS;
> -       }
> -
> -       /* add in history */
> -       memcpy(hist_buf->buf, buf, size);
> -       hist_buf->buf[size] = hist_buf->buf[hist_buf_size-1] = '\0';
> -       hist_buf->size = size;
> -       STAILQ_INSERT_TAIL(&log_history, hist_buf, next);
> -       log_history_size++;
> -       rte_spinlock_unlock(&log_list_lock);
> -
>         return 0;
>  }
>
>  void
> -rte_log_set_history(int enable)
> +rte_log_set_history(int enable __rte_unused)
>  {
> -       history_enabled = enable;
>  }

Maybe a warning here for the people who did not read the deprecation notices ?
  

2016-06-09 16:45, David Marchand:
> On Thu, Jun 9, 2016 at 4:09 PM, Thomas Monjalon
> <thomas.monjalon@6wind.com> wrote:
> > The log history uses rte_mempool. In order to remove the mempool
> > dependency in EAL (and improve the build), this feature is deprecated.
> > The ABI is kept but the behaviour is now voided because it seems this
> > function was not used. The history can be read from syslog.
> 
> It does look like it is not really used.
> I am for this change unless someone complains.
> 
> Comments below.

All your comments will be addressed in a v2. Thanks

> - Since you are looking at this, what keeps us from removing the
> dependency on librte_ring ?

Please see this first small cleanup:
	http://dpdk.org/ml/archives/dev/2016-June/040798.html

> I would say it was mainly because of mempool.
> Maybe ivshmem ?

Yes CONFIG_RTE_LIBRTE_IVSHMEM brings dependencies to rte_ring and rte_ivshmem.
This "feature" also pollute the memory allocator and makes rework harder.
That's why I would be in favor of removing CONFIG_RTE_LIBRTE_IVSHMEM.

Otherwise, as an alternative proposal, the file
	lib/librte_eal/linuxapp/eal/eal_ivshmem.c
could be moved outside of EAL. Probably that lib/librte_ivshmem/
would be a good place.
The tricky operation would be to remove ivshmem init from eal:

#ifdef RTE_LIBRTE_IVSHMEM
    if (rte_eal_ivshmem_init() < 0)                                                                              
        rte_panic("Cannot init IVSHMEM\n");
#endif

    if (rte_eal_memory_init() < 0)
        rte_panic("Cannot init memory\n");

    /* the directories are locked during eal_hugepage_info_init */
    eal_hugedirs_unlock();

    if (rte_eal_memzone_init() < 0)
        rte_panic("Cannot init memzone\n");

    if (rte_eal_tailqs_init() < 0)
        rte_panic("Cannot init tail queues for objects\n");

#ifdef RTE_LIBRTE_IVSHMEM
    if (rte_eal_ivshmem_obj_init() < 0)
        rte_panic("Cannot init IVSHMEM objects\n");
#endif
  

Hi,
in I totally like it - thanks Thomas for picking that up.

I just wanted to mention that the Makefile still refers to mempool, but
David beat me in time and Detail a lot.

I'll certainly try to follow and help the bit I can.



Christian Ehrhardt
Software Engineer, Ubuntu Server
Canonical Ltd

On Thu, Jun 9, 2016 at 4:45 PM, David Marchand <david.marchand@6wind.com>
wrote:

> Thomas,
>
> On Thu, Jun 9, 2016 at 4:09 PM, Thomas Monjalon
> <thomas.monjalon@6wind.com> wrote:
> > The log history uses rte_mempool. In order to remove the mempool
> > dependency in EAL (and improve the build), this feature is deprecated.
> > The ABI is kept but the behaviour is now voided because it seems this
> > function was not used. The history can be read from syslog.
>
> It does look like it is not really used.
> I am for this change unless someone complains.
>
> Comments below.
>
> > Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
> > ---
> >  app/test-pmd/cmdline.c                       |   3 -
> >  app/test/autotest_test_funcs.py              |   5 --
> >  app/test/commands.c                          |   4 +-
> >  app/test/test_logs.c                         |   3 -
> >  doc/guides/rel_notes/deprecation.rst         |   3 +
> >  lib/librte_eal/bsdapp/eal/eal_debug.c        |   6 --
> >  lib/librte_eal/common/eal_common_log.c       | 128
> +--------------------------
> >  lib/librte_eal/common/include/rte_log.h      |   8 ++
> >  lib/librte_eal/linuxapp/eal/eal_debug.c      |   6 --
> >  lib/librte_eal/linuxapp/eal/eal_interrupts.c |   1 -
> >  lib/librte_eal/linuxapp/eal/eal_ivshmem.c    |   1 -
> >  lib/librte_eal/linuxapp/eal/eal_log.c        |   3 -
> >  lib/librte_mempool/rte_mempool.c             |   4 -
> >  13 files changed, 16 insertions(+), 159 deletions(-)
>
> - You missed autotest_data.py.
>
> $ git grep dump_log_history
> app/test/autotest_data.py:               "Command" :    "dump_log_history",
>
> - eal Makefile still refers to librte_mempool.
>
> - Since you are looking at this, what keeps us from removing the
> dependency on librte_ring ?
> I would say it was mainly because of mempool.
> Maybe ivshmem ?
>
>
> [snip]
>
> > diff --git a/doc/guides/rel_notes/deprecation.rst
> b/doc/guides/rel_notes/deprecation.rst
> > index ad05eba..f11df93 100644
> > --- a/doc/guides/rel_notes/deprecation.rst
> > +++ b/doc/guides/rel_notes/deprecation.rst
> > @@ -8,6 +8,9 @@ API and ABI deprecation notices are to be posted here.
> >  Deprecation Notices
> >  -------------------
> >
> > +* The log history is deprecated in release 16.07.
> > +  It is voided and will be completely removed in release 16.11.
> > +
>
> It is voided in 16.07 and will be completely removed in release 16.11.
>
>
> >  * The ethdev hotplug API is going to be moved to EAL with a notification
> >    mechanism added to crypto and ethdev libraries so that hotplug is now
> >    available to both of them. This API will be stripped of the device
> arguments
>
> > diff --git a/lib/librte_eal/common/eal_common_log.c
> b/lib/librte_eal/common/eal_common_log.c
> > index b5e37bb..94ecdd2 100644
> > --- a/lib/librte_eal/common/eal_common_log.c
> > +++ b/lib/librte_eal/common/eal_common_log.c
> > @@ -56,29 +56,11 @@
> >  #include <rte_spinlock.h>
> >  #include <rte_branch_prediction.h>
> >  #include <rte_ring.h>
> > -#include <rte_mempool.h>
> >
> >  #include "eal_private.h"
> >
> >  #define LOG_ELT_SIZE     2048
>
> We don't need LOG_ELT_SIZE.
>
> >
> > -#define LOG_HISTORY_MP_NAME "log_history"
> > -
> > -STAILQ_HEAD(log_history_list, log_history);
> > -
> > -/**
> > - * The structure of a message log in the log history.
> > - */
> > -struct log_history {
> > -       STAILQ_ENTRY(log_history) next;
> > -       unsigned size;
> > -       char buf[0];
> > -};
> > -
> > -static struct rte_mempool *log_history_mp = NULL;
> > -static unsigned log_history_size = 0;
> > -static struct log_history_list log_history;
> > -
> >  /* global log structure */
> >  struct rte_logs rte_logs = {
> >         .type = ~0,
> > @@ -86,10 +68,7 @@ struct rte_logs rte_logs = {
> >         .file = NULL,
> >  };
> >
> > -static rte_spinlock_t log_dump_lock = RTE_SPINLOCK_INITIALIZER;
> > -static rte_spinlock_t log_list_lock = RTE_SPINLOCK_INITIALIZER;
> >  static FILE *default_log_stream;
> > -static int history_enabled = 1;
> >
> >  /**
> >   * This global structure stores some informations about the message
> > @@ -106,59 +85,14 @@ static RTE_DEFINE_PER_LCORE(struct log_cur_msg,
> log_cur_msg);
> >  /* default logs */
> >
> >  int
> > -rte_log_add_in_history(const char *buf, size_t size)
> > +rte_log_add_in_history(const char *buf __rte_unused, size_t size
> __rte_unused)
> >  {
> > -       struct log_history *hist_buf = NULL;
> > -       static const unsigned hist_buf_size = LOG_ELT_SIZE -
> sizeof(*hist_buf);
> > -       void *obj;
> > -
> > -       if (history_enabled == 0)
> > -               return 0;
> > -
> > -       rte_spinlock_lock(&log_list_lock);
> > -
> > -       /* get a buffer for adding in history */
> > -       if (log_history_size > RTE_LOG_HISTORY) {
> > -               hist_buf = STAILQ_FIRST(&log_history);
> > -               if (hist_buf) {
> > -                       STAILQ_REMOVE_HEAD(&log_history, next);
> > -                       log_history_size--;
> > -               }
> > -       }
> > -       else {
> > -               if (rte_mempool_mc_get(log_history_mp, &obj) < 0)
> > -                       obj = NULL;
> > -               hist_buf = obj;
> > -       }
> > -
> > -       /* no buffer */
> > -       if (hist_buf == NULL) {
> > -               rte_spinlock_unlock(&log_list_lock);
> > -               return -ENOBUFS;
> > -       }
> > -
> > -       /* not enough room for msg, buffer go back in mempool */
> > -       if (size >= hist_buf_size) {
> > -               rte_mempool_mp_put(log_history_mp, hist_buf);
> > -               rte_spinlock_unlock(&log_list_lock);
> > -               return -ENOBUFS;
> > -       }
> > -
> > -       /* add in history */
> > -       memcpy(hist_buf->buf, buf, size);
> > -       hist_buf->buf[size] = hist_buf->buf[hist_buf_size-1] = '\0';
> > -       hist_buf->size = size;
> > -       STAILQ_INSERT_TAIL(&log_history, hist_buf, next);
> > -       log_history_size++;
> > -       rte_spinlock_unlock(&log_list_lock);
> > -
> >         return 0;
> >  }
> >
> >  void
> > -rte_log_set_history(int enable)
> > +rte_log_set_history(int enable __rte_unused)
> >  {
> > -       history_enabled = enable;
> >  }
>
> Maybe a warning here for the people who did not read the deprecation
> notices ?
>
>
> --
> David Marchand
>
  

Looking a bit more into librte_ivshmem, the documentation says we need
a Qemu patch but the URL doesn't exist anymore:
	https://01.org/packet-processing/intel%C2%AE-ovdk
	-> 404 Oops, we couldn't find that page

I've never understood why we should keep this wart and now I'm going
to be upset.
To sum up the situation, eal depends on ivshmem which depends on
ring/mempool which depends... on eal.
The truth is that eal should not depends on librte_ivshmem.
And the option CONFIG_RTE_LIBRTE_IVSHMEM should not exist.

There are 3 parts to distinguish:

1/ The librte_ivshmem API to export some data structures from host.
No real problem here.

2/ The scan of the ivshmem devices in the guest init.
It should be handled as any other PCI device with an appropriate driver.
The scan is done by rte_eal_pci_init.

3/ The automatic mapped allocation of DPDK objects in the guest.
It should not be done in EAL.
An ivshmem driver would be called by rte_eal_dev_init.
It would check where are the shared DPDK structures, as currently done
with the IVSHMEM_MAGIC (0x0BADC0DE), and do the appropriate allocations.
Thus only the driver would depend on ring and mempool.

The last step of the ivshmem cleanup will be to remove the memory hack
RTE_EAL_SINGLE_FILE_SEGMENTS. Then CONFIG_RTE_LIBRTE_IVSHMEM could be
removed.

So this is my proposal:
Someone start working on the above cleanup now, otherwise the whole
rte_ivshmem feature will be deprecated in 16.07 and removed in 16.11.
We already talked about the rte_ivshmem design issues several times
and nobody declared using it.


---- original thread for reference ----

2016-06-09 17:01, Thomas Monjalon:
> 2016-06-09 16:45, David Marchand:
> > - Since you are looking at this, what keeps us from removing the
> > dependency on librte_ring ?
> 
> Please see this first small cleanup:
> 	http://dpdk.org/ml/archives/dev/2016-June/040798.html
> 
> > I would say it was mainly because of mempool.
> > Maybe ivshmem ?
> 
> Yes CONFIG_RTE_LIBRTE_IVSHMEM brings dependencies to rte_ring and rte_ivshmem.
> This "feature" also pollute the memory allocator and makes rework harder.
> That's why I would be in favor of removing CONFIG_RTE_LIBRTE_IVSHMEM.
> 
> Otherwise, as an alternative proposal, the file
> 	lib/librte_eal/linuxapp/eal/eal_ivshmem.c
> could be moved outside of EAL. Probably that lib/librte_ivshmem/
> would be a good place.
> The tricky operation would be to remove ivshmem init from eal:
> 
> #ifdef RTE_LIBRTE_IVSHMEM
>     if (rte_eal_ivshmem_init() < 0)                                                                              
>         rte_panic("Cannot init IVSHMEM\n");
> #endif
> 
>     if (rte_eal_memory_init() < 0)
>         rte_panic("Cannot init memory\n");
> 
>     /* the directories are locked during eal_hugepage_info_init */
>     eal_hugedirs_unlock();
> 
>     if (rte_eal_memzone_init() < 0)
>         rte_panic("Cannot init memzone\n");
> 
>     if (rte_eal_tailqs_init() < 0)
>         rte_panic("Cannot init tail queues for objects\n");
> 
> #ifdef RTE_LIBRTE_IVSHMEM
>     if (rte_eal_ivshmem_obj_init() < 0)
>         rte_panic("Cannot init IVSHMEM objects\n");
> #endif
  

Hi Thomas,

Just a few notes:

> 3/ The automatic mapped allocation of DPDK objects in the guest.
> It should not be done in EAL.
> An ivshmem driver would be called by rte_eal_dev_init.
> It would check where are the shared DPDK structures, as currently done with
> the IVSHMEM_MAGIC (0x0BADC0DE), and do the appropriate allocations.
> Thus only the driver would depend on ring and mempool.

The problem here is IVSHMEM doesn't allocate the memory from DPDK, it allocates new memory segments by mapping a PCI device. I.e. it doesn't do mallocs, it modifies mem_config and adds memory to DPDK. Can that be done from within a PMD?

> 
> The last step of the ivshmem cleanup will be to remove the memory hack
> RTE_EAL_SINGLE_FILE_SEGMENTS. Then CONFIG_RTE_LIBRTE_IVSHMEM
> could be removed.

The reason for that hack is that we often need to map several hugepages, and some of those pages could be 2M in size. If you're sharing 1G worth of contiguous memory backed by 2M pages, that's 512 files in the command line in vanilla DPDK, but can be made into one with RTE_EAL_SINGLE_FILE_SEGMENTS, so that QEMU command-line doesn't get overly long.

So removing this hack, while definitely desired, will adversely affect some use cases, such as using IVSHMEM on platforms where 1G pages aren't supported. Whether we want to go with the effort of supporting those is of course an open question - I personally don't have any data on IVSHMEM userbase. Maybe Kevin/other OVS devs could help me out here :)

Thanks,
Anatoly
  

2016-06-10 09:05, Burakov, Anatoly:
> Hi Thomas,
> 
> Just a few notes:
> 
> > 3/ The automatic mapped allocation of DPDK objects in the guest.
> > It should not be done in EAL.
> > An ivshmem driver would be called by rte_eal_dev_init.
> > It would check where are the shared DPDK structures, as currently done with
> > the IVSHMEM_MAGIC (0x0BADC0DE), and do the appropriate allocations.
> > Thus only the driver would depend on ring and mempool.
> 
> The problem here is IVSHMEM doesn't allocate the memory from DPDK, it allocates new memory segments by mapping a PCI device. I.e. it doesn't do mallocs, it modifies mem_config and adds memory to DPDK. Can that be done from within a PMD?

Everything is possible :)
Maybe you just need to add an API to add some memory segments.
Other question: why is it so important to register these memory segments
in EAL? I think they just need to be known by the ivshmem driver which
map some objects on top.

> > The last step of the ivshmem cleanup will be to remove the memory hack
> > RTE_EAL_SINGLE_FILE_SEGMENTS. Then CONFIG_RTE_LIBRTE_IVSHMEM
> > could be removed.
> 
> The reason for that hack is that we often need to map several hugepages, and some of those pages could be 2M in size. If you're sharing 1G worth of contiguous memory backed by 2M pages, that's 512 files in the command line in vanilla DPDK, but can be made into one with RTE_EAL_SINGLE_FILE_SEGMENTS, so that QEMU command-line doesn't get overly long.
> 
> So removing this hack, while definitely desired, will adversely affect some use cases, such as using IVSHMEM on platforms where 1G pages aren't supported. Whether we want to go with the effort of supporting those is of course an open question - I personally don't have any data on IVSHMEM userbase. Maybe Kevin/other OVS devs could help me out here :)

We can keep supporting 2M pages by having a command line option,
instead of the #ifdef RTE_EAL_SINGLE_FILE_SEGMENTS.
But as I said, it is not the top priority to remove this hack.
  

> > Hi Thomas,
> >
> > Just a few notes:
> >
> > > 3/ The automatic mapped allocation of DPDK objects in the guest.
> > > It should not be done in EAL.
> > > An ivshmem driver would be called by rte_eal_dev_init.
> > > It would check where are the shared DPDK structures, as currently
> > > done with the IVSHMEM_MAGIC (0x0BADC0DE), and do the appropriate
> allocations.
> > > Thus only the driver would depend on ring and mempool.
> >
> > The problem here is IVSHMEM doesn't allocate the memory from DPDK, it
> allocates new memory segments by mapping a PCI device. I.e. it doesn't do
> mallocs, it modifies mem_config and adds memory to DPDK. Can that be
> done from within a PMD?
> 
> Everything is possible :)
> Maybe you just need to add an API to add some memory segments.
> Other question: why is it so important to register these memory segments in
> EAL? I think they just need to be known by the ivshmem driver which map
> some objects on top.

That's because we need the memzone_lookup functionality. We can get by without it with rings because those are tailq-based, so we can just put rings there, but memzones are looked up through the memconfig, so IVSHMEM memzones have to be present there in order for the code to work without any additional API's.

Although, I guess we don't really need to have _memsegs_ in order to lookup memzones - we just have to create some memzones directly inside mcfg, bypassing the normal memzone_reserve stuff. That would still be a hack, but probably much less of a hack than what there is right now :) 

Another possible issue here is the order in which the memory is allocated. We put IVSHMEM init in EAL because we have to map things at specific addresses. The later IVSHMEM initializes, the more chance something will take up memory space that IVSHMEM needs. This could probably be solved with --base-virtaddr, so documentation will have to be updated to include advice to use that flag.

> 
> > > The last step of the ivshmem cleanup will be to remove the memory
> > > hack RTE_EAL_SINGLE_FILE_SEGMENTS. Then
> CONFIG_RTE_LIBRTE_IVSHMEM
> > > could be removed.
> >
> > The reason for that hack is that we often need to map several hugepages,
> and some of those pages could be 2M in size. If you're sharing 1G worth of
> contiguous memory backed by 2M pages, that's 512 files in the command line
> in vanilla DPDK, but can be made into one with
> RTE_EAL_SINGLE_FILE_SEGMENTS, so that QEMU command-line doesn't get
> overly long.
> >
> > So removing this hack, while definitely desired, will adversely affect
> > some use cases, such as using IVSHMEM on platforms where 1G pages
> > aren't supported. Whether we want to go with the effort of supporting
> > those is of course an open question - I personally don't have any data
> > on IVSHMEM userbase. Maybe Kevin/other OVS devs could help me out
> here
> > :)
> 
> We can keep supporting 2M pages by having a command line option, instead
> of the #ifdef RTE_EAL_SINGLE_FILE_SEGMENTS.
> But as I said, it is not the top priority to remove this hack.

Ah, so you're not suggesting removing the _functionality_, just the #ifdef? That could be made to work I guess...

Also, please correct me if I'm wrong, but I seem to remember some patches about putting all memory in a single file - I think that should work for IVSHMEM as well, because I believe IVSHMEM handles holes in files just fine, and can map even if everything resides inside a single file. So if that patch does what I think it does, we might just integrate it and remove the single file segments code entirely.

Thanks,
Anatoly
  

2016-06-10 09:47, Burakov, Anatoly:
> > > > The last step of the ivshmem cleanup will be to remove the memory
> > > > hack RTE_EAL_SINGLE_FILE_SEGMENTS. Then CONFIG_RTE_LIBRTE_IVSHMEM
> > > > could be removed.
> > >
> > > The reason for that hack is that we often need to map several hugepages,
> > and some of those pages could be 2M in size. If you're sharing 1G worth of
> > contiguous memory backed by 2M pages, that's 512 files in the command line
> > in vanilla DPDK, but can be made into one with
> > RTE_EAL_SINGLE_FILE_SEGMENTS, so that QEMU command-line doesn't get
> > overly long.
> > >
> > > So removing this hack, while definitely desired, will adversely affect
> > > some use cases, such as using IVSHMEM on platforms where 1G pages
> > > aren't supported. Whether we want to go with the effort of supporting
> > > those is of course an open question - I personally don't have any data
> > > on IVSHMEM userbase. Maybe Kevin/other OVS devs could help me out
> > here
> > > :)
> > 
> > We can keep supporting 2M pages by having a command line option, instead
> > of the #ifdef RTE_EAL_SINGLE_FILE_SEGMENTS.
> > But as I said, it is not the top priority to remove this hack.
> 
> Ah, so you're not suggesting removing the _functionality_, just the #ifdef? That could be made to work I guess...
> 
> Also, please correct me if I'm wrong, but I seem to remember some patches about putting all memory in a single file - I think that should work for IVSHMEM as well, because I believe IVSHMEM handles holes in files just fine, and can map even if everything resides inside a single file. So if that patch does what I think it does, we might just integrate it and remove the single file segments code entirely.

Yes it can be considered in a memory allocator rework.
Please jump in this thread:
	http://dpdk.org/ml/archives/dev/2016-April/037444.html
  

Hi Thomas, 

> 2016-06-10 09:47, Burakov, Anatoly:
> > > > > The last step of the ivshmem cleanup will be to remove the
> > > > > memory hack RTE_EAL_SINGLE_FILE_SEGMENTS. Then
> > > > > CONFIG_RTE_LIBRTE_IVSHMEM could be removed.
> > > >
> > > > The reason for that hack is that we often need to map several
> > > > hugepages,
> > > and some of those pages could be 2M in size. If you're sharing 1G
> > > worth of contiguous memory backed by 2M pages, that's 512 files in
> > > the command line in vanilla DPDK, but can be made into one with
> > > RTE_EAL_SINGLE_FILE_SEGMENTS, so that QEMU command-line doesn't
> get
> > > overly long.
> > > >
> > > > So removing this hack, while definitely desired, will adversely
> > > > affect some use cases, such as using IVSHMEM on platforms where 1G
> > > > pages aren't supported. Whether we want to go with the effort of
> > > > supporting those is of course an open question - I personally
> > > > don't have any data on IVSHMEM userbase. Maybe Kevin/other OVS
> > > > devs could help me out
> > > here
> > > > :)
> > >
> > > We can keep supporting 2M pages by having a command line option,
> > > instead of the #ifdef RTE_EAL_SINGLE_FILE_SEGMENTS.
> > > But as I said, it is not the top priority to remove this hack.
> >
> > Ah, so you're not suggesting removing the _functionality_, just the #ifdef?
> That could be made to work I guess...
> >
> > Also, please correct me if I'm wrong, but I seem to remember some
> patches about putting all memory in a single file - I think that should work for
> IVSHMEM as well, because I believe IVSHMEM handles holes in files just fine,
> and can map even if everything resides inside a single file. So if that patch
> does what I think it does, we might just integrate it and remove the single file
> segments code entirely.
> 
> Yes it can be considered in a memory allocator rework.
> Please jump in this thread:
> 	http://dpdk.org/ml/archives/dev/2016-April/037444.html

I've read through that thread, and I don't think anything should be done to support that in IVSHMEM. IVSHMEM library should correctly detect all mapped files, whatever the memory layout on the host side (contiguous, noncontiguous, single file...). On the guest, we're simply adding memzones which we map from a PCI device, so IVSHMEM shouldn't be affected by any changes to how memzones/mempools are allocated (mempools aren't even allowed for use with IVSHMEM).

Thanks,
Anatoly
  

2016-06-10 12:08, Burakov, Anatoly:
> Hi Thomas, 
> 
> > 2016-06-10 09:47, Burakov, Anatoly:
> > > > > > The last step of the ivshmem cleanup will be to remove the
> > > > > > memory hack RTE_EAL_SINGLE_FILE_SEGMENTS. Then
> > > > > > CONFIG_RTE_LIBRTE_IVSHMEM could be removed.
> > > > >
> > > > > The reason for that hack is that we often need to map several
> > > > > hugepages,
> > > > and some of those pages could be 2M in size. If you're sharing 1G
> > > > worth of contiguous memory backed by 2M pages, that's 512 files in
> > > > the command line in vanilla DPDK, but can be made into one with
> > > > RTE_EAL_SINGLE_FILE_SEGMENTS, so that QEMU command-line doesn't
> > get
> > > > overly long.
> > > > >
> > > > > So removing this hack, while definitely desired, will adversely
> > > > > affect some use cases, such as using IVSHMEM on platforms where 1G
> > > > > pages aren't supported. Whether we want to go with the effort of
> > > > > supporting those is of course an open question - I personally
> > > > > don't have any data on IVSHMEM userbase. Maybe Kevin/other OVS
> > > > > devs could help me out
> > > > here
> > > > > :)
> > > >
> > > > We can keep supporting 2M pages by having a command line option,
> > > > instead of the #ifdef RTE_EAL_SINGLE_FILE_SEGMENTS.
> > > > But as I said, it is not the top priority to remove this hack.
> > >
> > > Ah, so you're not suggesting removing the _functionality_, just the #ifdef?
> > That could be made to work I guess...
> > >
> > > Also, please correct me if I'm wrong, but I seem to remember some
> > patches about putting all memory in a single file - I think that should work for
> > IVSHMEM as well, because I believe IVSHMEM handles holes in files just fine,
> > and can map even if everything resides inside a single file. So if that patch
> > does what I think it does, we might just integrate it and remove the single file
> > segments code entirely.
> > 
> > Yes it can be considered in a memory allocator rework.
> > Please jump in this thread:
> > 	http://dpdk.org/ml/archives/dev/2016-April/037444.html
> 
> I've read through that thread, and I don't think anything should be done to support that in IVSHMEM. IVSHMEM library should correctly detect all mapped files, whatever the memory layout on the host side (contiguous, noncontiguous, single file...). On the guest, we're simply adding memzones which we map from a PCI device, so IVSHMEM shouldn't be affected by any changes to how memzones/mempools are allocated (mempools aren't even allowed for use with IVSHMEM).

You misunderstood me. I was talking about a rework of
RTE_EAL_SINGLE_FILE_SEGMENTS. I just say it must be reworked and
it can happen in the discussion of a global rework of the memory allocator.
It should not be related to IVSHMEM.
  

On 6/9/2016 10:26 PM, Thomas Monjalon wrote:
> Looking a bit more into librte_ivshmem, the documentation says we need
> a Qemu patch but the URL doesn't exist anymore:
> 	https://01.org/packet-processing/intel%C2%AE-ovdk
> 	-> 404 Oops, we couldn't find that page
> 
> I've never understood why we should keep this wart and now I'm going
> to be upset.
> To sum up the situation, eal depends on ivshmem which depends on
> ring/mempool which depends... on eal.
> The truth is that eal should not depends on librte_ivshmem.
> And the option CONFIG_RTE_LIBRTE_IVSHMEM should not exist.
> 
> There are 3 parts to distinguish:
> 
> 1/ The librte_ivshmem API to export some data structures from host.
> No real problem here.
> 
> 2/ The scan of the ivshmem devices in the guest init.
> It should be handled as any other PCI device with an appropriate driver.
> The scan is done by rte_eal_pci_init.
> 
> 3/ The automatic mapped allocation of DPDK objects in the guest.
> It should not be done in EAL.
> An ivshmem driver would be called by rte_eal_dev_init.
> It would check where are the shared DPDK structures, as currently done
> with the IVSHMEM_MAGIC (0x0BADC0DE), and do the appropriate allocations.
> Thus only the driver would depend on ring and mempool.
> 
> The last step of the ivshmem cleanup will be to remove the memory hack
> RTE_EAL_SINGLE_FILE_SEGMENTS. Then CONFIG_RTE_LIBRTE_IVSHMEM could be
> removed.
> 
> So this is my proposal:
> Someone start working on the above cleanup now, otherwise the whole
> rte_ivshmem feature will be deprecated in 16.07 and removed in 16.11.

I would like to note that at least SPP (soft patch panel) is using
rte_ivshmem feature.

> We already talked about the rte_ivshmem design issues several times
> and nobody declared using it.
>
  

2016-06-15 19:16, Ferruh Yigit:
> On 6/9/2016 10:26 PM, Thomas Monjalon wrote:
> > So this is my proposal:
> > Someone start working on the above cleanup now, otherwise the whole
> > rte_ivshmem feature will be deprecated in 16.07 and removed in 16.11.
> 
> I would like to note that at least SPP (soft patch panel) is using
> rte_ivshmem feature.

Why?
Can SPP work without ivshmem?
  

On 6/15/2016 7:34 PM, Thomas Monjalon wrote:
> 2016-06-15 19:16, Ferruh Yigit:
>> On 6/9/2016 10:26 PM, Thomas Monjalon wrote:
>>> So this is my proposal:
>>> Someone start working on the above cleanup now, otherwise the whole
>>> rte_ivshmem feature will be deprecated in 16.07 and removed in 16.11.
>>
>> I would like to note that at least SPP (soft patch panel) is using
>> rte_ivshmem feature.
> 
> Why?
> Can SPP work without ivshmem?
> 

Can do, it supports both ivshmem and vhost.
But ivshmem gives clearly more performance for VM to VM communication.
  

2016-06-20 16:44, Ferruh Yigit:
> On 6/15/2016 7:34 PM, Thomas Monjalon wrote:
> > 2016-06-15 19:16, Ferruh Yigit:
> >> On 6/9/2016 10:26 PM, Thomas Monjalon wrote:
> >>> So this is my proposal:
> >>> Someone start working on the above cleanup now, otherwise the whole
> >>> rte_ivshmem feature will be deprecated in 16.07 and removed in 16.11.
> >>
> >> I would like to note that at least SPP (soft patch panel) is using
> >> rte_ivshmem feature.
> > 
> > Why?
> > Can SPP work without ivshmem?
> > 
> 
> Can do, it supports both ivshmem and vhost.
> But ivshmem gives clearly more performance for VM to VM communication.

But SPP is not about performance? My understanding is that it supports
what DPDK supports.
And the question here is not about the benefit of QEMU ivshmem but the
current design of DPDK object sharing via ivshmem (librte_ivshmem).
  

On 06/10/2016 12:26 AM, Thomas Monjalon wrote:
> Looking a bit more into librte_ivshmem, the documentation says we need
> a Qemu patch but the URL doesn't exist anymore:
> 	https://01.org/packet-processing/intel%C2%AE-ovdk
> 	-> 404 Oops, we couldn't find that page
>
> I've never understood why we should keep this wart and now I'm going
> to be upset.

Good :)

> To sum up the situation, eal depends on ivshmem which depends on
> ring/mempool which depends... on eal.
> The truth is that eal should not depends on librte_ivshmem.
> And the option CONFIG_RTE_LIBRTE_IVSHMEM should not exist.
>
> There are 3 parts to distinguish:
>
> 1/ The librte_ivshmem API to export some data structures from host.
> No real problem here.
>
> 2/ The scan of the ivshmem devices in the guest init.
> It should be handled as any other PCI device with an appropriate driver.
> The scan is done by rte_eal_pci_init.
>
> 3/ The automatic mapped allocation of DPDK objects in the guest.
> It should not be done in EAL.
> An ivshmem driver would be called by rte_eal_dev_init.
> It would check where are the shared DPDK structures, as currently done
> with the IVSHMEM_MAGIC (0x0BADC0DE), and do the appropriate allocations.
> Thus only the driver would depend on ring and mempool.
>
> The last step of the ivshmem cleanup will be to remove the memory hack
> RTE_EAL_SINGLE_FILE_SEGMENTS. Then CONFIG_RTE_LIBRTE_IVSHMEM could be
> removed.
>
> So this is my proposal:
> Someone start working on the above cleanup now, otherwise the whole
> rte_ivshmem feature will be deprecated in 16.07 and removed in 16.11.
> We already talked about the rte_ivshmem design issues several times
> and nobody declared using it.

+1 (more like +100) to that.

In addition to the technical mess in EAL, there are quite some 
eyebrow-raisers related to IVSHMEM:

That it all starts with "you'll need to build a special version of qemu" 
with this special patch from the 'net, a patch which doesn't even exist 
anymore, is a complete non-starter. Such a situation can occur during 
early development, but its been years by now. Dependencies to 
non-upstreamed features in other projects are not a healthy sign.

Regardless of whether the patch has been integrated to qemu upstream or 
not, the situation is quite telling: nobody cares enough to have updated 
the information. I found a copy of the patch from my laptop, and as far 
as I can tell, the patch has never been proposed upstream, much less 
applied. Certainly the patch would not come even close to applying to 
current qemu. And apparently IVSHMEM is unmaintained in qemu upstream 
too (according to MAINTAINERS).

On DPDK side, that the most obvious (to me at least) user of memnic PMD 
has been unmaintained for two years no, and allowed to fall off the edge 
of the world (witness http://dpdk.org/browse/memnic/) is also quite telling.

Just deprecate it already. If somebody shows up with actual patches to 
clean it all up, the deprecation can be lifted of course, but cleaning 
up this abandonware seems like waste of engineering resources to me.

	- Panu -