[dpdk-dev] scripts: add script for generating customised build config

Message ID 1461061640-7611-1-git-send-email-bruce.richardson@intel.com (mailing list archive)
State Superseded, archived
Delegated to: Thomas Monjalon
Headers

Commit Message

Bruce Richardson April 19, 2016, 10:27 a.m. UTC
  This patch adds in the dpdk_config.py script file. It can be used
to generate custom build-time configurations for DPDK either manually on
the commandline or by calling it from other scripts. It takes as parameters
the base config template to use, and output directory to which the modified
configuration will be written. Other optional parameters are then taken
as y/n values which should be adjusted in the config file, and a special
-m flag is provided to override the default RTE_MACHINE setting in the
config template too.

Example, to create a build configuration with extra non-default PMDs
enabled, and the kernel modules disabled:

  ./scripts/dpdk_config.py -b $RTE_TARGET -o $RTE_TARGET PMD_PCAP=y \
     IGB_UIO=n KNI_KMOD=n MLX4_PMD=y MLX5_PMD=y SZEDATA2=y \
     NFP_PMD=y BNX2X_PMD=y

See the included help text in the script for more details.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 scripts/dpdk_config.py | 198 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 198 insertions(+)
 create mode 100755 scripts/dpdk_config.py
  

Comments

Thomas Monjalon April 19, 2016, 12:30 p.m. UTC | #1
Hi Bruce,

Thanks for pushing this idea.

2016-04-19 11:27, Bruce Richardson:
> This patch adds in the dpdk_config.py script file. It can be used
> to generate custom build-time configurations for DPDK either manually on
> the commandline or by calling it from other scripts. It takes as parameters
> the base config template to use, and output directory to which the modified
> configuration will be written. Other optional parameters are then taken
> as y/n values which should be adjusted in the config file, and a special
> -m flag is provided to override the default RTE_MACHINE setting in the
> config template too.
> 
> Example, to create a build configuration with extra non-default PMDs
> enabled, and the kernel modules disabled:
> 
>   ./scripts/dpdk_config.py -b $RTE_TARGET -o $RTE_TARGET PMD_PCAP=y \
>      IGB_UIO=n KNI_KMOD=n MLX4_PMD=y MLX5_PMD=y SZEDATA2=y \
>      NFP_PMD=y BNX2X_PMD=y

Would it be possible to use it without -b option to update a configuration?

Why not name it scripts/configure.py with a symlink ./configure in the
top directory?

Should we be able to list every options for a "-b defconfig"?

Would it be a good idea to manage dependencies checks in this script?
  
Bruce Richardson April 19, 2016, 4:25 p.m. UTC | #2
On Tue, Apr 19, 2016 at 02:30:08PM +0200, Thomas Monjalon wrote:
> Hi Bruce,
> 
> Thanks for pushing this idea.
> 
> 2016-04-19 11:27, Bruce Richardson:
> > This patch adds in the dpdk_config.py script file. It can be used
> > to generate custom build-time configurations for DPDK either manually on
> > the commandline or by calling it from other scripts. It takes as parameters
> > the base config template to use, and output directory to which the modified
> > configuration will be written. Other optional parameters are then taken
> > as y/n values which should be adjusted in the config file, and a special
> > -m flag is provided to override the default RTE_MACHINE setting in the
> > config template too.
> > 
> > Example, to create a build configuration with extra non-default PMDs
> > enabled, and the kernel modules disabled:
> > 
> >   ./scripts/dpdk_config.py -b $RTE_TARGET -o $RTE_TARGET PMD_PCAP=y \
> >      IGB_UIO=n KNI_KMOD=n MLX4_PMD=y MLX5_PMD=y SZEDATA2=y \
> >      NFP_PMD=y BNX2X_PMD=y
> 
> Would it be possible to use it without -b option to update a configuration?
> 
Interesting idea, but what would that really give us over manual editing? If
calling from a script, the user can just rebuild the config from scratch.

> Why not name it scripts/configure.py with a symlink ./configure in the
> top directory?

No objections here. It's just not really a "normal" configure script, instead
it's one designed to make it easy for me to generate directories with lots of
different build settings in them.
If we do want to make it to be "the" way to build DPDK, we can perhaps look at
additional enhancements to speed it up by working with the defconfigs directly,
as I'm not happy right now with the speed of the "make config" step.

On the other hand, I'm very happy with having it short and of limited scope.
Actual code only takes up about 50% of the file. :-)

> 
> Should we be able to list every options for a "-b defconfig"?
> 
Good idea, I think.
Under what conditions? Only if an invalid config is provided, or as part of
regular help text or otherwise?

> Would it be a good idea to manage dependencies checks in this script?
> 
I'd rather not do so here.
a) I don't think there are a huge number of dependencies to manage
b) I'd like to keep it from getting too long and complicated.

For 95% of use cases, being able to set a few y/n values on and off, and change
the machine type should suffice. I wasn't going for a 100% solution.

/Bruce
  

Patch

diff --git a/scripts/dpdk_config.py b/scripts/dpdk_config.py
new file mode 100755
index 0000000..01236d8
--- /dev/null
+++ b/scripts/dpdk_config.py
@@ -0,0 +1,198 @@ 
+#! /usr/bin/env python
+#
+#   BSD LICENSE
+#
+#   Copyright(c) 2016 Intel Corporation. All rights reserved.
+#   All rights reserved.
+#
+#   Redistribution and use in source and binary forms, with or without
+#   modification, are permitted provided that the following conditions
+#   are met:
+#
+#     * Redistributions of source code must retain the above copyright
+#       notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above copyright
+#       notice, this list of conditions and the following disclaimer in
+#       the documentation and/or other materials provided with the
+#       distribution.
+#     * Neither the name of Intel Corporation nor the names of its
+#       contributors may be used to endorse or promote products derived
+#       from this software without specific prior written permission.
+#
+#   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+#   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+#   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+#   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+#   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+#   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+#   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+#   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+#   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+#   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+#   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+#
+
+from __future__ import print_function
+import sys
+import os.path
+import getopt
+import subprocess
+
+out = "build"
+base = None
+machine = None
+
+
+def print_usage(cmd, f):
+    print("""Usage: %s -b <base_config> [-o <output_dir>] [options...] """
+          % os.path.basename(cmd), file=f)
+
+
+def print_help(cmd):
+    print_usage(cmd, sys.stdout)
+    print("""
+Generates a new DPDK build time configuration in the given output directory,
+using as a starting point the provided base configuration. Once completed
+successfully, DPDK can then be built using that configuration by running the
+command "make -C <output_dir>".
+
+-b, --base-config=CONFIG
+    Use the configuration given by CONFIG as the starting point for the
+    build configuration. CONFIG must be a valid DPDK default configuration
+    provided by a file "defconfig_CONFIG" in the top level "config"
+    directory.
+    Example: "-b x86_64-native-linuxapp-gcc"
+
+-o, --output-dir=DIR
+    Use DIR as the resulting output directory. If unspecified the default
+    value of "build" is used.
+
+-m, --machine-type=TYPE
+    Override the machine-type value given in the default configuration by
+    setting it to TYPE. Using regular options, regular y/n values can be
+    changed, but not string options, so this explicit override for machine
+    type exists to allow changing it.
+
+-h, --help
+    Print this help text and then exit
+
+Other options passed after these on the commandline should be in the format of
+"name=y|n" and are overrides to be applied to the configuration to toggle the
+y|n settings in the file.
+
+Matches are applied on partial values, so for example "DEBUG=y" will turn on
+all options ending in "DEBUG". This means that a full option need not always be
+specified, for example, "CONFIG_RTE_LIBRTE_" prefixes can generally be omitted.
+
+Examples:
+
+To generate a configuration for DPDK
+- based off x86_64-native-linuxapp-gcc,
+- changing the machine type to "default",
+- turning on the pcap PMD,
+- disabling the KNI and igb_uio kernel modules, and
+- placing the output in a suitably named folder
+use:
+
+    %s -o x86_64-default-linuxapp-gcc -b x86_64-native-linuxapp-gcc \\
+        -m default PMD_PCAP=y IGB_UIO=n KNI_KMOD=n
+
+""" % os.path.basename(cmd))
+
+
+def parse_opts(argv):
+    global out, base, machine
+    long_opts = ["output-dir", "base-config", "machine-type", "help"]
+    try:
+        opts, args = getopt.getopt(argv[1:], "o:b:m:h", long_opts)
+    except getopt.GetoptError as err:
+        print(str(err))
+        print_usage(argv[0], sys.stderr)
+        sys.exit(1)
+
+    for o, a in opts:
+        if o in ["-o", "--output-dir"]:
+            out = a
+        elif o in ["-b", "--base-config"]:
+            base = a
+        elif o in ["-m", "--machine-type"]:
+            machine = a
+        elif o in ["-h", "--help"]:
+            print_help(argv[0])
+            sys.exit()
+        else:
+            print("Error: Unhandled option '%s'\n" % o,
+                  file=sys.stderr)
+            sys.exit(1)
+
+    if base is None:
+        print_usage(argv[0], sys.stderr)
+        sys.exit(1)
+    if not os.path.exists("config/defconfig_%s" % base):
+        print("Error, invalid base configuration specified: %s" % base)
+        sys.exit(1)
+
+    enable_args = [a[:-2] for a in args if a[-2:] == "=y"]
+    disable_args = [a[:-2] for a in args if a[-2:] == "=n"]
+    invalid = [a for a in args if a[-2:] not in ("=y", "=n")]
+    if len(invalid) > 0:
+        print("Error, options not ending in '=y' or '=n': %s"
+              % str(invalid))
+        sys.exit(1)
+    return (enable_args, disable_args)
+
+
+def run_make_config():
+    try:
+        # the flush parameter is not always implemented so provide fallback
+        print("Running 'make config'...", end=' ', flush=True)
+    except Exception:
+        print("Running 'make config'...")
+
+    cmd = "make config T=%s O=%s" % (base, out)
+    if sys.platform.startswith("freebsd"):
+        cmd = cmd.replace("make", "gmake", 1)
+    if subprocess.call(cmd.split()) != 0:
+        print("make config called failed")
+        sys.exit(1)
+
+
+def load_file(fn):
+    fd = open(fn)
+    contents = [line.strip() for line in fd.readlines()]
+    fd.close()
+    return contents
+
+
+def write_file(fn, config):
+    fd = open(fn, "w")
+    fd.writelines(["%s\n" % line for line in config])
+    fd.close()
+
+
+def change_settings(enable, disable):
+    config = load_file("%s/.config" % out)
+    print("Enabling settings %s" % str(enable))
+    for entry in enable:
+        config = [line.replace("%s=n" % entry, "%s=y" % entry)
+                  for line in config]
+    print("Disabling settings %s" % str(disable))
+    for entry in disable:
+        config = [line.replace("%s=y" % entry, "%s=n" % entry)
+                  for line in config]
+
+    if machine is not None:
+        print("Setting machine to '%s'" % machine)
+        config.append("CONFIG_RTE_MACHINE=\"%s\"" % machine)
+    write_file("%s/.config" % out, config)
+
+
+def main():
+    enable, disable = parse_opts(sys.argv)
+    run_make_config()
+    change_settings(enable, disable)
+    print("Done")
+
+
+if __name__ == "__main__":
+    main()