v1.0 - cleanup

README.md

@@ -1,3 +1,84 @@
 # libinstaller
 
-A simple Python tool to install libraries with commonly used compiler- and MPI combinations 
+This Python tool has been developed to automate the installation of libraries frequently used by HPC applications.
+
+## Basic Usage
+To install the library `<lib>` and all it's dependencies (if there are any) using compiler `<compiler>` and MPI `<mpi>`, make sure to have a configuration file `<lib>.json` in the `config` directory and invoke the tool as follows:
+```
+./libinstaller --compiler <compiler> --mpi <mpi> --<lib>
+```
+To install all libraries with build recipes in `config`:
+```
+./libinstaller --compiler <compiler> --mpi <mpi> --all
+```
+For further supported options and their default values, please have a look at `./libinstaller -h`:
+```
+usage: libinstaller [-h] [--config CONFIG] [--prefix PREFIX] [--src SRC] [--work WORK] [--keep-work] [--compiler COMPILER] [--mpi MPI] [--threads THREADS] [--verbose] [--separate-lib64] [--disable-shared] [--all]
+
+options:
+  -h, --help           show this help message and exit
+  --config CONFIG      Path to config directory [$pwd/config]
+  --prefix PREFIX      Path where install directory should be generated [$pwd]
+  --src SRC            Path where to download source code to [$pwd/src]
+  --work WORK          Path to working directory for builds [$pwd/work]
+  --keep-work          Disable removal of work directory after successful builds
+  --compiler COMPILER  Select compiler (gnu, intel, aocc) [gnu]
+  --mpi MPI            Select compiler (hpcx, intelmpi, openmpi) [hpcx]
+  --threads THREADS    Number of threads used for make [8]
+  --verbose            Print build output to screen instead piping it to logfile
+  --separate-lib64     Do not create symbolic links of files from lib64 in lib
+  --disable-shared     Disable building of shared libraries
+  --all                Install all libraries with config file in config/
+  --ignore-deps        Do not add dependencies of libraries to the install process
+```
+For each library configuration file available in `config`, the corresponding `--<lib>` and `--<lib>-version` options are added to the parser automatically, eg:
+```
+--hdf5                          Enable build of hdf5
+--hdf5-version HDF5_VERSION     Set hdf5 version [1.13.2]
+```
+The name of the installation directory is automatically generated and has the form
+```
+<prefix>/inst-<compiler>_<compiler_version>_<mpi>_<mpi_version>
+```
+
+## Library configuration files
+The `config` directory contains various json files describing how to obtain and build the corresponding library. It must have the following structure (all keys are mandantory):
+```
+{
+    "name" : "mylib",
+    "dependencies" : "lib_a,lib_b",
+    "default version" : "x.y.z",
+    "download" : "git clone --depth 1 --branch hdf5-$VERSION https://myurl.com/mylib.git mylib-$VERSION",
+    "configure" : "autoreconf -fi ; ./configure --config-option --prefix=$PREFIX $SHARED",
+    "build" : "make -j $BUILDTHREADS",
+    "install" : "make install",
+    "object files" : "123"
+}
+```
+### name, default version
+On startup, the installer tool will read the configuration files in `config` and add coreesponding options to the argument parser. In case `--<lib>-version` is not set, the default version from the json file is used.
+
+### dependencies
+Dependencies are used to determine the best installation order of the selected libraries to satisfy all dependencies.
+When the installer encounters a library to install, all dependencies are also added to the installation process if not already selected. In case `--ignore-deps` is set, the dependencies need to be activated manually when invoking libinstaller or be present in `LD_LIBRARY_PATH`.
+Note that `$PREFIX/lib` is automatically added to `LD_LIBRARY_PATH`
+
+### download
+Download command. This is being executed in a subshell. Make sure that it works in the shell used to start libinstaller.
+The command may contain the variables `$VERSION` which will be replaced by the selected code version of the form `XXX.YYY.ZZZ` or `$VERSION_UNDERSCORE` which will be replaced by `XXX_YYY_ZZZ`. 
+The files will be downloaded to the `<src>`directory
+
+### configure
+Configure command. This is being executed in a subshell. Make sure that it works in the shell used to start libinstaller. 
+The variable `$PREFIX` will be replaced by the installation directory. The vairable `$SHARED` will be replaced by `--disable-shared` in case the respective option is set when invoking libinstaller.
+
+### build
+Build command. This is being executed in a subshell. Make sure that it works in the shell used to start libinstaller. 
+The variable `$BUILDTHREADS` will be replaced by the number of threads set when invoking libinstaller (`--threads`).
+
+### install
+Build command. This is being executed in a subshell. Make sure that it works in the shell used to start libinstaller. 
+The variable `$PREFIX` will be replaced by the installation directory.
+
+### object files
+The number of `*.o` files generated during compilation. It will be used to draw progress bars in case `--verbose` is not set.

config/aec.json

@@ -0,0 +1,10 @@
+{
+    "name" : "aec",
+    "dependencies" : "",
+    "default version" : "1.0.6",
+    "download" : "git clone --depth 1 --branch v$VERSION https://gitlab.dkrz.de/k202009/libaec.git  aec-$VERSION",
+    "configure" : "autoreconf -iv; ./configure --prefix=$PREFIX $SHARED",
+    "build" : "make -j $BUILDTHREADS",
+    "install" : "make install",
+    "object files" : "10"
+}

config/curl.json

@@ -0,0 +1,10 @@
+{
+    "name" : "curl",
+    "dependencies" : "",
+    "default version" : "7.85.0",
+    "download" : "git clone --depth 1 --branch curl-$VERSION_UNDERSCORE https://github.com/curl/curl.git curl-$VERSION",
+    "configure" : "autoreconf -fi; ./configure --with-openssl --prefix=$PREFIX",
+    "build" : "make -j $BUILDTHREADS",
+    "install" : "make install",
+    "object files" : "362"
+}

config/eccodes.json

@@ -0,0 +1,10 @@
+{
+    "name" : "eccodes",
+    "default version" : "2.27.0",
+    "dependencies" : "zlib,szip,curl,aec,xml2,netcdf-c,netcdf-f,hdf5",
+    "download" : "wget https://confluence.ecmwf.int/download/attachments/45757960/eccodes-$VERSION-Source.tar.gz ; tar zxf eccodes-$VERSION-Source.tar.gz ; mv eccodes-$VERSION-Source eccodes-$VERSION",
+    "configure" : "mkdir build; cd build; cmake .. -DCMAKE_INSTALL_PREFIX=$PREFIX",
+    "build" : "cd build; make -j $BUILDTHREADS",
+    "install" : "cd build; make install",
+    "object files" : "515"
+}

config/hdf5.json

@@ -3,8 +3,8 @@
     "dependencies" : "zlib,szip",
     "default version" : "1.13.2",
     "download" : "git clone --depth 1 --branch hdf5-$VERSION_UNDERSCORE https://github.com/HDFGroup/hdf5.git hdf5-$VERSION",
-    "configure" : "./configure --enable-fortran --enable-parallel --with-zlib=$PREFIX --with-szip=$PREFIX --prefix=$PREFIX",
+    "configure" : "./configure --enable-fortran --enable-parallel --with-zlib=$PREFIX --with-szip=$PREFIX --prefix=$PREFIX $SHARED",
     "build" : "make -j $BUILDTHREADS",
     "install" : "make install",
     "object files" : "1136"
-}
+}

config/netcdf-c.json

@@ -0,0 +1,10 @@
+{
+    "name" : "netcdf-c",
+    "default version" : "4.7.4",
+    "dependencies" : "zlib,szip,curl,hdf5",
+    "download" : "git clone --depth 1 --branch v$VERSION https://github.com/Unidata/netcdf-c.git netcdf-c-$VERSION",
+    "configure" : "./configure --prefix=$PREFIX CPPFLAGS=-I$PREFIX/include LDFLAGS=-L$PREFIX/lib $SHARED",
+    "build" : "make -j $BUILDTHREADS",
+    "install" : "make install",
+    "object files" : "435"
+}

config/netcdf-f.json

@@ -0,0 +1,10 @@
+{
+    "name" : "netcdf-f",
+    "default version" : "4.5.3",
+    "dependencies" : "zlib,szip,hdf5,curl,netcdf-c",
+    "download" : "git clone --depth 1 --branch v$VERSION https://github.com/Unidata/netcdf-fortran.git netcdf-f-$VERSION",
+    "configure" : "./configure --prefix=$PREFIX CPPFLAGS=-I$PREFIX/include LDFLAGS=-L$PREFIX/lib  $SHARED",
+    "build" : "make -j $BUILDTHREADS",
+    "install" : "make install",
+    "object files" : "60"
+}

config/netcdf_c.json

@@ -1,4 +0,0 @@
-{
-    "name" : "netcdf-c",
-    "default version" : "4.2.1"
-}

config/netcdf_f.json

@@ -1,4 +0,0 @@
-{
-    "name" : "netcdf-f",
-    "default version" : "4.6.0"
-}

config/szip.json

@@ -3,7 +3,7 @@
     "dependencies" : "",
     "default version" : "2.1.1",
     "download" : "git clone https://github.com/erdc/szip.git szip-$VERSION",
-    "configure" : "./configure --prefix=$PREFIX",
+    "configure" : "./configure --prefix=$PREFIX $SHARED",
     "build" : "make -j $BUILDTHREADS",
     "install" : "make install",
     "object files" : "7"

config/xml2.json

@@ -0,0 +1,10 @@
+{
+    "name" : "xml2",
+    "dependencies" : "",
+    "default version" : "2.8.0",
+    "download" : "git clone --depth 1 --branch v$VERSION https://gitlab.gnome.org/GNOME/libxml2.git xml2-$VERSION",
+    "configure" : "./autogen.sh ; ./configure --prefix=$PREFIX $SHARED",
+    "build" : "make -j $BUILDTHREADS",
+    "install" : "make install",
+    "object files" : "133"
+}

lib/dependency.py

@@ -0,0 +1,102 @@
+import os
+import glob
+from lib.io import load_lib_data
+
+def topological_sort(source):
+    pending = [(name, set(deps)) for name, deps in source]
+    emitted = []
+    while pending:
+        next_pending = []
+        next_emitted = []
+        for entry in pending:
+            name, deps = entry
+            deps.difference_update(set((name,)), emitted)
+            if deps:
+                next_pending.append(entry)
+            else:
+                yield name
+                emitted.append(name)
+                next_emitted.append(name)
+        if not next_emitted:
+            raise ValueError("cyclic dependancy detected: %s %r" % (name, (next_pending,)))
+        pending = next_pending
+        emitted = next_emitted
+    return emitted
+
+
+def sort_libs_by_dependencies(selected_libs):
+    # no need for sorting in case of only one lib
+    if len(selected_libs) < 2:
+        return selected_libs
+
+    lib_names = []
+    for lib in selected_libs:
+        lib_names.append(lib['name'])
+
+    deplist = []
+    for lib in selected_libs:
+        name = lib['name']
+        # only sort after dependencies that are actually present to avoid cyclic dependencies
+        deps = []
+        for dep in lib['dependencies'].split(','):
+            if dep in lib_names:
+                deps.append(dep)
+        dependencies = set(deps)
+        if dependencies == {''}:
+            dependencies = {}
+        deplist.append([name, dependencies])
+    sorted_deplist = topological_sort(deplist)
+
+    sorted = []
+    for entry in sorted_deplist:
+        for lib in selected_libs:
+            if lib['name'] == entry:
+                sorted.append(lib)
+    return sorted
+
+
+def load_dependencies(library, selected_libs, config_dir):
+    lib_names = []
+    for lib in selected_libs:
+        lib_names.append(lib['name'])
+
+    dependencies = library['dependencies'].split(',')
+    for dep in dependencies:
+        if len(dep) > 1:
+            if dep not in lib_names:
+                config_file = config_dir + "/" + dep + ".json"
+                if os.path.exists(config_file):
+                    dep_data = load_lib_data(config_file)
+                    dep_data['version'] = dep_data['default version']
+                    selected_libs.append(dep_data)
+                else:
+                    print("Warning - library " + lib['name'] + " depends on " + dep + " for which no configration file was found. Presence in LD_LIBRARY_PATH is assumed.")
+
+
+# load library data from names in argparse result
+def load_selected_libs(config_dir, arg_namespace, args, install_all_libs, ignore_deps):
+    selected_libs = []
+    if install_all_libs:
+        for config_file in glob.glob(config_dir+"/*.json"):
+            data = load_lib_data(config_file)
+            data['version'] = data['default version']
+            selected_libs.append(data)
+    else:
+        ignore_names = ["config", "mpi", "compiler", "prefix", "src", "work", "keep_work", "threads", "verbose", "version", "disable_shared", "ignore_deps"]
+        for lib_name in args:
+            if lib_name not in ignore_names and "version" not in lib_name:
+                install = getattr(arg_namespace, lib_name)
+                if install:
+                    version = getattr(arg_namespace, lib_name + "_version")
+                    lib_name = lib_name.replace('_','-')
+                    config_file = config_dir + "/" + lib_name + ".json"
+                    data = load_lib_data(config_file)
+                    data['version'] = version
+                    selected_libs.append(data)
+
+    if not ignore_deps:
+        # also add all dependencies to the install list
+        for lib in selected_libs:
+            load_dependencies(lib, selected_libs, config_dir)
+
+    return selected_libs

lib/init.py

@@ -2,43 +2,59 @@ import os
 import argparse
 import glob
 import json
+import sys
+
+def check_python_version():
+    python_version = sys.version_info
+    if not (python_version.major >= 3 and python_version.minor >= 3):
+        print("Minimum Python version needed: 3.3.0")
+        sys.exit(1)
+
 
 def init():
-    # set up two parsers: first parse config directory, and add dynamic arguments to full parser
+    # set up two parsers: first parse config directory,  and add dynamic arguments to full parser
     config_parser = argparse.ArgumentParser(add_help=False)
     parser = argparse.ArgumentParser()
 
     # parsers need to share known arguments
-    config_parser.add_argument('--config',help='Path to config directory [$pwd/config]',default=os.getcwd()+"/config")
-    config_parser.add_argument('--prefix',help='Path where install directory should be generated [$pwd]',default=os.getcwd())
-    config_parser.add_argument('--src' ,help='Path where to download source  code to [$pwd/src]',default=os.getcwd()+"/src")
-    config_parser.add_argument('--work',help='Path to working directory for builds [$pwd/work',default=os.getcwd()+"/work")
-    config_parser.add_argument('--keep-work',help='Disable removal of work directory after successful builds',action='store_true')
-    config_parser.add_argument('--compiler',help='Select compiler (gnu,intel,aocc) [gnu]',default='gnu')
-    config_parser.add_argument('--mpi',help='Select compiler (hpcx,intelmpi,openmpi) [hpcx]',default='hpcx')
-    config_parser.add_argument('--threads',help='Number of threads used for make [8]',default='8')
-    config_parser.add_argument('--verbose',help='Print build output to screen instead piping it to logile',action='store_true')
+    config_parser.add_argument('--config', help='Path to config directory [$pwd/config]', default=os.getcwd()+"/config")
+    config_parser.add_argument('--prefix', help='Path where install directory should be generated [$pwd]', default=os.getcwd())
+    config_parser.add_argument('--src', help='Path where to download source  code to [$pwd/src]', default=os.getcwd()+"/src")
+    config_parser.add_argument('--work', help='Path to working directory for builds [$pwd/work', default=os.getcwd()+"/work")
+    config_parser.add_argument('--keep-work', help='Disable removal of work directory after successful builds', action='store_true')
+    config_parser.add_argument('--compiler', help='Select compiler (gnu, intel, aocc) [gnu]', default='gnu')
+    config_parser.add_argument('--mpi', help='Select compiler (hpcx, intelmpi, openmpi) [hpcx]', default='hpcx')
+    config_parser.add_argument('--threads', help='Number of threads used for make [8]', default='8')
+    config_parser.add_argument('--verbose', help='Print build output to screen instead piping it to logfile', action='store_true')
+    config_parser.add_argument('--separate-lib64', help='Do not create symbolic links of files from lib64 in lib', action='store_true')
+    config_parser.add_argument('--disable-shared', help='Disable building of shared libraries', action='store_true')
+    config_parser.add_argument('--all', help='Install all libraries with config file in config/', action='store_true')
+    config_parser.add_argument('--ignore-deps', help='Do not add dependencies of libraries to the install process', action='store_true')
 
-    parser.add_argument('--config',help='Path to config directory [$pwd/config]',default=os.getcwd()+"/config")
-    parser.add_argument('--prefix',help='Path where install directory should be generated [$pwd]',default=os.getcwd())
-    parser.add_argument('--src' ,help='Path where to download source  code to [$pwd/src]',default=os.getcwd()+"/src")
-    parser.add_argument('--work' ,help='Path to working directory for builds [$pwd/work]',default=os.getcwd()+"/work")
-    parser.add_argument('--keep-work',help='Disable removal of work directory after successful builds',action='store_true')
-    parser.add_argument('--compiler',help='Select compiler (gnu,intel,aocc) [gnu]',default='gnu')
-    parser.add_argument('--mpi',help='Select compiler (hpcx,intelmpi,openmpi) [hpcx]',default='hpcx')
-    parser.add_argument('--threads',help='Number of threads used for make [8]',default='8')
-    parser.add_argument('--verbose',help='Print build output to screen instead piping it to logile',action='store_true')
+    parser.add_argument('--config', help='Path to config directory [$pwd/config]', default=os.getcwd()+"/config")
+    parser.add_argument('--prefix', help='Path where install directory should be generated [$pwd]', default=os.getcwd())
+    parser.add_argument('--src', help='Path where to download source  code to [$pwd/src]', default=os.getcwd()+"/src")
+    parser.add_argument('--work', help='Path to working directory for builds [$pwd/work]', default=os.getcwd()+"/work")
+    parser.add_argument('--keep-work', help='Disable removal of work directory after successful builds', action='store_true')
+    parser.add_argument('--compiler', help='Select compiler (gnu, intel, aocc) [gnu]', default='gnu')
+    parser.add_argument('--mpi', help='Select compiler (hpcx, intelmpi, openmpi) [hpcx]', default='hpcx')
+    parser.add_argument('--threads', help='Number of threads used for make [8]', default='8')
+    parser.add_argument('--verbose', help='Print build output to screen instead piping it to logfile', action='store_true')
+    parser.add_argument('--separate-lib64', help='Do not create symbolic links of files from lib64 in lib', action='store_true')
+    parser.add_argument('--disable-shared', help='Disable building of shared libraries', action='store_true')
+    parser.add_argument('--all', help='Install all libraries with config file in config/', action='store_true')
+    parser.add_argument('--ignore-deps', help='Do not add dependencies of libraries to the install process', action='store_true')
 
-    # run config parser and serach  config/*.json to add a build and version argument for it to the full parser
+    # run config parser and search config/*.json to add a build and version argument for it to the full parser
     config_dir = config_parser.parse_known_args()[0].config
     for cf in glob.glob(config_dir+"/*.json"):
         with open(cf, 'r') as f:
             lib_data = json.load(f)
-            parser.add_argument('--'+lib_data['name'],help='Enable build of '+lib_data['name'],action='store_true')
-            parser.add_argument('--'+lib_data['name']+'-version',help='Set '+lib_data['name']+' version ['+lib_data['default version']+']',default=str(lib_data['default version']))
+            parser.add_argument('--'+lib_data['name'], help='Enable build of '+lib_data['name'], action='store_true')
+            parser.add_argument('--'+lib_data['name']+'-version', help='Set '+lib_data['name']+' version ['+lib_data['default version']+']', default=str(lib_data['default version']))
 
     # run full parser
     arg_namespace = parser.parse_args()
     args = vars(arg_namespace)
 
-    return arg_namespace,args
+    return arg_namespace, args

lib/installer.py

@@ -4,10 +4,11 @@ import subprocess
 import time
 import glob
 import shutil
-from lib.ui import bordered, underlined, progressbar
+from lib.ui import bordered,  underlined,  progressbar
 
-def install_lib(lib,src_dir,work_dir,inst_dir,comp_cc,comp_cxx,comp_fc,build_threads,verbose):
-    print(bordered("Installing "+lib['name']+" v."+lib['version']))
+
+def install_lib(lib, src_dir, work_dir, inst_dir, comp_cc, comp_cxx, comp_fc, build_threads, disable_shared_libs, verbose):
+    print(bordered("Installing " + lib['name'] + " v." + lib['version']))
     if not os.path.exists(src_dir):
         os.makedirs(src_dir)
     if not os.path.exists(work_dir):
@@ -19,11 +20,11 @@ def install_lib(lib,src_dir,work_dir,inst_dir,comp_cc,comp_cxx,comp_fc,build_thr
 
     lib_name = lib['name']
     lib_version = lib['version']
-    if verbose == False:
+    if not verbose:
         logfile_path = work_dir + "/" + lib_name + "-" + lib_version + ".log"
         if os.path.exists(logfile_path):
             os.remove(logfile_path)
-        logfile = open(logfile_path,'a')
+        logfile = open(logfile_path, 'a')
     else:
         logfile = None
 
@@ -31,49 +32,62 @@ def install_lib(lib,src_dir,work_dir,inst_dir,comp_cc,comp_cxx,comp_fc,build_thr
     if not os.path.exists(lib['name']+"-"+lib['version']):
 
         print("Downloading Source Code")
-        load_command = lib['download'].replace("$VERSION_UNDERSCORE",lib_version.replace(".","_")).replace("$VERSION",lib_version).split()
-        err = subprocess.call(load_command,stdout=logfile,stderr=logfile)
+        load_command = lib['download'].replace("$VERSION_UNDERSCORE", lib_version.replace(".", "_")).replace("$VERSION", lib_version)
+        err = subprocess.call(load_command, stdout=logfile, stderr=logfile, shell=True)
         if err != 0:
             print("Error: Download of "+lib_name+" v."+lib_version+" failed!")
             print("See "+logfile_path+" for details")
             sys.exit(0)
+
     # configure library
     os.chdir(work_dir)
     if os.path.exists(work_dir+"/"+lib_name+"-"+lib_version):
         shutil.rmtree(work_dir+"/"+lib_name+"-"+lib_version)
-    shutil.copytree(src_dir+"/"+lib_name+"-"+lib_version, work_dir+"/"+lib_name+"-"+lib_version)
+    shutil.copytree(src_dir+"/"+lib_name+"-"+lib_version,  work_dir+"/"+lib_name+"-"+lib_version)
     os.chdir(work_dir+"/"+lib_name+"-"+lib_version)
 
-    config_command = lib['configure'].replace("$PREFIX",inst_dir)
+    if disable_shared_libs:
+        shared_option = "--disable-shared"
+    else:
+        shared_option = ""
+    config_command = lib['configure'].replace("$PREFIX", inst_dir).replace("$SHARED",shared_option)
+
+    # TODO: THIS IS UGLY AND NEEDS TO BE MOVED TO EITHER LIB OR COMPILER CONFIGURATION
+    # workaround for bad autotools reconization of flang
+    if (lib_name == "hdf5" or lib_name == "netcdf-f") and "aocc" in inst_dir:
+        config_command = config_command + " FCFLAGS=-fPIC"
+
     if verbose:
         print(underlined("\nConfiguring Library"))
     else:
         print("Configuring library...")
-    err = subprocess.call(config_command,stdout=logfile,stderr=logfile,shell=True)
+    err = subprocess.call(config_command, stdout=logfile, stderr=logfile, shell=True)
     if err != 0:
         print("Error: Configuring of "+lib_name+" v."+lib_version+" failed!")
         print("See "+logfile_path+" for details")
         sys.exit(1)
 
+    # TODO: THIS IS UGLY AND NEEDS TO BE MOVED TO EITHER LIB OR COMPILER CONFIGURATION
+    # workaround for bad autotools reconization of flang
+    if (lib_name == "hdf5" or lib_name == "netcdf-f") and "aocc" in inst_dir:
+        err = subprocess.call("sed -i -e 's/wl=\"\"/wl=\"-Wl,\"/g' libtool", shell=True)
+
     # build library
-    build_command = lib['build'].replace("$BUILDTHREADS",build_threads)
+    build_command = lib['build'].replace("$BUILDTHREADS", build_threads)
     if verbose:
         print(underlined("\nBuilding Library"))
-        err = subprocess.call(build_command,stdout=logfile,stderr=logfile,shell=True)
+        err = subprocess.call(build_command, stdout=logfile, stderr=logfile, shell=True)
     else:
-        build_task = subprocess.Popen(build_command,stdout=logfile,stderr=logfile,shell=True)
+        build_task = subprocess.Popen(build_command, stdout=logfile, stderr=logfile, shell=True)
         num_objects = int(lib['object files'])
         finished = False
-        while build_task.poll() == None:
+        while build_task.poll() is None:
             if not finished:
-                # bar_width = num_objects
-                # while (bar_width > 100):
-                #    bar_width = int(bar_width/10)
                 bar_width = 50
-                for i in progressbar(range(num_objects), "Building library: ",bar_width):
-                    while len(glob.glob('**/*.o',recursive=True)) < i:
+                for i in progressbar(range(num_objects),  "Building library: ", bar_width):
+                    while len(glob.glob('**/*.o', recursive=True)) < i:
                         time.sleep(0.5)
-                        if(build_task.poll != None):
+                        if build_task.poll is not None:
                             break
                 finished = True
         build_task.wait()
@@ -84,20 +98,22 @@ def install_lib(lib,src_dir,work_dir,inst_dir,comp_cc,comp_cxx,comp_fc,build_thr
             print("See "+logfile_path+" for details")
             sys.exit(1)
 
-    #install library
-    install_command = lib['install'].replace("$BUILDTHREADS",build_threads)
+    # install library
+    install_command = lib['install'].replace("$BUILDTHREADS", build_threads)
     if verbose:
         print(underlined("\nInstalling Library"))
     else:
         print("Installing library...")
-    err = subprocess.call(install_command,stdout=logfile,stderr=logfile,shell=True)
+    err = subprocess.call(install_command, stdout=logfile, stderr=logfile, shell=True)
     if err != 0:
             print("Error: Installing "+lib_name+" v."+lib_version+" failed!")
             print("See "+logfile_path+" for details")
             sys.exit(1)
     print("Library "+lib_name+" has been installed successfully!\n")
-    build_output_path = inst_dir+"/"+"build_info"
-    if not os.path.exists(build_output_path):
-            os.makedirs(build_output_path)
-    shutil.copyfile(logfile_path,build_output_path+"/"+lib_name+".log")
 
+    # copy logfile to install location
+    if not verbose:
+        build_output_path = inst_dir+"/"+"build_info"
+        if not os.path.exists(build_output_path):
+                os.makedirs(build_output_path)
+        shutil.copyfile(logfile_path, build_output_path+"/"+lib_name+".log")

lib/io.py

@@ -0,0 +1,8 @@
+import json
+
+
+# load library data from json file
+def load_lib_data(lib_path):
+    with open (lib_path, 'r') as config_file:
+        data = json.load(config_file)
+        return data

lib/shell.py

@@ -1,6 +1,7 @@
 import subprocess
 import sys
 
+
 def get_from_command(command):
     ret = subprocess.check_output(command).decode(sys.stdout.encoding)
     return ret

lib/sort.py

@@ -6,12 +6,12 @@ def topological_sort(source):
         next_emitted = []
         for entry in pending:
             name, deps = entry
-            deps.difference_update(set((name,)), emitted) # <-- pop self from dep, req Py2.6
+            deps.difference_update(set((name,)), emitted)  # <-- pop self from dep, req Py2.6
             if deps:
                 next_pending.append(entry)
             else:
                 yield name
-                emitted.append(name) # <-- not required, but preserves original order
+                emitted.append(name)  # <-- not required, but preserves original order
                 next_emitted.append(name)
         if not next_emitted:
             raise ValueError("cyclic dependancy detected: %s %r" % (name, (next_pending,)))
@@ -19,15 +19,30 @@ def topological_sort(source):
         emitted = next_emitted
     return emitted
 
+
 def sort_libs_by_dependencies(selected_libs):
-    deplist=[]
+    # no need for sorting in case of only one lib
+    if len(selected_libs) < 2:
+        return selected_libs
+
+    lib_names = []
+    for lib in selected_libs:
+        lib_names.append(lib['name'])
+
+    deplist = []
     for lib in selected_libs:
         name = lib['name']
-        dependencies = set(lib['dependencies'].split(','))
+        # only sort after dependencies that are actually present to avoid cyclic dependencies
+        deps = []
+        for dep in lib['dependencies'].split(','):
+            if dep in lib_names:
+                deps.append(dep)
+        dependencies = set(deps)
         if dependencies == {''}:
-         dependencies = {}
-        deplist.append([name,dependencies])
+            dependencies = {}
+        deplist.append([name, dependencies])
     sorted_deplist = topological_sort(deplist)
+
     sorted = []
     for entry in sorted_deplist:
         for lib in selected_libs:

lib/toolchain.py

@@ -1,16 +1,25 @@
 import os
+import sys
 from lib.shell import get_from_command
 
+
 def get_compiler_version(compiler):
     if compiler == "gnu":
-        cc="gcc"
+        cc = "gcc"
+        compstr = get_from_command([cc, "-dumpversion"]).strip()
     elif compiler == "intel":
-        cc="icc"
+        cc = "icc"
+        compstr = get_from_command([cc, "-dumpversion"]).strip()
     elif compiler == "aocc":
-        cc="clang"
-    compstr = get_from_command([cc,"-dumpversion"]).strip()
+        cc = "clang"
+        compstr = get_from_command([cc, "-dumpversion"]).strip()
+    elif compiler == "nec":
+        cc = "ncc"
+        rawstr = get_from_command([cc, "--version"])
+        compstr = rawstr.split()[2]
     return compstr
 
+
 def get_mpi_version(mpi):
     if mpi == "hpcx":
         hpcx_dir = os.getenv('HPCX_DIR')
@@ -22,24 +31,34 @@ def get_mpi_version(mpi):
         rawstr = get_from_command(["ompi_info"]).splitlines()[1]
         mpistr = rawstr[rawstr.find(":")+2:]
     elif mpi == "intelmpi":
-        rawstr = get_from_command(["ompi_info"]).splitlines()[0]
-        mpistr = rawstr[rawstr.find("Version")+8:rawstr.find("Build")-1]
-    elif mpi == "mpich":
         rawstr = get_from_command(["mpirun","--version"])
+        mpistr = rawstr.split()[7]
+    elif mpi == "mpich":
+        rawstr = get_from_command(["mpirun", "--version"])
+        mpistr = rawstr.split()[4]
+    elif mpi == "necmpi":
+        rawstr = get_from_command(["mpirun", "--version"])
         mpistr = rawstr.split()[4]
     return mpistr
 
 
-def set_toolchain(compiler,mpi):
+def set_toolchain(compiler, mpi):
     if compiler == "gnu":
         if mpi == "hpcx" or mpi == "openmpi" or mpi == "mpich":
             cc = "mpicc"
             cxx = "mpic++"
             fc = "mpifort"
-        elif mpi == "intel":
+        elif mpi == "intelmpi":
             cc = "mpicc"
             cxx = "mpicxx"
             fc = "mpif90"
+        elif mpi == "necmpi":
+            cc = "mpincc -vh"
+            cxx = "mpinc++ -vh"
+            fc = "mpinfort -vh"
+            os.environ["NMPI_CC_H"] = "gcc"
+            os.environ["NMPI_CXX_H"] = "g++"
+            os.environ["NMPI_FC_H"] = "gfortran"
     elif compiler == "aocc":
         if mpi == "hpcx" or mpi == "openmpi":
             cc = "mpicc"
@@ -50,10 +69,17 @@ def set_toolchain(compiler,mpi):
             os.environ["OMPI_MPIFC"] = "flang"
             os.environ["OMPI_MPIF90"] = "flang"
             os.environ["OMPI_MPIF77"] = "flang"
-        elif (mpi == "intel"):
-            cc = "\"mpicc -cc=clang\""
-            cxx = "\"mpicxx -cxx=clang++\""
-            fc = "\"mpif90 -fc=flang\""
+        elif (mpi == "intelmpi"):
+            cc = "mpicc -cc=clang"
+            cxx = "mpicxx -cxx=clang++"
+            fc = "mpif90 -fc=flang"
+        elif mpi == "necmpi":
+            cc = "mpincc -vh"
+            cxx = "mpic++ -vh"
+            fc = "mpinfort -vh"
+            os.environ["NMPI_CC_H"] = "clang"
+            os.environ["NMPI_CXX_H"] = "clang++"
+            os.environ["NMPI_FC_H"] = "flang"
     elif compiler == "intel":
         if mpi == "hpcx" or mpi == "openmpi":
             cc = "mpicc"
@@ -64,11 +90,23 @@ def set_toolchain(compiler,mpi):
             os.environ["OMPI_MPIFC"] = "ifort"
             os.environ["OMPI_MPIF90"] = "ifort"
             os.environ["OMPI_MPIF77"] = "ifort"
-        elif mpi == "intel":
+        elif mpi == "intelmpi":
             cc = "mpiicc"
             cxx = "mpiicpc"
             fc = "mpiifort"
-        
+        elif mpi == "necmpi":
+            cc = "mpincc -vh"
+            cxx = "mpic++ -vh"
+            fc = "mpinfort -vh"
+            os.environ["NMPI_CC_H"] = "icc"
+            os.environ["NMPI_CXX_H"] = "icpc"
+            os.environ["NMPI_FC_H"] = "ifort"
+    elif compiler == "nec":
+        # only NEC MPI
+        cc = "mpincc"
+        cxx = "mpinc++"
+        fc = "mpinfort"
+
     # set environment variables
     os.environ["CC"] = cc
     os.environ["CXX"] = cxx
@@ -76,4 +114,4 @@ def set_toolchain(compiler,mpi):
     os.environ["F90"] = fc
     os.environ["F77"] = fc
 
-    return cc,cxx,fc
+    return cc, cxx, fc

lib/ui.py

@@ -1,18 +1,21 @@
 import sys
 import time
 
-def progressbar(it, prefix="", size=60, out=sys.stdout): 
+
+def progressbar(it, prefix="", size=60, out=sys.stdout):
     count = len(it)
+
     def show(j):
         x = int(size*j/count)
-        print("{}[{}{}] {}/{}".format(prefix, u"█"*x, "."*(size-x), j, count),
-                end='\r', file=out, flush=True)
+        print("{}[{}{}] {}/{}".format(prefix, u"█"*x, "."*(size-x), j, count), end='\r', file=out, flush=True)
+
     show(0)
     for i, item in enumerate(it):
         yield item
         show(i+1)
     print("", flush=True, file=out)
 
+
 def bordered(text):
     lines = text.splitlines()
     width = max(len(s) for s in lines)
@@ -22,19 +25,28 @@ def bordered(text):
     res.append('└' + '─' * width + '┘')
     return '\n'.join(res)
 
+
 def underlined(text):
     lines = text.splitlines()
     width = max(len(s) for s in lines)
-    res=[]
+    res = []
     for s in lines:
         res.append(s)
     res.append('─' * width)
     return '\n'.join(res)
 
-def print_welcome(script_version, compiler, compiler_version, mpi, mpi_version, instDir, installs):
+
+def print_welcome(script_version, compiler, compiler_version, mpi, mpi_version, build_threads, disable_shared_libs, instDir, installs):
+    if not disable_shared_libs:
+        shared_str = "Yes"
+    else:
+        shared_str = "No"
+    
     out = underlined("Patricks Simple Library Installer " + script_version)
     out = out + "\n" + "Compiler: " + compiler.upper() + ", Version: " + compiler_version
     out = out + "\n" + "MPI: " + mpi.upper() + ", Version: " + mpi_version
+    out = out + "\n" + "Number of threads to use: " + build_threads
+    out = out + "\n" + "Building of shared libraries: " + shared_str
     out = out + "\n" + "Install Directory: "+instDir
     out = out + "\n"
     out = out + "\n" + "The following Libraries will be installed:"
@@ -43,7 +55,7 @@ def print_welcome(script_version, compiler, compiler_version, mpi, mpi_version,
     print(bordered(out) + "\n")
 
     # short sleep to enable user to abprt if selected options are wrong
-    for i in progressbar(range(5), "Waiting 5s to start build: ",10):
+    for i in progressbar(range(5), "Waiting 5s to start build: ", 10):
         time.sleep(1)
 
     print("\n")

libinstaller

@@ -0,0 +1,80 @@
+#!/usr/bin/env python3
+import shutil
+import os
+import glob
+from lib.ui import bordered, print_welcome
+from lib.toolchain import get_compiler_version, get_mpi_version, set_toolchain
+from lib.dependency import sort_libs_by_dependencies, load_selected_libs
+from lib.installer import install_lib
+from lib.init import init, check_python_version
+
+SCRIPT_VERSION = "v1.0"
+
+# check if Python >=3.3.0 is used
+check_python_version()
+
+# initialization by argparser
+arg_namespace, args = init()
+
+# extract known arguments from argparse namespace
+prefix = arg_namespace.prefix
+config_dir = arg_namespace.config
+src_dir = arg_namespace.src
+work_dir = arg_namespace.work
+keep_work = arg_namespace.keep_work
+compiler = arg_namespace.compiler
+mpi = arg_namespace.mpi
+build_threads = arg_namespace.threads
+verbose = arg_namespace.verbose
+separate_lib64 = arg_namespace.separate_lib64
+disable_shared = arg_namespace.disable_shared
+install_all_libs = arg_namespace.all
+ignore_deps = arg_namespace.ignore_deps
+
+# extract libraries and versions selected for installation
+selected_libs = load_selected_libs(config_dir, arg_namespace, args, install_all_libs, ignore_deps)
+
+# set up install directory name
+compiler_version = get_compiler_version(compiler)
+mpi_version = get_mpi_version(mpi)
+inst_str = "inst-"+compiler + "_" + compiler_version + "_" + mpi + "_" + mpi_version
+inst_dir = prefix + "/" + inst_str
+
+# set up mpi wrappers and environment variables
+cc, cxx, fc = set_toolchain(compiler, mpi)
+
+# set up library install order by dependencies
+sorted_libs = sort_libs_by_dependencies(selected_libs)
+
+# print welcome screen
+print_welcome(SCRIPT_VERSION, compiler, compiler_version, mpi, mpi_version, build_threads, disable_shared, inst_dir, sorted_libs)
+
+# install selected libraries
+if len(sorted_libs) > 0:
+    # add install dir to environment
+    os.environ["LIBRARY_PATH"] = inst_dir + "/lib" + os.pathsep + inst_dir + "/lib64" + os.pathsep + os.environ["LIBRARY_PATH"]
+    os.environ["LD_LIBRARY_PATH"] = inst_dir + "/lib" + os.pathsep + inst_dir + "/lib64" + os.pathsep + os.environ["LD_LIBRARY_PATH"]
+    # install libraries
+    for lib in sorted_libs:
+        install_lib(lib, src_dir, work_dir, inst_dir, cc, cxx, fc, build_threads, disable_shared, verbose)
+    print(bordered("ALL INSTALLS COMPLETED SUCCESSFULLY\nPlease add "+inst_dir+" to your environment"))
+else:
+    print("NO LIBRARIES SELECTED FOR INSTALLATION")
+
+# create symbolic links
+if not separate_lib64:
+    lib_dir = inst_dir + "/lib"
+    lib64_dir = inst_dir + "/lib64"
+    if os.path.exists(lib_dir) and os.path.exists(lib64_dir):
+        print("Creating symbolic links...")
+        for file in glob.glob(lib64_dir+"/*.*"):
+            if not os.path.exists(file.replace("lib64","lib")):
+                os.symlink(file, file.replace("lib64","lib"))
+
+# cleanup
+if (not keep_work and os.path.exists(work_dir)):
+    print("Cleaning up work directory...")
+    shutil.rmtree(work_dir)
+    print("Done.")
+
+print("\n")

libinstaller.py

@@ -1,71 +0,0 @@
-#!/usr/bin/env python3
-import shutil
-import json
-import os
-from lib.ui import progressbar, bordered, underlined, print_welcome
-from lib.shell import get_from_command
-from lib.toolchain import get_compiler_version, get_mpi_version, set_toolchain
-from lib.sort import sort_libs_by_dependencies
-from lib.installer import install_lib
-from lib.init import init
-
-SCRIPT_VERSION="v0.3"
-
-# initialization by argparser
-arg_namespace,args = init()
-
-# extract known arguments from argparse namespace
-prefix = arg_namespace.prefix
-config_dir = arg_namespace.config
-src_dir = arg_namespace.src
-work_dir = arg_namespace.work
-keep_work = arg_namespace.keep_work
-compiler = arg_namespace.compiler
-mpi = arg_namespace.mpi
-build_threads = arg_namespace.threads
-verbose = arg_namespace.verbose
-
-# extract libraries and versions selected for installation
-selected_libs=[]
-ignore_names=["config", "mpi", "compiler", "prefix", "src", "work", "threads", "verbose", "version"]
-for lib_name in args:
-    if (lib_name not in ignore_names and "version" not in lib_name):
-        install = getattr(arg_namespace,lib_name)
-        if (install == True):
-            version = getattr(arg_namespace,lib_name+"_version")
-            config_file = config_dir+"/"+lib_name+".json"
-            with open(config_file,'r') as cf:
-                data = json.load(cf)
-                data['version'] = version
-                selected_libs.append(data)
-
-# set up install directory name
-compiler_version=get_compiler_version(compiler)
-mpi_version=get_mpi_version(mpi)
-inst_str="inst-"+compiler+"_"+compiler_version+"_"+mpi+"_"+mpi_version
-inst_dir=prefix+"/"+inst_str
-
-# set up mpi wrappers and environment variables   
-cc,cxx,fc = set_toolchain(compiler,mpi)
-
-# set up library install order by dependencies
-sorted_libs = sort_libs_by_dependencies(selected_libs)
-
-# print welcome screen
-print_welcome(SCRIPT_VERSION, compiler, compiler_version, mpi, mpi_version, inst_dir, sorted_libs)
-
-# install selected libraries
-if len(sorted_libs) > 0:
-    for lib in sorted_libs:
-        install_lib(lib,src_dir,work_dir,inst_dir,cc,cxx,fc,build_threads,verbose)
-    print(bordered("ALL INSTALLS COMPLETED SUCCESSFULLY\nPlease add "+inst_dir+" to your environment"))
-else:
-    print("NO LIBRARIES SELECTED FOR INSTALLATION")
-
-# cleanup
-if (not keep_work and os.path.exists(work_dir)):
-    print("Cleaning up work directory...")
-    shutil.rmtree(work_dir)
-    print("Done.")
-
-print("\n")