libinstaller

This Python tool has been developed to automate the installation of libraries frequently used by HPC applications.

Basic Usage

To install the library <lib> and all it's dependencies (if there are any) using compiler <compiler> and MPI <mpi>, make sure to have a configuration file <lib>.json in the config directory and invoke the tool as follows:

./libinstaller --compiler <compiler> --mpi <mpi> --<lib>

To install all libraries with build recipes in config:

./libinstaller --compiler <compiler> --mpi <mpi> --all

For further supported options and their default values, please have a look at ./libinstaller -h:

usage: libinstaller [-h] [--config CONFIG] [--prefix PREFIX] [--src SRC] [--work WORK] [--keep-work] [--compiler COMPILER] [--mpi MPI] [--threads THREADS] [--verbose] [--separate-lib64] [--disable-shared] [--all]

options:
  -h, --help           show this help message and exit
  --config CONFIG      Path to config directory [$pwd/config]
  --prefix PREFIX      Path where install directory should be generated [$pwd]
  --src SRC            Path where to download source code to [$pwd/src]
  --work WORK          Path to working directory for builds [$pwd/work]
  --keep-work          Disable removal of work directory after successful builds
  --compiler COMPILER  Select compiler (gnu, intel, aocc) [gnu]
  --mpi MPI            Select compiler (hpcx, intelmpi, openmpi) [hpcx]
  --threads THREADS    Number of threads used for make [8]
  --verbose            Print build output to screen instead piping it to logfile
  --separate-lib64     Do not create symbolic links of files from lib64 in lib
  --disable-shared     Disable building of shared libraries
  --all                Install all libraries with config file in config/
  --ignore-deps        Do not add dependencies of libraries to the install process

For each library configuration file available in config, the corresponding --<lib> and --<lib>-version options are added to the parser automatically, eg:

--hdf5                          Enable build of hdf5
--hdf5-version HDF5_VERSION     Set hdf5 version [1.13.2]

The name of the installation directory is automatically generated and has the form

<prefix>/inst-<compiler>_<compiler_version>_<mpi>_<mpi_version>

Library configuration files

The config directory contains various json files describing how to obtain and build the corresponding library. It must have the following structure (all keys are mandantory):

{
    "name" : "mylib",
    "dependencies" : "lib_a,lib_b",
    "default version" : "x.y.z",
    "download" : "git clone --depth 1 --branch hdf5-$VERSION https://myurl.com/mylib.git mylib-$VERSION",
    "configure" : "autoreconf -fi ; ./configure --config-option --prefix=$PREFIX $SHARED",
    "build" : "make -j $BUILDTHREADS",
    "install" : "make install",
    "object files" : "123"
}

name, default version

On startup, the installer tool will read the configuration files in config and add coreesponding options to the argument parser. In case --<lib>-version is not set, the default version from the json file is used.

dependencies

Dependencies are used to determine the best installation order of the selected libraries to satisfy all dependencies. When the installer encounters a library to install, all dependencies are also added to the installation process if not already selected. In case --ignore-deps is set, the dependencies need to be activated manually when invoking libinstaller or be present in LD_LIBRARY_PATH. Note that $PREFIX/lib is automatically added to LD_LIBRARY_PATH

download

Download command. This is being executed in a subshell. Make sure that it works in the shell used to start libinstaller. The command may contain the variables $VERSION which will be replaced by the selected code version of the form XXX.YYY.ZZZ or $VERSION_UNDERSCORE which will be replaced by XXX_YYY_ZZZ. The files will be downloaded to the <src>directory

configure

Configure command. This is being executed in a subshell. Make sure that it works in the shell used to start libinstaller. The variable $PREFIX will be replaced by the installation directory. The vairable $SHARED will be replaced by --disable-shared in case the respective option is set when invoking libinstaller.

build

Build command. This is being executed in a subshell. Make sure that it works in the shell used to start libinstaller. The variable $BUILDTHREADS will be replaced by the number of threads set when invoking libinstaller (--threads).

install

Build command. This is being executed in a subshell. Make sure that it works in the shell used to start libinstaller. The variable $PREFIX will be replaced by the installation directory.

object files

The number of *.o files generated during compilation. It will be used to draw progress bars in case --verbose is not set.