Pmodules - an improved environment modules system

4. BUILDING MODULES

4.1. Introduction

In the simplest case adding a module can be done by adding a modulefile to a certain directory. Below is an example of a "do nothing" modulefile.

Example: null/1.0.0

#%Pmodule

module-whatis           "does absolutely nothing"
module-maintainer       "Achim Gsell <achim.gsell@psi.ch>"
module-license          "MIT"

module-help     "
This is a do nothing module.
"

Modulefiles are written in the Tool Command Language (Tcl).

In principal, it is possible to install modules 'by hand' by installing the software and modulefiles at the right place. For reproducibility, documentation and re-usability it is highly recommended to write build-recipes for each module or - at least - a README.md and host everything on a Gitforge (gitea.psi.ch, github.com, gitlab.com …).

4.2. Predefined variables

In modulefiles and build-scripts the following variables are predefined:

P: the name of the module
V: the module version. The version number consists of a major- and a minor version number, a patch-level and a release number. All numbers but the major version number are optional. Major-, minor number and patch-level are separated by dots, the release number by a minus. Example: 1.0.3-2
V_MAJOR: the major version number. Example: 1
V_MINOR: the minor version number. Example: 0
V_PATCHLVL: the patch-level. Example: 3
V_RELEASE: the release number. Example: 2
V_PKG: version number without release.
PREFIX: installation prefix of the module.

In build-scripts the following variables are predefined too:

TEMP_DIR: directory for temporary files.
SRC_DIR: directory of unpacked sources, set to $PMODULES_TMPDIR/$P-$V/src
BUILD_DIR: build directory, set to $PMODULES_TMPDIR/$P-$V/build
BUILDBLOCK_DIR: Directory where the build-script is in.
BUILD_SCRIPT: Name of the build script.
JOBS: Number of parallel make jobs.

4.3. Building a module

Perform the following steps for a new module:

Create a directory with the name of the module.
Write the configuration file. See section Writing build configuration files.
Write the modulefile. See section [sec-modulefiles].
Code all required steps to install a software in a script. If it is not possible to script everything, code as much as possible and document additional steps in a README file. See section Writing build scripts.

The first - and sometimes also to most challenging - step in building a module from source is to learn how to download, configure, compile and install a software package and dependencies the package has. The next steps are writing a configuration file, a modulefile and a script to download, configure, compile and install it.

In general the following steps needs to be performed:

prepare: Download, unpack, apply patches (optional). Downloading the software might not be scriptable due to password protection or other measurements.
configure: (only for software distributed as source code) This step heavily depends on the software itself. In many cases autotools or CMake is used, in some cases you have to deal with Makefiles or other tools.
build: (only for software distributed as source code) compile/build everything
install: Install everything in the target directory. In case of a binary package this might not be scriptable.

4.4. Writing build configuration files

4.4.1. `files/config.yaml`

4.4.1.1. Structure

module-name-1:
  type: module | sub_package
  defaults:                                       # optional
    config-block
  shasums:                                        # optional
    filename-1: sha256sum-1
    ...
  versions:
    version-keys-1:
      config:                                     # optional
        config-block
      variants:                                   # optional
        - config-block-1
        - ...
    ...
module-name-2:
  ...

Field-Name Description

Field-Name	Description
`module-name-N`	Configuration for the module `module-name-N`. Example: `openmpi`
`type`	Type of the configuration block. Valid values are `module` and `sub_package`. Sub-packages are a convenient way to pack multiple software-packages into a single module. The default value is `module`
`defaults`	Default configuration. This block is optional.
`shasums`	SHA256 hash sums of source files. Keys are filenames, the value the corresponding SHA256 hash sum. This block is optional. A warning is printed if a hash-sum is missing for a required file.
`versions`	A version key is a semicolon separated string of version numbers. Version numbers can be specified with shell brace expansion. Example: `5.4.{0,1,2,3,4,5,8,9};5.2.{0,4,6,7,8};5.0.0;4.6.3`
`config`	Optional, version specific configuration block. Configurations specified here are overriding the defaults.
`variants`	Variants can be used to compile the some software with different options. Example: openmpi` with and without Slurm support are variants of the same software.

module-name-N

Configuration for the module module-name-N. Example: openmpi

type

Type of the configuration block. Valid values are module and sub_package. Sub-packages are a convenient way to pack multiple software-packages into a single module. The default value is module

defaults

Default configuration. This block is optional.

shasums

SHA256 hash sums of source files. Keys are filenames, the value the
corresponding SHA256 hash sum. This block is optional. A warning is printed if a hash-sum is missing for a required file.

versions

A version key is a semicolon separated string of version numbers. Version numbers can be specified with shell brace expansion. Example:
5.4.{0,1,2,3,4,5,8,9};5.2.{0,4,6,7,8};5.0.0;4.6.3

config

Optional, version specific configuration block. Configurations specified here are overriding the defaults.

variants

Variants can be used to compile the some software with different options. Example: openmpi` with and without Slurm support are variants of the same software.

4.4.1.2. Configuration blocks

Field-Name Description

Field-Name	Description
`build_requires`	A list of modules required to build this module. The default is an empty list.
`build_functions`	This structure can be used to define a list of functions to be called for each of the for build-steps. The build-functions defaults to the pre-defined build-functions. The schema of the structure is documented in a section below.
`build_variants`	If multiple variants exist for a version, this field can be used to build only the first variant that meets specific criteria. Criteria can be specified via the keys `kernels`, `systems` and `target_cpu`. Valid values are `all`, which is the default and `first_match`.
`compile_in_sourcetree`	Compile in source tree or in a dedicate build directory. Either `true` or `false`. The default value is `true`.
`configure_with`	Choose software configuration system. Allowed values are: `auto`: use autotools if available (this is the default value)+ `autotools`: use autotools `cmake`: use cmake+
`configure_args` `configure_args+`	List of arguments passed to the configuration system. The key `configure_args+` appends arguments to the list of previously defined arguments.
`default_variant`	Specifies the default variant to build if no variant has been passed on via the command-line. The default is an empty string.
`docfiles` `docfiles+`	List of documentation files to be installed in `$PREFIX/share/doc/`. The key `docfiles+` adds documentation files to the previously defined list. The default is an empty list.
`download_dir`	Directory where files are saved after downloading. See the structure `urls` for more details. The directory defaults to the directory specified with the environment variable `PMODULES_DISTFILES` or if the variable is not defined to `$HOME/.cache/Pmodules/distfiles`.
`group`	The name of the group in which the module is to be installed. The default group is `Tools`.
`group_deps`	This block is required if the module is to be installed in a hierarchical group. The schema of the structure is documented below.
`kernels`	List of kernels to which this configuration block applies. Valid values are `Linux`, `Darwin` and `any`. The default value is `any`.
`modulefile`	Name of the modulefile to be installed. The default is `modulefile`.
`overlay`	The name of the overlay in which the module is to be installed
`patch_files` `patch_files+`	List of patches to be applied on the sources. The default is an empty list.
`relstage`	The release stage of the module. Valid values are: `unstable`: the module is still under development `stable`: the module is ready to be used `deprecated`: the module is deprecated `remove`: the module is deprecated and marked to be removed `removed`: the module has been removed
`runtime_deps`	List of run-time dependencies of the module. These modules will be loaded together with the module.
`sub_packages`	A list of sub-package to be built. For details see below.
`suffix`	This string will be added to the version of the module as suffix.
`systems`	A list of systems to which this block applies. A 'system' is either a hostname or OS name like `rhel8`. For hostnames a regex like `.*.merlin7.psi.ch` can be used.
`target_cpus`	A list of CPU to which this block applies.
`urls` \| `sources`	List of sources from which the required software must be downloaded if not already cached. The details are documented in a following section.
`use_flags`	TBW
`use_overlays`	List of overlays that must be loaded before the module can be built
`variant`	Sequence of synonyms for a variant.

build_requires

A list of modules required to build this module. The default is an empty list.

build_functions

This structure can be used to define a list of functions to be called for each of the for build-steps. The build-functions defaults to the pre-defined build-functions. The schema of the structure is documented in a section below.

build_variants

If multiple variants exist for a version, this field can be used to build only the first variant that meets specific criteria. Criteria can be specified via the keys kernels, systems and target_cpu. Valid values are all, which is the default and first_match.

compile_in_sourcetree

Compile in source tree or in a dedicate build directory. Either true or false. The default value is true.

configure_with

Choose software configuration system. Allowed values are:
auto: use autotools if available (this is the default value)+ autotools: use autotools
cmake: use cmake+

configure_args
configure_args+

List of arguments passed to the configuration system. The key configure_args+ appends arguments to the list of previously defined arguments.

default_variant

Specifies the default variant to build if no variant has been passed on via the command-line. The default is an empty string.

docfiles
docfiles+

List of documentation files to be installed in $PREFIX/share/doc/. The key docfiles+ adds documentation files to the previously defined list. The default is an empty list.

download_dir

Directory where files are saved after downloading. See the structure urls for more details. The directory defaults to the directory specified with the environment variable PMODULES_DISTFILES or if the variable is not defined to $HOME/.cache/Pmodules/distfiles.

group

The name of the group in which the module is to be installed. The default group is Tools.

group_deps

This block is required if the module is to be installed in a hierarchical group. The schema of the structure is documented below.

kernels

List of kernels to which this configuration block applies. Valid values are Linux, Darwin and any. The default value is any.

modulefile

Name of the modulefile to be installed. The default is modulefile.

overlay

The name of the overlay in which the module is to be installed

patch_files
patch_files+

List of patches to be applied on the sources. The default is an empty list.

relstage

The release stage of the module. Valid values are:
unstable: the module is still under development
stable: the module is ready to be used
deprecated: the module is deprecated
remove: the module is deprecated and marked to be removed
removed: the module has been removed

runtime_deps

List of run-time dependencies of the module. These modules will be loaded together with the module.

sub_packages

A list of sub-package to be built. For details see below.

suffix

This string will be added to the version of the module as suffix.

systems

A list of systems to which this block applies. A 'system' is either a hostname or OS name like rhel8. For hostnames a regex like .*.merlin7.psi.ch can be used.

target_cpus

A list of CPU to which this block applies.

urls | sources

List of sources from which the required software must be downloaded if not already cached. The details are documented in a following section.

use_flags

TBW

use_overlays

List of overlays that must be loaded before the module can be built

variant

Sequence of synonyms for a variant.

Defining build-functions

Field-Name Description

Field-Name	Description
`prep`	List of functions to be called in the 'prep' step. The list defaults to `pbuild::pre_prep`, `pbuild::prep`, `pbuild::post_prep`.
`configure`	List of functions to be called in the 'configure' step. The list defaults to `pbuild::pre_configure`, `pbuild::configure`, `pbuild::post_configure`.
`compile`	List of functions to be called in the 'compile' step. The list defaults to `pbuild::pre_compile`, `pbuild::compile`, `pbuild::post_compile`.
`install`	List of functions to be called in the 'install' step. The list defaults to `pbuild::pre_install`, `pbuild::install`, `pbuild::post_install`.

prep

List of functions to be called in the 'prep' step. The list defaults to pbuild::pre_prep, pbuild::prep, pbuild::post_prep.

configure

List of functions to be called in the 'configure' step. The list defaults to pbuild::pre_configure, pbuild::configure, pbuild::post_configure.

compile

List of functions to be called in the 'compile' step. The list defaults to pbuild::pre_compile, pbuild::compile, pbuild::post_compile.

install

List of functions to be called in the 'install' step. The list defaults to pbuild::pre_install, pbuild::install, pbuild::post_install.

Example:

build_functions:
  prep: [pbuild::prep]
  configure: []
  compile: [pbuild::compile]
  install: [install-1]

Group dependencies

Field-Name Description

Field-Name	Description
`compiler`	Structure defining compilers and their versions. See example below.
`mpi`	Same for MPI implementations (e.g. openmpi, mpich).
`hdf5`	Same for parallel HDF5.
`hdf5_serial`	Same for serial HDF5.

compiler

Structure defining compilers and their versions. See example below.

mpi

Same for MPI implementations (e.g. openmpi, mpich).

hdf5

Same for parallel HDF5.

hdf5_serial

Same for serial HDF5.

Example: group dependencies for a module in the Compiler hierarchy

group_deps:
  compiler:
    gcc: [12.4.0, 14.2.0]
    intel: [2021.1]

Example: group dependencies for a module in the MPI hierarchy

group_deps:
  compiler:
    gcc: [13.4.0, 14.4.0, 15.2.0]
  mpi:
    openmpi: [5.0.10]

Defining sources

Field-Name Description

Field-Name	Description
`url`	Download link.
`name`	optional output filename. The filename defaults the last component of `url`.
`strip_dirs`	The value of this key is passed to the unpacker and specifies the number of directories to be stripped while unpacking. Stripping directories is supported only by `tar`. If another unpacker is specified, this value is ignored. The default value is `1`.
`unpacker`	The tool to be used to unpack the downloaded software. Supported unpackers are `tar`, `sevenz` and `none`. The default unpacker is `tar`.
`unpack_dir`	The directory into which the files should be extracted. The default directory is `$SRC_DIR`.
`patch_file`	A patch file which will be applied after downloading and unpacking the source package.
`patch_strip`	Number of directories to strip from the specification in the patch file. See the manual page for the tool `patch(1)` for more details.

url

Download link.

name

optional output filename. The filename defaults the last component of url.

strip_dirs

The value of this key is passed to the unpacker and specifies the number of directories to be stripped while unpacking. Stripping directories is supported only by tar. If another unpacker is specified, this value is ignored. The default value is 1.

unpacker

The tool to be used to unpack the downloaded software. Supported unpackers are tar, sevenz and none. The default unpacker is tar.

unpack_dir

The directory into which the files should be extracted. The default directory is $SRC_DIR.

patch_file

A patch file which will be applied after downloading and unpacking the source package.

patch_strip

Number of directories to strip from the specification in the patch file. See the manual page for the tool patch(1) for more details.

Default URLs can be defined by using the variables P, V, V_PKG, V_MAJOR, V_MINOR and V_PATCHLVL. Example:
https://sourceforge.net/projects/gnuplot/files/$P/$V/$P-${V_PKG}.tar.gz

Sub-packages

Example: Sub-packages (git and git-lfs)

---
# yamllint disable rule:line-length
format: 1
git:
  defaults:
    group: Tools
    overlay: base
    relstage: stable
    compile_in_sourcetree: true
    urls:
      - url: https://mirrors.edge.kernel.org/pub/software/scm/git/git-${V_PKG}.tar.xz
  shasums:
    git-2.54.0.tar.xz: f689162364c10de79ef89aa8dbf48731eb057e34edbbd20aca510ce0154681a3
  versions:
    2.54.0:
      config:
        relstage: unstable
        runtime_deps: [TclTk/8.6.16, openssl/3.4.1]
        build_requires: [gcc/10.4.0, curl/8.20.0, perl/5.40.2, asciidoctor/2.0.23, gettext/0.19.8, xmlto/0.0.28]
        docfiles: [COPYING, README.md]
        configure_args:
          - --with-tcltk
          - --with-iconv=${GETTEXT_PREFIX}
          - --with-perl=${PERL_PREFIX}/bin/perl
        sub_packages:
          - name: git-lfs
            version: 3.7.1
            build_args:

git-lfs:
  type: sub_package
  defaults:
    compile_in_sourcetree: true
  shasums:
    git-lfs-linux-amd64-v3.7.1.tar.gz: 1c0b6ee5200ca708c5cebebb18fdeb0e1c98f1af5c1a9cba205a4c0ab5a5ec08

  versions:
    3.7.1:
      variants:
        -
          kernels: [Linux]
          target_cpus: [x86_64]
          urls:
            - url: https://github.com/git-lfs/git-lfs/releases/download/v${V_PKG}/git-lfs-linux-amd64-v${V_PKG}.tar.gz
          build_functions:
            configure: []
            compile: []
        -
          kernels: [Linux]
          target_cpus: [aarch64]
          urls:
            - url: https://github.com/git-lfs/git-lfs/releases/download/v${V_PKG}/git-lfs-linux-arm64-v${V_PKG}.tar.gz
          build_functions:
            configure: []
            compile: []

4.4.1.3. Examples

Example: Gnuplot

format: 1
gnuplot:
  defaults:                                             (1)
    group: Tools                                        (2)
    overlay: base                                       (3)
    relstage: stable                                    (4)
    systems: [rhel8, rhel7, rhel6]                      (5)
    docfiles: [Copyright, NEWS, README]                 (6)
    urls:                                               (7)
      - url: https://sourceforge.net/projects/gnuplot/files/$P/$V/$P-${V_PKG}.tar.gz
  shasums:                                              (8)
    gnuplot-5.4.10.tar.gz: 975d8c1cc2c41c7cedc4e323aff035d977feb9a97f0296dd2a8a66d197a5b27c
    gnuplot-5.4.9.tar.gz: a328a021f53dc05459be6066020e9a71e8eab6255d3381e22696120d465c6a97
    gnuplot-5.4.8.tar.gz: 931279c7caad1aff7d46cb4766f1ff41c26d9be9daf0bcf0c79deeee3d91f5cf
    gnuplot-5.4.5.tar.gz: 66f679115dd30559e110498fc94d926949d4d370b4999a042e724b8e910ee478
    gnuplot-5.4.4.tar.gz: 372300b7867f5b3538b25fc5d0ac7734af6e3fe0d202b6db926e4369913f0902
    gnuplot-5.4.3.tar.gz: 51f89bbab90f96d3543f95235368d188eb1e26eda296912256abcd3535bd4d84
    gnuplot-5.4.2.tar.gz: e57c75e1318133951d32a83bcdc4aff17fed28722c4e71f2305cfc2ae1cae7ba
    gnuplot-5.4.1.tar.gz: 6b690485567eaeb938c26936e5e0681cf70c856d273cc2c45fabf64d8bc6590e
    gnuplot-5.4.0.tar.gz: eb4082f03a399fd1e9e2b380cf7a4f785e77023d8dcc7e17570c1b5570a49c47
    gnuplot-5.2.8.tar.gz: 60a6764ccf404a1668c140f11cc1f699290ab70daa1151bb58fed6139a28ac37
    gnuplot-5.2.7.tar.gz: 97fe503ff3b2e356fe2ae32203fc7fd2cf9cef1f46b60fe46dc501a228b9f4ed
    gnuplot-5.2.6.tar.gz: 35dd8f013139e31b3028fac280ee12d4b1346d9bb5c501586d1b5a04ae7a94ee
    gnuplot-5.2.4.tar.gz: 1515f000bd373aaa53b16183f274189d4f5e0ae47d22f434857933d16a4770cb
    gnuplot-5.0.0.tar.gz: 417d4bc5bc914a60409bb75cf18dd14f48b07f53c6ad3c4a4d3cd9a8d7370faf
    gnuplot-4.6.3.tar.gz: df5ffafa25fb32b3ecc0206a520f6bca8680e6dcc961efd30df34c0a1b7ea7f5
  versions:
    5.4.{0,1,2,3,4,5,8,9};5.2.{0,4,6,7,8};5.0.0;4.6.3:  (9)
    5.4.10:                                             (10)
      config:
        relstage: unstable

1	Configuration is for `gnuplot`
2	will be installed in the group `Tools`
3	will be installed in the base overlay (`/opt/psi`)
4	the default release stage is `stable`
5	the module is available on these systems
6	install these files to `$PREFIX/share/doc/gnuplot`
7	download via this link
8	SHA256 hash sums for all available versions
9	no specific configuration required for these versions
10	release stage of version `5.4.10` is still `unstable`, override default release stage. ---

Example: HDF5

---
# yamllint disable rule:line-length             (1)
format: 1
hdf5:                                           (2)
  defaults:
    group: MPI                                  (3)
    overlay: base                               (4)
    relstage: stable                            (5)
    systems: [rhel7, rhel8, rhel9]              (6)
    urls:                                       (7)
      - url: https://support.hdfgroup.org/ftp/HDF5/releases/$P-${V_MAJOR}.${V_MINOR}/$P-${V_PKG}/src/$P-${V_PKG}.tar.bz2
  shasums:                                      (8)
    hdf5-1.8.10-patch1.tar.bz2: 292afb3615ad9e68f4d5d18ebb11e4a73f2aece39f2da3875a457ff1e109fc41
    hdf5-1.8.12.tar.bz2: 10a369a4fc207bb09245f57c758e587420e06dfc0e445e337a58b0848b75a949
    ...
    hdf5-1.13.1.tar.bz2: e16973ec893e2d5aa9c8dc73e196db9b99a605578e7317b421c713936f8bf57d

  versions:
    1.8.12:
      config:
        relstage: deprecated                    (9)
      variants:
        -
          group_deps:
            compiler: {gcc: [4.7.4, 4.8.3, 4.8.4, 4.9.2]}
            mpi: {openmpi: [1.6.5, 1.8.2, 1.8.4]}
        -
          group_deps:
            compiler: {gcc: [4.8.2]}
            mpi: {openmpi: [1.6.5]}
        ...
        -
          group_deps:
            compiler: {gcc: [5.1.0], intel: [15.2, 15.3]}
            mpi: {openmpi: [1.8.4]}
    ...
    1.10.8_slurm:
      variants:                                (10)
        -
          group_deps:
            compiler: {gcc: [10.4.0]}
            mpi: {openmpi: [4.1.4_slurm]}
        -
          relstage: unstable
          group_deps:
            compiler: {gcc: [9.5.0, 10.4.0, 11.4.0, 12.3.0, 13.1.0]}
            mpi: {openmpi: [4.1.5_slurm]}
    ...
    1.12.0:
      variants:
        -
          group_deps:
            compiler: {gcc: [7.5.0, 8.4.0, 9.3.0, 10.2.0]}
            mpi: {openmpi: [4.0.5]}
        -
          relstage: unstable
          group_deps:
            compiler: {pgi: [21.5]}
            mpi: {pgi-mpi: [21.5]}

    1.13.1:
      variants:
        -
          suffix: _slurm                        (11)
          variant: [_slurm]
          relstage: unstable
          group_deps:
            compiler: {gcc: [11.2.0]}
            mpi: {openmpi: [4.1.3_slurm]}

1	disable rule to check line length in yamllint. Default is 80.
2	this configuration is for HDF5
3	default group is MPI.
4	default overlay is `base`
5	default release stage is `stable`
6	this build-block can be used on RHEL7, RHEL8 and RHEL9
7	download via this link
8	SHA256 hash sums for all available versions
9	all variants of version 1.8.12 are `deprecated`
10	hdf5/1.10.8_slurm has two variants. The first variant compiled with GCC 10.4.0 and openmpi/4.1.4_slurm. The second variant is unstable and compiled with with GCC 9.5.0, 10.4.0, 11.4.0, 12.3.0, 13.1.0 and openmpi/4.1.5_slurm.
11	TBW

4.5. Writing build scripts

Pmodules has it’s own 'interpreter' to run build-scripts written in Bash. The name of the interpreter is modbuild. Thus the first line (the so called 'shebang') of a build-script must be

#!/usr/bin/env modbuild

As already mentioned the four steps 'prepare', 'configure', 'build' and 'install' has to be performed to build a Pmodule. These steps maps to the functions:

prepare: pbuild::pre_prep()
pbuild::prep()
pbuild::post_prep()
configure: pbuild::pre_configure()
pbuild::configure()
pbuild::post_configure()
compile: pbuild::pre_compile()
pbuild::compile()
pbuild::post_compile()
install: pbuild::pre_install()
pbuild::install()
pbuild::post_install()

In many cases you don’t have to implement the the functions pbuild::prep(), pbuild::configure(), pbuild::compile() and pbuild::install() or at least not all of them. The build-system provides default implementations for these functions. In the simplest case, the build script consists only of the shebang line (only for Pmodules >= 1.1). In this case all further information for building the module is in the configuration file. For software that uses autotools or CMake for configuration and creation of Makefiles, the standard functions can practically always be used. The functions pbuild::pre_STEP and pbuild::post_STEP can be used to hook into each step before and after the default function has been called. For example pbuild::pre_configure() can be used to set arguments for autotools or CMake as in the build-script for Gnuplot:

4.5.1. Overloading the default implementations

If the build-system of the software package isn’t autotools or CMake you have to overload the default implementation of pbuild::configure.

pbuild::configure(){
        # add code to configure the software package
}

In case where no configuration is required, overload the default function with

pbuild::configure(){
        : # do nothing
}

You can do the same with the other default implementation.

4.5.2. Modules from binary packages

If you have a binary package like Matlab, Mathematica, ANSYS, it might not be possible to script the installation. In this case you should use a dummy build-script like below and document the installation in a README.md:

#!/usr/bin/env modbuild

pbuild::prep() { :; }
pbuild::configure() { :; }
pbuild::compile() { :; }
pbuild::install() { :; }

If the installation can be scripted, code this in the function pbuild::install.

#!/usr/bin/env modbuild

pbuild::prep() { :; }
pbuild::configure() { :; }
pbuild::compile() { :; }
pbuild::install() {
        # add code to call the installer here
}

4.5.3. Functions to compare versions

4.5.3.1. Compare two versions: `pbuild::version_compare`

Synopsis

pbuild::version_compare VERSION1 VERSION2
pbuild::version_lt VERSION1 VERSION2
pbuild::version_le VERSION1 VERSION2
pbuild::version_gt VERSION1 VERSION2
pbuild::version_ge VERSION1 VERSION2
pbuild::version_eq VERSION1 VERSION2

Description

The function splits the version strings at the dots an compares each component. If a component is a number the comparison is numerical otherwise lexical.

pbuild::version_compare VERSION1 VERSION2 returns
0 if the versions are equal
1 if VERSION1 is higher then VERSION2
2 if VERSION1 is lower then VERSION2

pbuild::version_lt VERSION1 VERSION2 returns
0 if VERSION1 is lower then VERSION2 otherwise a values greater than zero.

pbuild::version_le VERSION1 VERSION2 returns
0 if VERSION1 is lower then or equal to VERSION2 otherwise a values greater than zero.

pbuild::version_gt VERSION1 VERSION2 returns
0 if VERSION1 is higher then VERSION2 otherwise a values greater than zero.

pbuild::version_ge VERSION1 VERSION2 returns
0 if VERSION1 is higher then or equal to VERSION2 otherwise a values greater than zero.

pbuild::version_eq VERSION1 VERSION2 returns
0 if VERSION1 is equal to VERSION2 otherwise a values greater than zero.

Example

pbuild::version_lt '1.1.0' '1.2.2'  # result is 0
pbuild::version_ge '9.1.2' '10.0.0' # result is greater zero

4.5.4. Build functions

4.5.4.1. Download and unpack: `pbuild::prep`

Synopsis

pbuild::pre_prep
pbuild::prep
pbuild::post_prep

Description

Functions to prepare the sources. This includes

downloading required files a verifying the checksums
unpacking
applying patches

Before the prep-functions are called, the build-system changes to the source directory (${SRC_DIR}).

In the most uses cases the default function pbuild::prep provided by the build-system can be used and nothing must be implemented in the build-script. In some rare cases pre- or post-hooks are required to make the default function feasible. One use case with autotools for a post-hook is to create the configure scripts, if only configure.ac is shipped with the software.

The default hooks provided by the build system are only stubs and do nothing.

If the default function cannot be used, it must be implemented in the build-script.

Example

The source distribution of the IOAPI library contains object files! These files must be removed after unpacking:

pbuild::post_prep() {
        find "${SRC_DIR}" -name "*.mod" -exec rm {} \;
        find "${SRC_DIR}" -name "*.o" -exec rm {} \;
}

4.5.4.2. Configure software: `pbuild::configure`

Synopsis

pbuild::pre_configure
pbuild::configure
pbuild::post_configure

Description

Configure the software for compilation.

Before these functions are called, the build-system changes to the build directory (${BUILD_DIR}).

In the most uses cases the default function pbuild::configure provided by the build-system can be used and nothing must be implemented in the build-script.

The default function first searches whether the software can be configured with autotools. If yes, configuration with autotools will be performed. Otherwise the existence of a CMake script will be checked. If found, configuration will be done via CMake. If scripts for both configuration tools exist, the to be used tool can be selected with pbuild::use_autotools or pbuild::use_cmake. Arguments to configure or cmake can be set with pbuild::add_configure_args.

A common use case for the pre-configure hooks is to define arguments passed to autotools or CMake by the default function.

If the default function cannot be used, the pbuild::configure function must be implemented in the build-script.

Examples

Excerpt from the parallel HDF5 build-script

pbuild::pre_configure() {
        pbuild::add_configure_args "CC=${MPICC}"
        pbuild::add_configure_args "CXX=${MPICXX}"

        pbuild::add_configure_args "--enable-shared"
        pbuild::add_configure_args "--enable-parallel"
        pbuild::add_configure_args "--enable-cxx"
        pbuild::add_configure_args "--enable-unsupported"
        #pbuild::add_configure_args "--enable-threadsafe"
        pbuild::add_configure_args "--with-pic"

        local enable_fortran='yes'

        case "${COMPILER}" in
                clang-macos )
                        enable_fortran='no'
                        # we do not have Fortran in Xcode
                        ;;
                pgi )
                        # PGI uses GCC's include files, some object files and
                        # the STL implementation!
                        # The PGI C pre-processor is broken and doesn't work
                        # for HDF5. We use the pre-processor of the underlying
                        # GCC...
                        # This is a bit hackish!
                        #
                        # The following eval sets GCCDIR! Which is something
                        # like:
                        # /opt/psi/Programming/gcc/7.3.0/bin/../lib/gcc/x86_64-pc-linux-gnu/7.3.0
                        #
                        eval $(pgcc -show 2>/dev/null | \
                                awk '/^GCCDIR[[:space:]]*=/{gsub(/[[:space:]]/,""); print $0}')
                        pbuild::add_configure_args "CPP=${GCCDIR%%/..*}/cpp"
                        pbuild::add_configure_args "CFLAGS=-fPIC"
                        pbuild::add_configure_args "CXXFLAGS=-fPIC"
                        pbuild::add_configure_args "FCFLAGS=-fPIC"
                        ;;
        esac

        if [[ "${enable_fortran}" ===== 'yes' ]]; then
                pbuild::add_configure_args "F77=${MPIF77}"
                pbuild::add_configure_args "F90=${MPIF90}"
                pbuild::add_configure_args "FC=${MPIFC}"
                pbuild::add_configure_args "FORTRAN=${MPIFORTRAN}"
                pbuild::add_configure_args "--enable-fortran"
        fi

Simplified excerpt from the OpenBLAS build-script

pbuild::configure() {
        case ${COMPILER} in
        gcc )
                CC='gcc'
                ;;
        intel )
                CC='icc'
                ;;
        clang-macos )
                CC='gcc'
                ;;
        * )
                die 3 "Oops: unknown compiler: ${COMPILER}"
                ;;
        esac
        cat <<EOF > "${SRC_DIR}/make.inc"
SHELL = /bin/sh
PLAT =
DRVOPTS  = \$(NOOPT)
ARCHFLAGS= -ru
EOF
        echo "USE_SIMPLE_THREADED_LEVEL3 = 1" >> "${SRC_DIR}/Makefile.rule"
        echo "NO_AVX = 1" >> "${SRC_DIR}/Makefile.rule"
        echo "NO_AVX2 = 1" >> "${SRC_DIR}/Makefile.rule"
        if pbuild::use_flag "omp"; then
                echo "USE_THREAD = 1" >> "${SRC_DIR}/Makefile.rule"
        else
                echo "USE_THREAD = 0" >> "${SRC_DIR}/Makefile.rule"
        fi
}

4.5.4.3. Compile software: `pbuild::compile`

Synopsis

pbuild::pre_compile
pbuild::compile
pbuild::post_compile

Description

Compile the software.

Before these functions are called, the build-system changes to the build directory (${BUILD_DIR}).

In the most uses cases the default function pbuild::compile provided by the build-system can be used and nothing must be implemented in the build-script. The pre- and post-hooks might be useful in cases where

to run other make targets than all
multiple packages must be compiled into one module
to compile a dependency required by the main package.

If the default function cannot be used, the pbuild::compile function must be implemented in the build-script.

Example

Excerpt from perl build-script

pbuild::post_compile() {
        make test
}

4.5.4.4. Install software: `pbuild::install`

Synopsis

pbuild::pre_install
pbuild::install
pbuild::post_install

Description

Compile the software.

Before these functions are called, the build-system changes to the build directory (${BUILD_DIR}).

In the many uses cases the default function pbuild::install provided by the build-system can be used and nothing must be implemented in the build-script. A typical use case for the post-install hook is to install required libraries from other modules or the system to reduce or eliminate run-time dependencies

If the default function cannot be used, the pbuild::install function must be implemented in the build-script.

Example

TBW

Pmodules - an improved environment modules system

4. BUILDING MODULES

4.1. Introduction

4.2. Predefined variables

4.3. Building a module

4.4. Writing build configuration files

4.4.1. files/config.yaml

4.4.1.1. Structure

4.4.1.2. Configuration blocks

Defining build-functions

Group dependencies

Defining sources

Sub-packages

4.4.1.3. Examples

4.5. Writing build scripts

4.5.1. Overloading the default implementations

4.5.2. Modules from binary packages

4.5.3. Functions to compare versions

4.5.3.1. Compare two versions: pbuild::version_compare

Synopsis

Description

Example

4.5.4. Build functions

4.5.4.1. Download and unpack: pbuild::prep

Synopsis

Description

Example

4.5.4.2. Configure software: pbuild::configure

Synopsis

Description

Examples

4.5.4.3. Compile software: pbuild::compile

Synopsis

Description

Example

4.5.4.4. Install software: pbuild::install

Synopsis

Description

Example

4.4.1. `files/config.yaml`

4.5.3.1. Compare two versions: `pbuild::version_compare`

4.5.4.1. Download and unpack: `pbuild::prep`

4.5.4.2. Configure software: `pbuild::configure`

4.5.4.3. Compile software: `pbuild::compile`

4.5.4.4. Install software: `pbuild::install`