Multibuild for Developers

Overview

This document describes the current design of multibuild in Exherbo, focussing mainly on what exheres authors need to know. By “multibuild”, we mean the ability for a package to build and install multiple versions of itself in different configurations. This is currently used to build both 32- and 64-bit versions of libraries on 64-bit platforms. Some or all of the design may be reused in future to build Perl/Python/etc modules for use with multiple versions of the respective interpreters, for example.

Note that the multibuild design is not yet finalised and may change in future.

Example library exheres

The following example is libxslt-1.1.26-r1.exheres-0 at the time of writing, with some of the detail trimmed out. It will be used to illustrate various aspects of the design.

# Copyright 2008 Santiago M. Mola
# Distributed under the terms of the GNU General Public License v2
# Based in part upon 'libxslt-1.1.23.ebuild', which is:
#   Copyright 1999-2008 Gentoo Foundation

require autotools [ ... ] easy-multibuild python

# SUMMARY/HOMEPAGE/etc ...

MYOPTIONS="crypt debug examples python
    multibuild_c: 32 64"

DEPENDENCIES="
    build+run:
        dev-libs/libxml2[>=2.6.27][multibuild_c:*(-)?]
        crypt?  ( dev-libs/libgcrypt[>=1.1.92][multibuild_c:*(-)?] )
        python? ( dev-lang/python:=[multibuild_c:*(-)?] )
"

DEFAULT_SRC_PREPARE_PATCHES=( ... )

src_prepare() {
    default

    ...

    # Be warned: eautoreconf breaks tests
    eautomake
}

configure_one_multibuild() {
    local python_prefix='/usr'
    multibuild_default_target C || python_prefix="/usr/${CHOST}"
    econf ... \
        $(option_with python python "${python_prefix}")
}

install_one_multibuild() {
    default
    keepdir /usr/${LIBDIR}/${PN}-plugins
    if ! multibuild_default_target C; then
        dodir "/usr/${CHOST}/bin"
        edo mv "${IMAGE}"/usr/bin/"${PN#lib}"-config "${IMAGE}"/usr/"${CHOST}"/bin
    fi
}

src_install() {
    easy-multibuild_src_install

    if ! option examples; then
        edo rm -rf "${IMAGE}/usr/share/doc/${PN}-python-${PV}/examples"
    fi
}

Classes and targets

As described above, the multibuild infrastructure could be used to handle multiple axes of variation such as 32-/64-bit, Python versions, etc, although only the former is currently in use. Each of these axes is known as a “class”, and the possible configurations for each class are known as “targets”. These are defined in the profiles; currently there is a C class defined with available targets 32 and 64. These can be seen in the example exheres above in the multibuild_c suboption prefix, the declared options for this suboption and the argument given to the multibuild_default_target function.

Multibuild options

Users specify which configuration(s) they wish to build using the standard exheres options mechanism. Each multibuild class has its own suboption prefix, named as multibuild_ followed by the class name in lower case, and the options are simply the name of the targets. Currently the exheres needs to declare the options itself, as shown in the example.

If one library links to another, then naturally the lower-level library needs to be built with at least all the variations used for the higher-level library. This is expressed in the dependencies using the [multibuild_c:*(-)?] syntax, described in Exheres for Smarties. The multibuild-specific elements are the * and the (-). The * is a shortcut meaning “all options under the specified suboption that are declared in this exheres”. In the example, since libxslt itself declares multibuild_c:32 and multibuild_c:64, the dependency is equivalent to [multibuild_c:32(-)?][multibuild_c:64(-)?]. The (-) means “do not consider the package matching the dependency if it does not declare at least one option under the specified suboption”, so in the example, if any version of dev-libs/libxml2[>=2.6.27] didn’t have any multibuild_c options at all then it would not satisfy the dependency. Finally, the ? is used to mean that all targets enabled for this exheres must be enabled for the dependency, but targets disabled for this exheres can be either enabled or disabled.

easy-multibuild.exlib

As far as the exheres format is concerned, the package itself is responsible for building its code for each desired configuration – there is no magic applied by the package manager to do this. However, since most packages follow the same pattern in this respect, easy-multibuild.exlib is available to assist. It takes an array exparam named classes to indicate which multibuild classes the package supports; this defaults to [ C ], and since there currently aren’t any other classes defined, it will generally not be specified in practice.

easy-multibuild.exlib exports the src_configure, src_compile, src_test and src_install phase functions (and also src_unpack and src_prepare if the multiunpack exparam is specified as true – see below). Each of the exported phase functions calls the function ${EXHERES_PHASE}_one_multibuild if it is defined, or else default, once for each enabled target. (Note that ${EXHERES_PHASE} doesn’t include the src_ prefix, so the function names are configure_one_multibuild etc as in the example.) The default target for the class is run last – this is mainly important for the install phase, so that files not installed in a target-specific directory will be overwritten by the ones for the default. (This mainly applies to binaries – most other file types will either be the same for all targets anyway or should be installed in a target-specific location.) The example shows how to add custom code for each iteration of the configure and install phases, and how to add code for install that will be called just once (this is just the normal way of extending phases exported from an exlib).

Each call is made with the current directory set to a target-specific directory, namely ${WORKBASE}/${MULTIBUILD_CLASS}/${MULTIBUILD_TARGET}/ (with the latter two variables having the obvious values), so each target’s object code is kept separate. This directory will by default be created before running configure_one_multibuild. (If you specify the work exparam, this will be added to the end of the specified path when setting the current directory but not when creating it – see below for a use for this.) For this to work the build system will need to support having the source code in a separate directory from the object code. For configure_one_multibuild, easy-multibuild.exlib sets the ECONF_SOURCE variable to the values of ${WORK}, so exhereses for (correctly-written) autotools-based packages will generally not have to do anything special to support this.

If the package’s build system can’t easily be made to support separating the source code and the object code, you can specify the multiunpack exparam. If it is set to true, the exlib will additionally export the src_unpack and src_prepare phases, with the same multiple call behaviour as the usual ones. Furthermore, the work exparam defaults to ${PNV} in this case. Using multiunpack will result in a complete copy of the work-tree to be unpacked and prepared for each multibuild target. Therefore, it should only be used as a last resort, if the package does not support out-of-tree builds and cannot be easily patched to do so.

Each call is also made with various environment variables set which indicate how to build the software for the active configuration, defined by the profile. In this case of the C class, these are standard variables such as ${CHOST}, ${CFLAGS}, etc, so for many packages using a conventional build system the exheres won’t need to do anything special to configure the compiler to build the code for the correct target.

There can also be functions named ${EXHERES_PHASE}_prepare_one_multibuild – if these are defined they will be called after the environment variables have been set but before entering the target-specific directory. This mechanism is used to create the target-specific directory and to set ${ECONF_SOURCE} for the configure phase, so if you override configure_prepare_one_multibuild and still need this functionality you will need to replicate it in your function.

File locations

If multiple versions of a package’s code are being installed then they obviously need to be placed in different locations. For the C class, the files that vary between targets are mostly libraries – traditionally they are stored in /lib or /usr/lib, but with multibuild, the libraries for some targets may be placed in lib32 or lib64 instead. This is controlled by the profile, via the target-specific variable mechanism described above – the variable ${LIBDIR} is defined as lib, lib32, lib64 or whatever value is appropriate. (It should also be defined directly by the profile for the default target for use in non-multibuild packages.) The built-in econf function is aware of this variable, so exhereses for simple autotools-based packages may not need to handle this specially. Exlibs that provide support for other build systems ought to have similar support built in as well. The install_one_multibuild function in the example exheres shows how it might be used directly.

Some packages need to install other files, such as *-config programs or header files, in per-target locations. (Most programs, even compiled ones, only need to be installed for the default ABI, but *-configs need to be per-target so dependent packages can get the appropriate flags for the target that they are building. Similarly, most headers are target-independent, but a few have target-specific information built in.) In this case, the current convention is to place the ones for the default target in the normal location, and the ones for the other targets in /usr/${CHOST}/bin, /usr/${CHOST}/include, etc. (/usr/${CHOST}/bin is prepended to ${PATH} at the same time as the other target-specific variables are set, so the *-config scripts installed there will be used automatically as appropriate.)

The example exheres shows how to install the xslt-config script correctly, and also how to make sure the appropriate version of the Python headers is used. For packages that use pkg-config to find header paths and the like, the ${PKG_CONFIG_PATH} variable is automatically set to the appropriate value so no special action needs to be taken in this regard.

multibuild.exlib

multibuild.exlib is used to manage to target-specific variables defined in the profiles. In most cases it will be used via easy-multibuild.exlib (possibly with the multibuild_default_target function used explicitly as in the example), but packages can use it directly if they have requirements too complex for the latter.

(TODO: document the details here, I’m not too familiar with it and most readers won’t need it anyway.)

Profiles

(TODO: document.)


Copyright 2011 David Leverton