FCM System User Guide: The Build System
Author: FCM development team
Keywords: FCM, user guide, extract, build

# The Build System

The build system analyses the directory tree containing a set of source code, processes the configuration, and invokes make to compile/build the source code into the project executables. In this chapter, we shall use many examples to explain how to use the build system. At the end of this chapter, you should be able to use the build system, either by defining the build configuration file directly or by using the extract system to generate a suitable build configuration file.

## The Build Command

To invoke the build system, simply issue the command:

    fcm build

By default, the build system searches for a build configuration file "bld.cfg" in "$PWD" and then "$PWD/cfg". If a build configuration file is not found in these directories, the command fails with an error. If a build configuration file is found, the system will use the configuration specified in the file to perform the build. If you use the extract system to extract your source tree, a build configuration should be written for you automatically at the "cfg/" sub-directory of the destination root directory.

If the root directory of the build does not exist, the system performs a new full build at this directory. If a previous build already exists at this directory, the system performs an incremental build. If a full (fresh) build is required for whatever reason, you can invoke the build system using the "-f" option, (i.e. the command becomes "fcm build -f").

The build system uses GNU make to perform the majority of the build. GNU make has a "-j <jobs>" option to specify the number of <jobs> to run simultaneously. Invoking the build system with the same option triggers this option when the build system invokes the make command. The argument to the option <jobs> must be an integer. The default is 1. For example, the command "fcm build -j 4" will allow make to perform 4 jobs simultaneously.

For further information on the build command, please see FCM Command Reference > fcm build.

## Basic Features

The build configuration file is the user interface of the build system. It is a line based text file. You can create your own build configuration file or you can use the extract system to create one for you. For a complete set of build configuration file declarations, please refer to the Annex: Declarations in FCM build configuration file.

### Basic build configuration

Suppose we have a directory at "$HOME/example". Its sub-directory at "$HOME/example/src" contains a source tree to be built. You may want to have a build configuration file "$HOME/example/cfg/bld.cfg", which may contain:

Build configuration example 1 - basic build configuration
    cfg::type     bld                           # line 1
    cfg::version  1.0                           # line 2

    dir::root     $HOME/example                 # line 4

    target        foo.exe bar.exe               # line 6

    tool::fc      ifc                           # line 8
    tool::fflags  -O3                           # line 9
    tool::cc      gcc                           # line 10
    tool::cflags  -O3                           # line 11
    tool::ld      ifc                           # line 12
    tool::ldflags -O3 -L$(HOME)/lib -legg -lham # line 13

Here is an explanation of what each line does:

- line 1: the label CFG::TYPE declares the type of the configuration file. The value "bld" tells the system that it is a build configuration file.
- line 2: the label CFG::VERSION declares the version of the build configuration file. The current default is "1.0". Although it is not currently used, if we have to change the format of the configuration file at a later stage, we shall be able to use this number to determine whether we are reading a file with an older format or one with a newer format.
- line 4: the label DIR::ROOT declares the root directory of the current build.
- line 6: the label TARGET declares a list of "default" targets. The default targets of the current build will be "foo.exe" and "bar.exe".
- line 8: the label TOOL::FC declares the Fortran compiler command.
- line 9: the label TOOL::FFLAGS declares the options to be used when invoking the Fortran compiler command.
- line 10: the label TOOL::CC declares the C compiler command.
- line 11: the label TOOL::CFLAGS declares the options to be used when invoking the C compiler command.
- line 12: the label TOOL::LD declares the linker command.
- line 13: the label TOOL::LDFLAGS declares the options to be used when invoking the linker command.

When we invoke the build system, it reads the above configuration file. It will go through various internal processes, such as dependency generations, to obtain the required information to prepare the Makefile of the build. (All of which will be described in later sections.) The Makefile of the build will be placed at "$HOME/example/bld". The system will then invoke make to build the targets specified in line 6, i.e. "foo.exe" and "bar.exe" using the build tools specified between line 8 to line 13. On a successful build, the target executables will be sent to "$HOME/example/bin/". The build system also creates a shell script called "fcm_env.ksh" in "$HOME/example/". If you source the shell script, it will export your PATH environment variable to search the "$HOME/example/bin/" directory for executables.

N.B. You may have noticed that the "-c" (compile to object file only) option is missing from the compiler flags declarations. This is because the option is inserted automatically by the build system, unless it is already declared.

Note - declaration of source directories for build
Source directories do not have to reside in the source sub-directory of the build root directory. They can be anywhere, but you will have to declare them individually, using the label SRC::<pcks>, where <pcks> is the sub-package name in which the source directory belongs. E.g.
    # Declare a source directory in the sub-package "foo::bar"
    src::foo::bar  $HOME/foo/bar
By default, the build system searches the "src/" sub-directory of the build root directory for sub-package source directories. If all source directories are already declared explicitly, you can switch off the automatic directory search by setting the SEARCH_SRC flag to 0. E.g.
    search_src  0
As mentioned in the previous chapter, the name of a sub-package <pcks> provides a unique namespace for a file container. The name of a sub-package is a list of words delimited by the double colons "::", which is turned into the double underscores "__" by the build system internally. Please avoid using "::" and "__" for naming your files and directories. In the build system, the sub-package name also provides an "inheritance" relationship for sub-packages. For instance, we may have a sub-package called "foo::bar::egg", which belongs to the sub-package "foo::bar", which belongs to the package "foo". If we declare a global build tool, it applies to all packages. If we declare a build tool for "foo", it applies also to the sub-package "foo::bar" and "foo::bar::egg". If we declare a build tool for "foo::bar", it applies also to "foo::bar::egg", but not to other sub-packages in "foo".

### Build configuration via the extract system

As mentioned earlier, you can obtain a build configuration file through the extract system. The following example is what you may have in your extract configuration in order to obtain a similar configuration as example 1:

Build configuration example 2 - created by the extract system
    cfg::type          ext                           # line 1
    cfg::version       1.0                           # line 2

    dest::rootdir      $HOME/example                 # line 4

    bld::target        foo.exe bar.exe               # line 6

    bld::tool::fc      ifc                           # line 8
    bld::tool::fflags  -O3                           # line 9
    bld::tool::cc      gcc                           # line 10
    bld::tool::cflags  -O3                           # line 11
    bld::tool::ld      ifc                           # line 12
    bld::tool::ldflags -O3 -L$(HOME)/lib -legg -lham # line 13

    # ... and other declarations for repositories and source directories ...

It is easy to note the similarities and differences between example 1 and example 2. Example 2 is an extract configuration file. It extracts to a destination root directory that will become the root directory of the build. Line 6 to line 13 are the same declarations, except that they are now prefixed with "BLD::". In an extract configuration file, any lines prefixed with "BLD::" means that they are build configuration setting. These lines are "ignored" by the extract system but are parsed down to the output build configuration file, with the "BLD::" prefix removed. (Note: the "BLD::" prefix is optional for declarations in a build configuration file.)

N.B. If you use the extract system to mirror an extraction to a remote machine, the extract system will assume that the root directory of the remote destination is the root directory of the build, and that the build will be carried out in the remote machine.

### Naming of executables

If a source file called "foo.f90" contains a main program, the default behaviour of the system is to name its executable "foo.exe". The root name of the executable is the same as the original file name, but its file extension is replaced with ".exe". The output extension can be altered by re-registering the extension for output EXE files. How this can be done will be discussed later in the sub-section File Type.

If you need to alter the full name of the executable, you can use the "EXE_NAME::" declaration. For example, the declaration:

    bld::exe_name::foo  bar

will rename the executable of "foo.f90" from "foo.exe" to "bar".

Note: the declaration label is "bld::exe_name::foo" (not "bld::exe_name::foo.exe") and the executable will be named "bar" (not "bar.exe").

Another way to alter the full name of the executable is to use the package configuration file, which will be discussed later in the sub-section Using a package configuration file.

### Setting the compiler flags

As discussed in the first example, the compiler commands and their flags can be set via the "TOOL::" declarations. A simple "TOOL::FFLAGS" declaration, for example, alters the compiler options for compiling all Fortran source files in the build. If you need to alter the compiler options only for the source files in a particular sub-package, it is possible to do so by adding the sub-package name to the declaration label. For example, the declaration label "TOOL::FFLAGS::ops::code::OpsMod_Control" will ensure that the declaration only applies to the code in the sub-package "ops::code::OpsMod_Control". You can even make declarations down to the individual source file level. For example, the declaration label "TOOL::FFLAGS::ops::code::OpsMod_Control::Ops_Switch" will ensure that the declaration applies only for the file "Ops_Switch.f90".

N.B. Although the prefix "TOOL::" and the tool names are case-insensitive, sub-package names are case sensitive in the declarations. Internally, tool names are turned into uppercase, and the sub-package delimiters are changed from the double colons "::" to the double underscores "__". When the system generates the Makefile for the build, each "TOOL" declaration will be exported as an environment variable. For example, the declaration "tool::fflags::ops::code::OpsMod_Control" will be exported as "FFLAGS__ops__code__OpsMod_Control".

N.B. TOOL declarations for sub-packages are only accepted by the system when it is sensible to do so. For example, it allows you to declare different compiler flags, linker commands and linker flags for different sub-packages, but it does not accept different compilers for different sub-packages.

The following is an example setting in an extract configuration file based on example 2:

Build configuration example 3 - compiler flags for different sub-packages
    cfg::type              ext
    cfg::version           1.0

    dest::rootdir          $HOME/example

    bld::target            foo.exe bar.exe

    bld::tool::fc          ifc
    bld::tool::fflags      -O3    # line 9
    bld::tool::cc          gcc
    bld::tool::cflags      -O3
    bld::tool::ld          ifc
    bld::tool::ldflags     -L$(HOME)/lib -legg -lham

    bld::tool::fflags::ops -O1 -C # line 15
    bld::tool::fflags::gen -O2    # line 16

    # ... and other declarations for repositories and source directories ...

In the example above, line 15 alters the Fortran compiler flags for "ops", so that all source files in "ops" will be compiled with optimisation level 1 and will have runtime error checking switched on. Line 16, alters the Fortran compiler flags for "gen", so that all source files in "gen" will be compiled with optimisation level 2. All other Fortran source files will use the global setting declared at line 9, so they they will all be compiled with optimisation level 3.

Note - changing compiler flags in incremental builds
Suppose you have performed a successful build using the configuration in example 3, and you have decided to change some of the compiler flags, you can do so by altering the appropriate flags in the build configuration file. When you trigger an incremental build, the system will detect changes in compiler flags automatically, and update only the required targets. The following hierarchy is followed: If the compiler flags for a particular source file change, only that source file and any targets depending on that source file are re-built. If the compiler flags for a particular sub-package change, only source files within that sub-package and any targets depending on those source files are re-built. If the global compiler flags change, all source files are re-built. If the compiler command changes, all source files are re-built.

N.B. For a full list of build tools declarations, please see Annex: Declarations in FCM build configuration file > list of tools.

### Automatic Fortran 9X interface block

For each Fortran 9X source file containing standalone subroutines and/or functions, the system generates an interface file and sends it to the "inc/" sub-directory of the build root. An interface file contains the interface blocks for the subroutines and functions in the original source file. In an incremental build, if you have modified a Fortran 9X source file, its interface file will only be re-generated if the content of the interface has changed.

Consider a source file "foo.f90" containing a subroutine called "foo". In a normal operation, the system writes the interface file to "foo.interface" in the "inc/" sub-directory of the build root. By default, the root name of the interface file is the same as that of the source file, and is case sensitive. You can change this behaviour using a "TOOL::INTERFACE" declaration. E.g.:

    bld::tool::interface  program # The default is "file"

In such case, the root name of the interface file will be named in lower case after the first program unit in the file.

The default extension for an interface file is ".interface". This can be modified through the input and output file type register, which will be discussed in a later section on File Type.

In most cases, we modify procedures without altering their calling interfaces. Consider another source file "bar.f90" containing a subroutine "bar". If "bar" calls "foo", it is good practice for "bar" to have an explicit interface for "foo". This can be achieved if the subroutine "bar" has the following within its declaration section:

    INCLUDE 'foo.interface'

The source file "bar.f90" is now dependent on the interface file "foo.interface". This can make incremental build very efficient, as changes in the "foo.f90" file will not normally trigger the re-compilation of "bar.f90", provided that the interface of the subroutine "foo" remains unchanged. (However, the system is clever enough to know that it needs to re-link any executables that are dependent on the object file for the subroutine "bar".)

The default interface block generator is a piece of "plugged-in" Perl code originally developed by the ECMWF. It has been modified at the Met Office to work with FCM. Currently, the system can also work with the interface generator f90aib, which is a freeware obtained from Fortran 90 texts and programs, assembled by Michel Olagnon at the French Research Institute for Exploitation of the Sea. To do so, you need to make a declaration in the build configuration file using the label "TOOL::GENINTERFACE". As for any other TOOL declarations, you can attach a sub-package name to the label. The change will then apply only to source files within that sub-package. If "TOOL::GENINTERFACE" is declared to have the value "NONE", interface generation will be switched off. The following are some examples:

Build configuration example 4 - Fortran 9X interface block generator
    # This is an EXTRACT configuration file ...

    # ... some other declarations ...

    bld::tool::geninterface       f90aib # line 5
    bld::tool::geninterface::foo  ECMWF  # line 6
    bld::tool::geninterface::bar  none   # line 7

    # ... some other declarations ...

In line 5, the global interface generator is now set to f90aib. In line 6, the interface generator for the package "foo" is set to the ECMWF one. In line 7, by setting the interface generator for the package "bar" to the "none" keyword, no interface file will be generated for source files under the package "bar".

Switching off the interface block generator can be useful in many circumstances. For example, if the interface block is already provided manually within the source tree, or if the interface block is never used by other program units, it is worth switching off the interface generator for the source file to speed up the build process.

### Automatic dependency

The build system has a built-in dependency scanner, which works out the dependency relationship between source files, so that they can be built in the correct order. The system scans all source files of known types for all supported dependency patterns. Dependencies of source files in a sub-package are written in a cache, which can be retrieved for incremental builds. (In an incremental build, only changed source files need to be re-scanned for dependency information. Dependency information for other files are retrieved from the cache.) The dependency information is parsed to the make rule generator, which writes the Makefile fragment for building the source files in the sub-package. In an incremental build, a Makefile fragment for a sub-package is only re-generated if a source file in the sub-package has changed.

The make rule generator generates different make rules for different dependency types. The following dependency patterns are automatically detected by the current system:

- The [USE <module>] statement in a Fortran source file is the first pattern. The statement has two implications: 1) The current file compiles only if the module has been successfully compiled, and needs to be re-compiled if the module has changed. 2) The executable depending on the current file can only resolve all its externals by linking with the object file of the compiled module. The executable needs to be re-linked if the module and its dependencies has changed.
- The [INCLUDE '<name>.interface'] statement in a Fortran source file is the second pattern. (The default extension for an interface file is ".interface". This can be modified through the input and output file type register, which will be discussed in a later section on File Type.) It has two implications: 1) The current file compiles only if the included interface file is in the INCLUDE search path, and needs to be re-compiled if the interface file changes. 2) The executable depending on the current file can only resolve all its externals by linking with the object file of the source file that generates the interface file. The executable needs to be re-linked if the source file (and its dependencies) associated with the interface file has changed. It is worth noting that for this dependency to work, the root <name> of the interface file should match with that of the source file associated with the interface file. (Please note that you can use pre-processor [#include "<name>.interface] instead of Fortran INCLUDE, but it will not work if you switch on the pre-processing stage, which will be discussed in a later section.)
- The [INCLUDE '<file>'] statement (excluding the INCLUDE interface file statement) in a Fortran source file is the third pattern. It has two implications: 1) The current file compiles only if the included file is in the INCLUDE search path, and needs to be re-compiled if the include file changes. 2) The executable needs to be linked with any objects the include file is dependent on. It needs to be re-linked if these objects have changed.
- The [#include '<file>'] statement in a Fortran/C source or header file is the fourth pattern. It has similar implications as the Fortran INCLUDE statement. However, they have to be handled differently because "#include" statements are processed by the pre-processor, which may be performed in a separate stage of the FCM build process. This will be further discussed in a later sub-section on Pre-processing.

If you want your code to be built automatically by the FCM build system, you should also design your code to conform to the following rules:

1. Single compilable program unit, (i.e. program, subroutine, function or module), per file.
2. Unique name for each compilable program unit.
3. Always supply an interface for subroutines and functions, i.e.:
   - Put them in modules.
   - Put them in the CONTAINS section within the main program unit.
   - Use interface files.
4. If interface files are used, it is good practise to name each source file after the program unit it contains. It will make life a lot simpler when using the Automatic Fortran 9X interface block feature, which has already been discussed in the previous section.
   - The problem is that, by default, the root name of the interface file is the same as that of the source file rather than the program unit. If they differ then the build system will create a dependency on the wrong object file (since the object files are named according to the program unit).
   - This problem can be avoided by changing the behaviour of the interface file generator to use the name of the program unit instead (using a "TOOL::INTERFACE" declaration).

Note - setting build targets
The Makefile (and its include fragments) generated by the build system contains a list of targets that can be built. The build system allows you to build (or perform the actions of) any targets that are present in the generated Makefile. There are two ways to specify the targets to be built. Firstly, you can use the TARGET declarations in your build configuration file to specify the default targets to be built. These targets will be set as dependencies of the "all" target in the generated Makefile, which is the default target to be built when make is invoked by FCM. Alternatively, you can use the "-t" option when you invoke the "fcm build" command. The option takes an argument, which should be a colon ":" separated list of targets to be built. When the "-t" option is set, FCM invokes make to build these targets instead. (E.g. if we invoke the build system with the command "fcm build -t foo.exe:bar.exe", it will invoke make to build "foo.exe" and "bar.exe".) If you do not specify any explicit targets, the system will search your source tree for main programs: If there are main programs in your source tree, they will be set as the default targets automatically. Otherwise, the default is to build the top level library archive containing objects compiled from the source files in the current source tree. (For more information on building library archives, please see the section on Creating library archives.)

## Advanced Features

### Further dependency features

Apart from the usual dependency patterns described in the previous sub-section, the automatic dependency scanner also recognises two special directives when they are inserted into a source file:

- The directive [DEPENDS ON: <object>] in a comment line of a Fortran/C source file: It states that the current file is dependent on the declared external object. The executable depending on the current file needs to link with this external object in order to resolve all its external references. It needs to be re-linked if the declared external object (and its dependencies) has changed.
- The directive [CALLS: <executable>] in a comment line of a script: It states that the current script is dependent on the declared executable file, which can be another script or a binary executable. The current script can only function correctly if the declared executable is found in the search path. This directive is useful to ensure that all dependent executables are built or copied to the correct path.

There are situations when you need to bypass the automatic dependency scanner. In such case, you may need to use a package configuration file. For further information, please refer to the sub-section on Using a package configuration file.

Another way to specify external dependency is to use the EXE_DEP declaration to declare extra dependencies. The declaration normally applies to all main programs, but if the the form EXE_DEP::<target> is used, it will only apply to <target>, (which must be the name of a main program target). If the declaration is made without a value, the main programs will be set to depend on all object files. Otherwise, the value can be supplied as a space delimited list of items. Each item can be either the name of a sub-package or an object target. For the former, the main programs will be set to depend on all object files within the sub-package. For the latter, the main programs will be set to depend on the object target. The following are some examples:

Build configuration example 5 - extra executable dependency
    cfg::type          ext
    cfg::version       1.0

    bld::exe_dep::foo.exe  foo::bar egg.o # line 4
    bld::exe_dep                          # line 5
    # ... some other declarations ...

Here is an explanation of what each line does:

- line 4: this line declares the dependency on the sub-package "foo::bar" and the object target "egg.o" for building the main program target "foo.exe". The target "foo.exe" will now depends on all object files in the "foo::bar" sub-package as well as the object target "egg.o".
- line 5: this line declares that all other main program targets will depend on all (non-program) object files in the build.

Note - naming of object files
By default, object files are named with the suffix ".o". For a Fortran source file, the build system uses the lower case name of the first program unit within the file to name its object file. For example, if the first program unit in the Fortran source file "foo.f90" is "PROGRAM Bar", the object file will be "bar.o". For a C source file, the build system uses the lower case root name of the source file to name its object file. For example, a C source file called "egg.c" will have its object file named "egg.o". The reason for using lower case to name the object files is because Fortran is a case insensitive language. Its symbols can either be in lower or upper case. E.g. the SUBROUTINE "Foo" is the same as the SUBROUTINE "foo". It can be rather confusing if the subroutines are stored in different files. When they are compiled and archived into a library, there will be a clash of namespace, as the Fortran compiler thinks they are the same. However, this type of error does not normally get reported. If "Foo" and "foo" are very different code, the user may end up using the wrong subroutine, which may lead to a very long debugging session. By naming all object files in lower case, this type of situation can be avoided. If there is a clash in names due to the use of upper/lower cases, it will be reported as warnings by make, (as "duplicated targets" for building "foo.o").

It is realised that there are situations when an automatically detected dependency should not be written into the Makefile. For example, the dependency may be a standard module provided by the Fortran compiler, and does not need to be built in the usual way. In such case, we need to have a way to exclude this module during an automatic dependency scan.

The EXCL_DEP declaration can be used to do just that. The following extract configuration contains some examples of the basic usage of the EXCL_DEP declaration:

Build configuration example 6 - exclude dependency
    cfg::type          ext
    cfg::version       1.0

    bld::excl_dep  USE::YourFortranMod             # line 4
    bld::excl_dep  INTERFACE::HerFortran.interface # line 5
    bld::excl_dep  INC::HisFortranInc.inc          # line 6
    bld::excl_dep  H::TheirHeader.h                # line 7
    bld::excl_dep  OBJ::ItsObject.o                # line 8

    # ... some other declarations ...

Here is an explanation of what each line does:

- line 4: this line declares that the Fortran module "YourFortranMod" should be excluded. The value of each EXCL_DEP declaration has two parts. The first part is a label that is used to define the type of dependency to be excluded. For a full list of these labels, please see the dependency types table in the Annex: Declarations in FCM build configuration file. The label "USE" denotes a Fortran module. The second part of the label is the dependency itself. For instance, if a Fortran source file contains the line: "USE YourFortranMod", the dependency scanner will ignore it.
- line 5: this line declares that the include statement for the Fortran 9X interface file "HerFortran.interface" should be excluded. The label "INTERFACE" denotes a Fortran INCLUDE statement for a Fortran 9X interface block file. For example, if a Fortran source file contains the line: "INCLUDE 'HerFortran.interface'", the dependency scanner will ignore it.
- line 6: this line declares that the include statement for "HisFortranInc.inc" should be excluded. The label "INC" denotes a Fortran INCLUDE statement other than an INCLUDE statement for an interface block file. For example, if a Fortran source file contains the line: "INCLUDE 'HisFortranInc.inc'", the dependency scanner will ignore it.
- line 7: this line declares that the header include statement "TheirHeader.h" should be excluded. The label "H" denotes a pre-processing #include statement. For example, if a source file contains the line: "#include 'TheirHeader.h'", the dependency scanner will ignore it.
- line 8: this line declares that the external dependency for "ItsObject.o" should be excluded. The label "OBJ" denotes a compiled binary object. These dependencies are normally inserted into the source files as special comments. For example, if a source file contains the line: "! depends on: ItsObject.o", the dependency scanner will ignore it.

An EXCL_DEP declaration normally applies to all files in the build. However, you can suffix it with the name of a sub-package, i.e. EXCL_DEP::<pcks>. In such case, the declaration will only apply while scanning for dependencies in the source files in the sub-package named <pcks>.

You can also exclude all dependency scan of a particular type. To do so, simply declare the type in the value. For example, if you do not want the build system to scan for the [CALLS: <executable>] directive in the comment lines of your scripts, you can make the following declaration:

    bld::excl_dep  EXE

N.B. Currently, the build system is unable to detect changes in EXCL_DEP declarations. Therefore, you will need to invoke the build system in full build mode if you have changed these settings.

### Linking a Fortran executable with a BLOCKDATA program unit

If it is required to link Fortran executables with BLOCKDATA program units, you must declare the executable targets and the objects containing the BLOCKDATA program units using the BLOCKDATA::<target> declarations. For example, if "foo.exe" is an executable target depending on the objects of the BLOCKDATA program units "blkdata.o" and "fbk.o", you will make the following declarations:

    bld::blockdata::foo.exe  blkdata fbk

If all your executables are dependent on "blkdata.o" and "fbk.o", you will make the following declarations:

    bld::blockdata  blkdata fbk

### Creating library archives

If you are interested in building library archives, the build system allows you to do it in a relatively simple way. For each sub-package in the source tree, there is a target to build a library containing all the objects compiled from the source files (that are not main programs) within the sub-package. If the sub-package contains children sub-packages, the object files of the children will also be included recursively. By default, the library archive is named after the sub-package, in the format "lib<pcks>.a". (For example, the library archive for the package "foo::bar::egg" will be named "libfoo__bar__egg.a" by default.) If you do not like the default name for the sub-package library, you can use the LIB::<pcks> declaration to rename it, as long as the new name does not clash with other targets. For example, to rename "libfoo__bar__egg.a" to "libham.a", you will make the following declaration in your extract configuration file:

    bld::lib::foo::bar::egg  ham

In addition to sub-package libraries, you can also build a global library archive for the whole source tree. By default, the library is named "libfcm_default.a", but you can rename it using the LIB declaration as above. For example, to rename the library to "libmy-lib.a", you will make the following declaration in your extract configuration file:

    bld::lib  my-lib

When a library archive is created successfully, the build system will automatically generate the relevant exclude dependency configurations in the "etc/" sub-directory of the build root. You will be able to include these configurations in subsequent builds that utilise the library. The root names of the configuration files match those of the library archives that you can create in the current build, but the extension "*.a" is replaced with "*.cfg". For example, the exclude dependency configuration for "libmy-lib.a" is "libmy-lib.cfg".

### Pre-processing

As most modern compilers can handle pre-processing, the build system leaves pre-processing to the compiler by default. However, it is recognised that there are code written with pre-processor directives that can alter the argument list of procedures and/or their dependencies. If a source file requires pre-processing in such a way, we have to pre-process before running the interface block generator and the dependency scanner. The PP declaration can be used to switch on this pre-processing stage. The pre-processing stage can be switched on globally or for individual sub-packages only. The following is an example, using an extract configuration file:

Build configuration example 7 - pre-processing switch
    cfg::type          ext
    cfg::version       1.0

    bld::pp::gen       1                     # line 4
    bld::pp::var::foo  1                     # line 5

    bld::tool::cppkeys GOOD WEATHER FORECAST # line 7
    bld::tool::fppkeys FOO BAR EGG HAM       # line 8

    # ... some other declarations ...

Here is an explanation of what each line does:

- line 4 to 5: these switches on the pre-processing stage for all sub-packages under "gen" and "var::foo".
- line 7: this declares a list of pre-defined macros "GOOD", "WEATHER" and "FORECAST" for pre-processing all C files.
- line 8: this declares a list of pre-defined macros "FOO", "BAR", "EGG" and "HAM" for pre-processing all Fortran files that require processing.

Source files requiring pre-processing may contain "#include" statements to include header files. For including a local file, its name should be embedded within a pair of quotes, i.e. 'file.h' or "file.h". If the header file is embedded within a pair of "<file.h>" angle brackets, the system will assume that the file can be found in a standard location.

The build system allows header files to be placed anywhere within the declared source tree. The system uses the dependency scanner, as described in the previous sub-section to scan for any header file dependencies. All source files requiring pre-processing and all header files are scanned. Header files that are required are copied to the "inc/" subdirectory of the build root, which is automatically added to the pre-processor search path via the "-I<dir>" option. The build system uses an internal logic similar to make to perform pre-processing. Header files are only copied to the "inc/" sub-directory if they are used in "#include" statements.

Unlike make, which only uses the timestamp to determine whether an item is out of date, the internal logic of the build system does this by inspecting the content of the file as well. In an incremental build, the pre-processed file is only updated if its content has changed. This avoids unnecessary updates (and hence unnecessary re-compilation) in an incremental build if the changed section of the code does not affect the output file.

Pre-processed code generated during the pre-processing stage are sent to the "ppsrc/" sub-directory of the build root. It will have a relative path that reflects the name of the declared sub-package. The pre-processed source file will have the same root name as the original source file. For C files, the same extension ".c" will be used. For Fortran files, the case of the extension will normally be dropped, e.g. from ".F90" to ".f90".

Following pre-processing, the system will use the pre-processed source file as if it is the original source file. The interface generator will generate the interface file using the pre-processed file, the dependency scanner will scan the pre-processed file for dependencies, and the compiler will compile the pre-processed source.

The TOOL::CPPKEYS and TOOL::FPPKEYS declarations are used to pre-define macros in the C and Fortran pre-processor respectively. This is implemented by the build system using the pre-processor "-D" option on each word in the list. The use of these declarations are not confined to the pre-process stage. If any source files requiring pre-processing are left to the compiler, the declarations will be used to set up the commands for compiling these source files.

The TOOL::CPPKEYS and TOOL::FPPKEYS declarations normally applies globally, but like any other TOOL declarations, they can be suffixed with sub-package names. In such cases, the declarations will apply only to the specified sub-packages.

Note - changing pre-processor flags
As for compiler flags, the build system detects changes in pre-processor flags (TOOL::CPPFLAGS and TOOL::FPPFLAGS) and macro definitions (TOOL::CPPKEYS and TOOL::FPPKEYS). If the pre-processor flags or the macro definitions have changed in an incremental build, the system will re-do all the necessary pre-processing. The following hierarchy is followed: If the pre-processor flags or macro definitions for a particular source file change, only that source file will be pre-processed again. If the pre-processor flags or macro definitions for a particular sub-package change, only source files within that sub-package will be pre-processed again. If the global pre-processor flags or macro definitions change, all source files will be pre-processed again. If the pre-processor command changes, all source files are pre-processed again.

### File type

The build system only knows what to do with an input source file if it knows what type of file it is. The type of a source file is normally determined automatically using one of the following three methods (in order):

1. If the file is named with an extension, its extension will be matched against a set of registered file extensions. If a match is found, the file type will be set according to the register.
2. If a file does not have an extension or does not match with a registered extension, its name is compared with a set of pre-defined patterns. If a match is found, the file type will be set according to the file type associated with the pattern.
3. If the above two methods failed and if the file is a text file, the system will attempt to read the first line of the file. If the first line begins with a "#!" pattern, the line will be compared with a set of pre-defined patterns. If a match is found, the file type will be set according to the file type associated with the pattern.

In addition to the above, if a file is a Fortran or C source file, the system will attempt to open the source file to determine whether it contains a main program, module (Fortran only) or just standalone procedures. All these information will be used later by the build system to process the source file.

The build system registers a file type with a set of type flags delimited by the double colons "::". For example, a Fortran 9X source file is registered as "FORTRAN::FORTRAN9X::SOURCE". (Please note that the order of the type flags in the list is insignificant. For example, "FORTRAN::SOURCE" is the same as "SOURCE::FORTRAN".) For a list of all the type flags used by the build system, please see the input file extension type flags table in the Annex: Declarations in FCM build configuration file.

The following is a list of default input file extensions and their associated types:

Extensions | Type flags | Description
--- | --- | ---
.f .for .ftn .f77 | FORTRAN::SOURCE | Fortran 77 source file (assumed to be fixed format)
.f90 .f95 | FORTRAN::FORTRAN9X::SOURCE | Fortran 9X source file (assumed to be free format)
.F .FOR .FTN .F77 | FPP::SOURCE | Fortran 77 source file (assumed to be fixed format) that requires pre-processing
.F90 .F95 | FPP::FPP9X::SOURCE | Fortran 9X source file (assumed to be free format) that requires pre-processing
.c | C::SOURCE | C source file
.h .h90 | CPP::INCLUDE | Pre-processor "#include" header file
.o .obj | BINARY::OBJ | Compiled binary object
.exe | BINARY::EXE | Binary executable
.a | BINARY::LIB | Binary object library archive
.sh .ksh .bash .csh | SHELL::SCRIPT | Unix shell script
.pl .pm | PERL::SCRIPT | Perl script
.py | PYTHON::SCRIPT | Python script
.tcl | TCL::SCRIPT | Tcl/Tk script
.pro | PVWAVE::SCRIPT | PVWave program
.cfg | CFGFILE | FCM configuration file
.inc | FORTRAN::FORTRAN9X::INCLUDE | Fortran INCLUDE file
.interface | FORTRAN::FORTRAN9X::INCLUDE::INTERFACE | Fortran 9X INCLUDE interface block file

N.B. The extension must be unique. For example, the system does not support the use of ".inc" files for both "#include" and Fortran "INCLUDE".

The following is a list of supported file name patterns and their associated types:

Patterns | Type flags | Description
--- | --- | ---
*Scr_* *Comp_* *IF_* *Suite_* *Interface_* | SHELL::SCRIPT | Unix shell script, GEN-based project naming conventions
*List_* | SHELL::SCRIPT::GENLIST | Unix shell script, GEN "list" file
*Sql_* | SCRIPT::SQL | SQL script, GEN-based project naming conventions

The following is a list of supported "#!" line patterns and their associated types:

Patterns | Type flags | Description
--- | --- | ---
*sh* *ksh* *bash* *csh* | SHELL::SCRIPT | Unix shell script
*perl* | PERL::SCRIPT | Perl script
*python* | PYTHON::SCRIPT | Python script
*tclsh* *wish* | TCL::SCRIPT | Tcl/Tk script

The build system allows you to add or modify the register for input file extensions and their associated type using the INFILE_EXT::<ext> declaration, where <ext> is a file name extension without the leading dot. For example, in an extract configuration file, you may have:

Build configuration example 8 - add/modify input file extension types
    cfg::type             ext
    cfg::version          1.0

    bld::infile_ext::foo  CPP::INCLUDE                # line 4
    bld::infile_ext::bar  FORTRAN::FORTRAN9X::INCLUDE # line 5

    # ... some other declarations ...

Here is an explanation of what each line does:

- line 4: this line registers the extension ".foo" to be of type "CPP::INCLUDE". This means that any input files with ".foo" extension will be treated as if they are pre-processor header files.
- line 5: this line registers the extension ".bar" to be of type "FORTRAN::FORTRAN9X::INCLUDE". This means that any input file with ".bar" extension will be treated as if they are Fortran 9X INCLUDE files.

The INFILE_EXT declarations deal with extensions of input files. There is also a OUTFILE_EXT::<type> declaration that deals with extensions of output files. The declaration is opposite that of INFILE_EXT. The file <type> is now declared with the label, and the extension is declared as the value. It is worth noting that OUTFILE_EXT declarations use very different syntax for <type>, and the declared extension must include the leading dot. For a list of output types used by the build system, please see the output file extension types table in the Annex: Declarations in FCM build configuration file. An example is given below:

Build configuration example 9 - modify output file extensions
    cfg::type                   ext
    cfg::version                1.0

    bld::outfile_ext::mk        .rule  # line 4
    bld::outfile_ext::mod       .MOD   # line 5
    bld::outfile_ext::interface .intfb # line 6

    # ... some other declarations ...

Here is an explanation of what each line does:

- line 4: this line modifies the extension of output Makefile fragments, output of the dependency scanner, from the default ".mk" to ".rule".
- line 5: this line modifies the extension of compiled Fortran 9X module information files from the default ".mod" to ".MOD".
- line 6: this line modifies the extension of INCLUDE Fortran 9X interface block files from the default ".interface" to ".intfb".

N.B. If you have made changes to the file type registers, whether it is for input files or output files, it is always worth re-building your code in full-build mode to avoid unexpected behaviour.

### Incremental build based on a pre-compiled build

As you can perform incremental extractions against pre-extracted source code, you can perform incremental builds against pre-compiled builds. The very same USE statement can be used to declare a build, which the current build will depend on. The only difference is that the declared location must contain a valid build configuration file. In fact, if you use the extract system to obtain your build configuration file, any USE declarations in the extract configuration file will also be USE declarations in the output build configuration file.

By declaring a USE statement, the current build automatically inherits settings from the pre-compiled build. The following points are worth noting:

- Build targets are not normally inherited. However, you can switch on inheritance of build targets using an INHERIT::TARGET declaration, such as:

      inherit::target  1

- The build root directory and its sub-directories of the pre-compiled build are placed into the search paths. For example, if we have a pre-compiled build at "/path/to/pre/compiled", and it is used by an incremental build at "/path/to/incremental", the search path of executable files will become "/path/to/incremental/bin:/path/to/pre/compiled/bin", so that the "bin/" sub-directory of the incremental build is searched before the "bin/" sub-directory of the pre-compiled build. If two or more USE statements are declared, the USE statement declared last will have higher priority. For example, if the current build is "C", and it USEs build "A" before build "B", the search path will be "C:B:A".
- Sub-package source directories are inherited by default. If a source directory is declared in the current build that has the same sub-package name as a source directory of the pre-compiled build, any source files in the source directory of the current build will override those in the source directory of the pre-compiled build. Any source files missing from the source directory of the current build will be taken from the source directory of the pre-compiled build.

  For example, if build "B" USEs build "A", and both of them have a sub-package called "basic::element". In build "A", the sub-package contains the source files "earth.f90", "wind.f90" and "water.f90". In build "B", the sub-package contains "earth.f90" and "fire.f90". When the system searches for source files, it will look for source files in "B" before "A". Therefore, the source file "earth.f90" in "B" will override that of "A", and the files used for the build will be "earth.f90" and "fire.f90" from "B" and "wind.f90" and "water.f90" from "A".

  You can switch off inheritance of source directories using an INHERIT::SRC declaration. This declaration can be suffixed with the name of a sub-package. In such case, the declaration applies only to the inheritance of the sub-package. Otherwise, it applies to all source directories. For example:

      # Switch off inheritance of source directories in the "gen" sub-package
      inherit::src::gen  0

- All build tools are automatically inherited. If the same tool is declared in the current incremental build, it overrides the declaration in the pre-compiled build.
- "PP", "EXCL_DEP", "INFILE_EXT" and "OUTFILE_EXT" declarations are inherited using a similar mechanism as build tool declarations.

As an example, suppose we have already performed an extract and build based on the configuration in example 2, we can set up an extract configuration file as follows:

Build configuration example 10 - "USE" a pre-compiled build
    cfg::type            ext
    cfg::version         1.0

    use                  $HOME/example               # line 4

    dest::rootdir        $HOME/example9              # line 6

    bld::inherit::target 1                           # line 8
    bld::target          ham.exe egg.exe             # line 9

    bld::tool::fflags    -O2 -w                      # line 11
    bld::tool::cflags    -O2                         # line 12

    # ... and other declarations for repositories and source directories ...

Here is an explanation of what each line does:

- line 4: this line declares a previous extraction at "$HOME/example" which the current extraction will be based on. The same line will be written to the output build configuration file at "$HOME/example9/cfg/bld.cfg". The subsequent build will then be based on the build at "$HOME/example".
- line 6: this declares the destination root directory of the current extraction, which will become the root directory of the current build. Search paths of the build sub-directories will be set automatically. For example, the search path for executable files created by the current build will be "$HOME/example9/bin:$HOME/example/bin".
- line 8: this line switches on inheritance of build targets. The build targets in example 1, i.e. "foo.exe" and "bar.exe" will be built as part of the current build.
- line 9: this declares two new build targets "ham.exe" and "egg.exe" to be added to the inherited ones. The default build targets of the current build will now be "foo.exe", "bar.exe", "ham.exe" and "egg.exe".
- line 11-12: these lines modify options used by the Fortran and the C compilers, overriding those inherited from example 1.

### Using a package configuration file

It is recognised that the build system needs to be able to work with code that is not designed to work with the automatic dependency scanner. There are various techniques which can be used to achieve this, some of which are already described. However, it should be noted that these techniques are simply designed to allow code to be successfully built. They will not necessarily result in a good environment for developing such code, since for example, dependencies need to be manually maintained by the developer.

The final technique is the package configuration file. The package configuration file is a special source file to inform the build system about any additional information of the source files in the package. It must be a file called "@PACKAGE.cfg" in a source directory. For a full list of available declarations in the package configuration file, please refer to the Annex: Declarations in FCM build package configuration file.

The following is an example of how a package configuration file can be used to provide additional information to the build system for a source file:

Package configuration example 1 - dependency information for a source file
    type::MyFortranProg.f90    FORTRAN::FORTRAN9X::SOURCE::PROGRAM # line 1
    scan::MyFortranProg.f90    0                                   # line 2
    intname::MyFortranProg.f90 myprog                              # line 3
    target::MyFortranProg.f90  hello_world                         # line 4

    dep::MyFortranProg.f90     USE::YourFortranMod                 # line 6
    dep::MyFortranProg.f90     INTERFACE::HerFortran.interface     # line 7
    dep::MyFortranProg.f90     INC::HisFortranInc.inc              # line 8
    dep::MyFortranProg.f90     H::TheirHeader.h                    # line 9
    dep::MyFortranProg.f90     OBJ::ItsObject.o                    # line 10

    # ... some other declarations ...

Here is an explanation of what each line does:

- line 1: this declares a source file "MyFortranProg.f90" in the package of type "FORTRAN::FORTRAN9X::SOURCE::PROGRAM", i.e. a Fortran 9X source file containing a main program. The list of type flags used in the value is the same list of type flags used in the INFILE_EXT declaration of a build configuration file. For a list of these flags, please see the Input file extension type flags table in the Annex: Declarations in FCM build configuration file.
- line 2: this declaration switches off (0) automatic dependency scan for this source file. If this switch is not declared, it is automatically set to on (1) for a source file. If the switch is on, the dependency scanner will scan the source file for further dependencies. Otherwise, dependency information for this source file will only be read from this configuration file.
- line 3: this declares the internal name of the source file to be "myprog". If this is not set and the dependency scanner is switched on, the internal name of the source file will be taken from the first program unit of the source file. The resulting object file will be named after the internal name, plus the ".o" extension. If internal name is not set and cannot be determined automatically, the default is to use the root name of the source file to name the object file.
- line 4: this declares the executable target name of a main program. This declaration is only used when the source file contains a main program. If the target name is not set, the name of the executable will be named after the root name of the source file plus the ".exe" extension. In the above example, the executable will be named "hello_world".
- line 6-10: these lines declare the dependencies of the source file. The syntax of the values are exactly the same as those used in the EXCL_DEP declarations in the build configuration file. For a list of these flags, please see the dependency types table in the Annex: Declarations in FCM build configuration file. In the example, the source file is dependent on a Fortran module called "YourFortranMod", a Fortran INCLUDE interface block file called "HerFortran.interface", a Fortran INCLUDE file called "HistFortran.inc, a pre-processor header file called "TheirHeader.h", and an object file called "ItsObject.o".

### Building data files

While the usual targets to be built are the executables associated with source files containing main programs, libraries or scripts, the build system also allows you to build "data" files. All files with no registered type are considered to be "data" files. For each sub-package, there is an automatic target for copying all "data" files to the "etc/" sub-directory of the build root. The name of the target has the form "<pcks>.etc", where <pcks> is the name of the sub-package (with package names delimited by the double underscore "__"). For example, the target name for sub-package "foo::bar" is "foo__bar.etc". This target is particularly useful for copying, say, all namelists in a sub-package to the "etc/" sub-directory of the build root.

At the end of a successful build, if the "etc/" sub-directory is not empty, the "fcm_env.ksh" script will export the environment variable FCM_ETCDIR to point to the "etc/" sub-directory. You should be able to use this environment variable to locate your data files.

## Diagnostic verbose level

The amount of diagnostic messages generated by the build system is normally set to a level suitable for normal everyday operation. This is the default diagnostic verbose level 1. If you want a minimum amount of diagnostic messages, you should set the verbose level to 0. If you want more diagnostic messages, you can set the verbose level to 2 or 3. You can modify the verbose level in two ways. The first way is to set the environment variable FCM_VERBOSE to the desired verbose level. The second way is to invoke the build system with the "-v <level>" option. (If set, the command line option overrides the environment variable.)

The following is a list of diagnostic output at each verbose level:

Verbose level | Possible output
--- | ---
0 | Report the time taken at the end of each stage of the build process. Run the make command in silent mode.
1 | Everything at verbose level 0. Report the name of the build configuration file. Report date/time at the beginning of each stage of the build process. Report removed directories. Report number of pre-processed files. Report number of generated F9X interface files. Report number of sub-packages that have source files scanned for dependencies. Report number of generated/updated sub-package make rule fragment files. Report name of updated Makefile. Print compiler/linker commands. Report total time.
2 | Everything at verbose level 1. For incremental build in archive mode, report the commands used to extract the archives. Report creation and removal of directories. Report pre-processor commands. Report number of files scanned for dependency in each sub-package. Report names of generated/updated sub-package make rule fragment files. Print compiler/linker commands with timestamps. Print usage of the runtime environment set up script.
3 | Everything at verbose level 2. Report update of dummy files. Report all shell commands. Report pre-processor commands with timestamps. If F9X interface is generated by f90aib, print commands with timestamps. If F9X interface is generated by ECMWF code, report start date/time and time taken to generate each interface file. Report start date/time and time taken of dependency scan for each source file. Run make on normal mode (as opposed to silent mode). Report start date/time and time taken of make commands.

## Overview of the build process

The FCM build process can be summarised in five stages. Here is a summary of what is done in each stage:

1. Setup: in this first stage, the build system reads and processes the configuration file. The source sub-directory is searched recursively for source directories. For full builds, it ensures that the sub-directories created by the build system are removed. For inherited (i.e. pre-compiled) builds, it sets up the search paths for the sub-directories, inherits the targets, source directories, etc. For incremental builds, the system also works out whether the current list of build tools have changed.
2. Pre-process: if any files in any source directories require pre-processing, they will be pre-processed at this stage. The resulting pre-processed source files will be sent to the "ppsrc/" sub-directory of the build root.
3. Generate dependency: the system scans source files of registered types for dependency information. For an incremental build, the information is only updated if a source file is changed. The system then uses the information to write a Makefile for the main build. The Makefile and any of its included fragments are generated in the "bld/" sub-directory of the build root.
4. Generate interface: if there are Fortran 9X source files with standalone subroutines and functions, the build system generates interface blocks for them. The result of which will be written to the interface files in the "inc/" sub-directory of the build root.
5. Make: the system invokes make on the Makefile generated in the previous stage to perform the main build. Following a build, the "root" directory of the build may contain the following sub-directories (empty ones are removed automatically at the end of the build process):

   Sub-directory | Contents | Note
   --- | --- | ---
   .cache/ | <cache files> | used internally by FCM
   bin/ | <executable binaries and scripts> | -
   bld/ | Makefile and <Makefile include files> | -
   cfg/ | bld.cfg | this directory is not changed by the build system
   done/ | <dummy "done" files> | used internally by the Makefile generated by FCM
   etc/ | <miscellaneous data files> | -
   flags/ | <dummy "flags" files> | used internally by the Makefile generated by FCM
   inc/ | <include files> | such as *.h, *.inc, *.interface and *.mod files
   lib/ | <object library archives> | -
   obj/ | <compiled object files> | -
   ppsrc/ | <source directories with pre-processed files> | -
   src/ | <source directories> | this directory is not changed by the build system
   tmp/ | <temporary objects and binaries> | files generated by the compiler/linker may be left here