<commit_before>---
layout: default
title: Configure, Build and Use the Toolkit with CMake
nav: pages/ch_cmconfig
---


{{ page.title }}
=================================================

## Introduction

This chapter describes how to configure, build, and use the NCBI C++ Toolkit, or selected components of it using CMake.

[CMake](https://cmake.org) is an open-source, cross-platform family of tools designed to build, test and package software. It uses compiler-independent configuration files to generate native makefiles and workspaces which can be used in a variety of compiler environments.

At NCBI, we use NCBIptb – CMake wrapper, written in CMake scripting language. It adds many convenient features, facilitates handling of large source trees and simplifies CMake build target descriptions, while still allowing use of “native” CMake.


## Chapter Outline

-   [Configure build tree](#ch_cmconfig._Configure)

-   [Use prebuilt Toolkit](#ch_cmconfig._Use_prebuilt)

-   [NCBIptb build system](#ch_cmconfig._NCBIptb)

    -   [What is it?](#ch_cmconfig._What)
    
    -   [Examples](#ch_cmconfig._Examples)
    
    -   [How does it work?](#ch_cmconfig._How)
    
    -   [Tree structure and variable scopes](#ch_cmconfig._Tree)

    -   [Directory entry](#ch_cmconfig._Dir)
    
    -   [Library and application targets.](#ch_cmconfig._Target)
    
    -   [Application target tests.](#ch_cmconfig._Test)
    
    -   [Custom target.](#ch_cmconfig._Custom)

-   [Testing](#ch_cmconfig._Testing)


<a name="ch_cmconfig._Configure"></a>

## Configure build tree

Having checked out the source tree, run the following command in the root directory:

    On Linux:   src/build-system/cmake/cmake-cfg-unix.sh --help
    On Windows: src\build-system\cmake\cmake-cfg-vs.bat --help
    For XCode: src/build-system/cmake/cmake-cfg-xcode.sh --help

It lists available options used to generate the build tree. Several of them limit the build scope:

-   *--with-projects=*”FILE” – build projects listed in FILE. This is either a Project List File, or a list of subdirectories of *src* directory. Each entry is a regular expression (see [CMake documentation](https://cmake.org/cmake/help/v3.14/command/string.html#regex-specification) for details). List of subdirectories consists of entries separated by semicolon (hyphen in the beginning of the entry means that targets from that directory should be excluded). If "FILE" is an existing file, it should be in plain text with one such entry per line. For example:

```
    --with-projects="corelib$;serial;-serial/test"
    --with-projects="scripts/projects/ncbi_cpp.lst"
```

-   *--with-targets=*”NAMES” – lists targets to build. Again, each entry is a regular expression. For example:

```
    --with-targets="^cgi;-test"
```

-   *--with-tags=*”TAGS” – build targets with the listed tags only. Tags are **not** treated as regular expressions. For example:

```
    --with-tags="core;-test"
```

Examples of configuration commands:

    src/build-system/cmake/cmake-cfg-unix.sh --with-dll --with-debug --with-projects="sra"
    src/build-system/cmake/cmake-cfg-vs.sh --with-projects="misc"

Once the build tree is generated, go into build directory – for example, *CMake-GCC730-ReleaseDLL/build* or *CMake-VS2017\build*, and run *make [target]* command or open a generated solution in an IDE and build *target*.

<a name="ch_cmconfig._Use_prebuilt"></a>

## Use prebuilt Toolkit

The prebuilt Toolkit is available in several configurations. ***Note*** that this must be built using CMake – that is, it must contain CMake import target configuration files. To create a new project which uses libraries from it, use *new_cmake_project* script:

    new_cmake_project <name> <type> <builddir>

The script will create a subdirectory *name*, source subdirectories with a sample project *type* and a configuration script. Run the script, then build the project.

<a name="ch_cmconfig._NCBIptb"></a>

## NCBIptb build system

<a name="ch_cmconfig._What"></a>

### What is it?

Imagine a large source tree with thousands of projects. It takes sources from several repositories and uses numerous external packages. It consists of “core” part and subtrees. Different teams work on multiple projects. To do their work, these teams assemble their own build trees, which include certain parts of core as well as their own projects. “Core” has several official releases; team projects have their own ones. There are several high frequency builds which work on different subtrees and ensure that everything stays compatible.

NCBIptb was designed to facilitate handling of such large source tree in a dynamic build environment. The purpose of NCBIptb is to extract from the source tree only requested projects. This includes analyzing and collecting build target dependencies on other targets, analyzing dependencies on external packages and excluding targets for which such dependencies cannot be satisfied, adding sources and headers, organizing them into source groups, defining precompiled header usage.  Doing this in “pure” CMake is either impossible or requires complex project descriptions. Still, all these tasks are pretty standard and can be automated. Using NCBIptb is convenience, not a requirement; it extends functionality but still allows using “native” CMake.

Probably, limiting the set of projects to build is not such a big problem for an automated build – it must build everything anyway. Still, it is a problem for an individual developer working in an Integrated Development Environment. IDEs can handle solutions with hundreds of build targets, but they can become very slow. And, it is simply not needed, it is a waste of computer resources. Developer needs a solution with only few build targets. NCBIptb can do exactly that.

Another challenge is working with prebuilt trees. Let us say, a developer needs to add features into some applications or libraries. He or she checks out part of the source tree. Some libraries are present in the local tree, others should be taken from the prebuilt one. The problem is that local libraries may have the same names as prebuilt ones. CMake will not add them into the build saying that these targets are already defined as “imported” ones. NCBIptb solves this problem by renaming such local build targets and adjusting target dependencies accordingly. Developer’s intervention is not required, everything is made automatically.

<a name="ch_cmconfig._Examples"></a>

### Examples

To define a simple build target in CMake we use the following command:

    add_executable(hello hello.cpp)

The same operation in NCBIptb will look like this:

    NCBI_begin_app(hello)
    NCBI_sources(hello.cpp)
    NCBI_end_app()

So far, it does not look like NCBIptb makes a lot of sense.<commit_msg>Update ch_cmconfig.md<commit_after>---
layout: default
title: Configure, Build and Use the Toolkit with CMake
nav: pages/ch_cmconfig
---


{{ page.title }}
=================================================

## Introduction

This chapter describes how to configure, build, and use the NCBI C++ Toolkit, or selected components of it using CMake.

[CMake](https://cmake.org) is an open-source, cross-platform family of tools designed to build, test and package software. It uses compiler-independent configuration files to generate native makefiles and workspaces which can be used in a variety of compiler environments.

At NCBI, we use NCBIptb – CMake wrapper, written in CMake scripting language. It adds many convenient features, facilitates handling of large source trees and simplifies CMake build target descriptions, while still allowing use of “native” CMake.


## Chapter Outline

-   [Configure build tree](#ch_cmconfig._Configure)

-   [Use prebuilt Toolkit](#ch_cmconfig._Use_prebuilt)

-   [NCBIptb build system](#ch_cmconfig._NCBIptb)

    -   [What is it?](#ch_cmconfig._What)

    -   [All features at a glance](#ch_cmconfig._Features)
    
    -   [Examples](#ch_cmconfig._Examples)
    
    -   [How does it work?](#ch_cmconfig._How)
    
    -   [Tree structure and variable scopes](#ch_cmconfig._Tree)

    -   [Directory entry](#ch_cmconfig._Dir)
    
    -   [Library and application targets.](#ch_cmconfig._Target)
    
    -   [Application target tests.](#ch_cmconfig._Test)
    
    -   [Custom target.](#ch_cmconfig._Custom)

-   [Testing](#ch_cmconfig._Testing)


<a name="ch_cmconfig._Configure"></a>

## Configure build tree

Having checked out the source tree, run the following command in the root directory:

    On Linux:   src/build-system/cmake/cmake-cfg-unix.sh --help
    On Windows: src\build-system\cmake\cmake-cfg-vs.bat --help
    For XCode: src/build-system/cmake/cmake-cfg-xcode.sh --help

It lists available options used to generate the build tree. Several of them limit the build scope:

-   *--with-projects=*”FILE” – build projects listed in FILE. This is either a Project List File, or a list of subdirectories of *src* directory. Each entry is a regular expression (see [CMake documentation](https://cmake.org/cmake/help/v3.14/command/string.html#regex-specification) for details). List of subdirectories consists of entries separated by semicolon (hyphen in the beginning of the entry means that targets from that directory should be excluded). If "FILE" is an existing file, it should be in plain text with one such entry per line. For example:

```
    --with-projects="corelib$;serial;-serial/test"
    --with-projects="scripts/projects/ncbi_cpp.lst"
```

-   *--with-targets=*”NAMES” – lists targets to build. Again, each entry is a regular expression. For example:

```
    --with-targets="^cgi;-test"
```

-   *--with-tags=*”TAGS” – build targets with the listed tags only. Tags are **not** treated as regular expressions. For example:

```
    --with-tags="core;-test"
```

Examples of configuration commands:

    src/build-system/cmake/cmake-cfg-unix.sh --with-dll --with-debug --with-projects="sra"
    src\build-system\cmake\cmake-cfg-vs.bat --with-projects="misc"

Once the build tree is generated, go into build directory – for example, *CMake-GCC730-ReleaseDLL/build* or *CMake-VS2017\build*, and run *make [target]* command or open a generated solution in an IDE and build *target*.

<a name="ch_cmconfig._Use_prebuilt"></a>

## Use prebuilt Toolkit

The prebuilt Toolkit is available in several configurations. ***Note*** that this must be built using CMake – that is, it must contain CMake import target configuration files. To create a new project which uses libraries from it, use *new_cmake_project* script:

    new_cmake_project <name> <type> <builddir>

The script will create a subdirectory *name*, source subdirectories with a sample project *type* and a configuration script. Run the script, then build the project.

<a name="ch_cmconfig._NCBIptb"></a>

## NCBIptb build system

<a name="ch_cmconfig._What"></a>

### What is it?

Imagine a large source tree with thousands of projects. It takes sources from several repositories and uses numerous external packages. It consists of “core” part and subtrees. Different teams work on multiple projects. To do their work, these teams assemble their own build trees, which include certain parts of core as well as their own projects. “Core” has several official releases; team projects have their own ones. There are several high frequency builds which work on different subtrees and ensure that everything stays compatible.

NCBIptb was designed to facilitate handling of such large source tree in a dynamic build environment. The purpose of NCBIptb is to extract from the source tree only requested projects. This includes analyzing and collecting build target dependencies on other targets, analyzing dependencies on external packages and excluding targets for which such dependencies cannot be satisfied, adding sources and headers, organizing them into source groups, defining precompiled header usage.  Doing this in “pure” CMake is either impossible or requires complex project descriptions. Still, all these tasks are pretty standard and can be automated. Using NCBIptb is convenience, not a requirement; it extends functionality but still allows using “native” CMake.

Probably, limiting the set of projects to build is not such a big problem for an automated build – it must build everything anyway. Still, it is a problem for an individual developer working in an Integrated Development Environment. IDEs can handle solutions with hundreds of build targets, but they can become very slow. And, it is simply not needed, it is a waste of computer resources. Developer needs a solution with only few build targets. NCBIptb can do exactly that.

Another challenge is working with prebuilt trees. Let us say, a developer needs to add features into some applications or libraries. He or she checks out part of the source tree. Some libraries are present in the local tree, others should be taken from the prebuilt one. The problem is that local libraries may have the same names as prebuilt ones. CMake will not add them into the build saying that these targets are already defined as “imported” ones. NCBIptb solves this problem by renaming such local build targets and adjusting target dependencies accordingly. Developer’s intervention is not required, everything is made automatically.

<a name="ch_cmconfig._Features"></a>

### All features at a glance

When analyzing source tree, NCBIptb does not expect much from  it, anything may be absent - subdirectories, projects, external components. NCBIptb issues a warning and it is up to developer to decide what to do with it.

When configuring the build tree, NCBIptb does the following:

-   Filters build targets by their location in the source tree, by name, and by tag, in any combination.

-   Collects all dependencies of requested build targets.

-   Excludes targets for which requirements are not met.

-   Adds header files.

-   Adds certain source and resource files, when required. This includes DLL or Windows GUI application entry points on Windows.

-   Creates source groups.

-   Defines precompiled header usage.

-   When using prebuilt trees, identifies local targets with the same names as imported ones, and renames them, adjusting dependencies. Using local targets has priority.

-   Defines tests.

In addition, there is an option of creating "composite" build targets. Such targets are composed of other targets, including their sources and requirements. In the build tree, such "hosted" targets are excluded and replaced with "composite" ones.

There are also tasks of source code generation, installation and testing. NCBIptb does not support them directly. Instead, there is a mechanism of plugin extensions which handle them.

### Examples

To define a simple build target in CMake we use the following command:

    add_executable(hello hello.cpp)

The same operation in NCBIptb will look like this:

    NCBI_begin_app(hello)
    NCBI_sources(hello.cpp)
    NCBI_end_app()

So far, it does not look like NCBIptb makes a lot of sense.
{'fraction_non_alphanumeric': 0.07928694958094985, 'fraction_numerical': 0.0013303179459890914, 'mean_word_length': 4.245987438939288, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1030087', 'n_tokens_mistral': 4023, 'n_tokens_neox': 3837, 'n_words': 1918}