<commit_before># gnumake-molds
Molds for building software with GNU make

Goals
=====

The goal of gnumake-mold is to make it easy to create cross-compiled libraries
and executables for a variety of architectures using only GNU make.

Highlights:

- compile large code bases that bring together multiple libraries
- manage multiple compilers, multiple build types, and multiple architectures
- streamline library compilation and linking
- streamline include path management by creating 'namespaces' for libraries

There are a variety of package management systems and root filesystem creation
tools available today that will build large systems from source code.  This is
not the focus of gnumake-mold. Instead, the focus is on creating complete,
fully linked executables that are compiled completely from source code, or that
are statically linked against a number of libraries that are also compiled from
source.

Cross-compiling
---------------

- Cross-compile embedded applications for different architectures with a single Makefile
- Out-of-the-box support for gcc and clang x86/ARM
- Create libaraies with different complier options (e.g. ARM vs Thumb code)

Target generation
-----------------

- Build the same source file with different compiler settings
- Create multiple executable and library target outputs with ease
- Separate definition of program source and structure from creating rules

Large projects
--------------

- Manage large projects with multiple "sub-components"
- Combine components from different parts of a filesystem (re-use installed library source in multiple projects)
- Create release directory structures with libraries, binaries, and includes
- Build against libraries either from source or from released package

Languages
---------

The initial vision is to support core languages in the GCC/LLVM toolchains:

- C
- C++
- Objective C

A longer term vision might be to expand this support to include languages such as

- Rust
- Swift
- Go

Usability
---------

- Tab auto-completion works with targets defined by Mold
- Makes it easier to debug by dumping full commands with options
- Unit testable for validation and performance testing
- Work with in-place object files for the smallest projects
- Support for automatic dependency generation

Key Concepts
============

Toolchain
---------

A toolchain is a set of tools that are used to translate source code into binary form for execution on a specific architecture.

Code generated by one toolchain cannot be linked or combined with code generated for another toolchain.

Examples: gcc, clang, gcc-debug, clang-release, clang-debug.

While a toolchain must include all tools required to produce an executable binary or library output, the name of the toolchain is generally based upon the compiler.

Generally, debug versus release build variants are handled at the toolchain level due to the significant impact that may be present on generated code.

- `MOLD_TOOLS`: All available toolchains
- `MOLD_TOOL`: Current toolchain

Architecture
------------

A build architecture is defined by a specific CPU type and associated instruction set.

Examples: host, amd64, cortex-m4, cortex-r7, avr, mipsel, etc.

Code generated for different architectures does not interoperate and cannot be linked together.

- `MOLD_ARCHS`: All available architectures for current toolchain
- `MOLD_ARCH`: Current architecture for current toolchain

Good discussion on architecture naming here: http://clang.llvm.org/docs/CrossCompilation.html

Target
------

A target is a single output that is produced by mold. Mold can produce
two different types of targets:

- Executable: a fully-linked binary executable
- Library: a static library with header files

All targets share the following variables:

- `target_<source>` - source files used to create target
- `target_<name>` - base name used as part of output files
- `target_<lib>` - libraries that are required by this target

If a library is used by an executable, the headers for that library are
automatically included as part of the system includes. If external,
pre-installed libraries are to be used, an appropriate mold template must be
created for them.

	test_name = mytester
	test_lib = freertos libc lwip

Separate target lists for each of the above types:

	MOLD_EXE_TARGETS
	MOLD_LIB_TARGETS

Each of the outputs are placed in the following directories:

	static_lib, dynamic_lib -> LIB_DIR (e.g. $PREFIX/lib)
	exe -> EXE_DIR (e.g. $PREFIX/bin)
	headers -> HEADER_DIR (e.g. $PREFIX/include)

Embedded platforms may make use of some additional special targets:

	memory_map - guidance for linker on how to place symbols
	raw_image - raw read only code for execution out of flash
	hex_image - hex output of raw flash image

Each target also has a unique set of variables:

	source - files required for target generation in any language supported by toolchain

	c_defines - preprocessor defines for C compiler
	c_include - include paths for C compiler

	cpp_defines - preprocessor defines for CPP compiler
	cpp_include - include paths for CPP compiler

	exe_static_lib - static libraries linked into exe
	exe_dynamic_lib - dynamic libraries linked against exe

These target options may need to be extended to support system include paths
and system libraries. However, this may also be achieved through an appropriate
use of toolchains. The best option remains to be seen.

Example of a static library:

	freertos_headers = $(freertos_home)/include
	freertos_headers_prefix = freertos
	freertos_source = $(freertos_home)/src/*
	freertos_c_defines = ENABLE_MPU=1 MEMORY_MODEL=1
	freertos_c_include = targets/light
	freertos_static_lib = libfreertos

Ideally, producing an executable from libraries that are built from source
would be identical (or as close as possible) to producing the same executable
from an installed (pre-built) version of the library.

One way an executable might be built with dependencies:

	main_source = main.cpp
	main_name = main
	main_lib = freertos
	main_exe_memory_map = main_flash_binary.ldf

A file system example for this might be:

	/opt/mylib
		include
			freertos
			core
			crypto
		bin
			armv5.gcc
			armv5.gcc-debug
		lib
			armv5.gcc
					libfreertos.a
					libcore.a
					libcrypto.a
			armv5.gcc-debug
					libcore.a
					libcrypto.a

Object File Organization
========================

Object files may be built in place with the source tree or may be built out of
place in a separate `OBJ_DIR`. If compiled in place, the convention is

In-place object file naming convention:

	<SOURCE_PATH>/<SOURCE_BASENAME>.<MOLD_TARGET>.<MOLD_ARCH>.<MOLD_TOOL>.<MOLD_OBJ_EXT>

Out-of-place object file naming convention:

	<MOLD_OBJDIR>/<SOURCE_PATH>/<SOURCE_BASENAME>.<MOLD_TARGET>.<MOLD_ARCH>.<MOLD_TOOL>.<MOLD_OBJ_EXT>

Object file examples:

	my_file.core_bare.arm-cm3.gcc-debug.o
	my_file.core_bare.arm-cm3.llvm-debug.o
	my_file.core_bare.host.gcc.o
	my_file.core_freertos.cortex-m3.gcc-release.o

<commit_msg>Rework with spore concept for target management<commit_after># gnumake-molds
Molds for building software with GNU make

TODO: Consider name change from mold to yeast or possibly hyphae. Which one
would be a better analogy for compiling and linking software?

Goals
=====

The goal of gnumake-mold is to make it easy to create cross-compiled libraries
and executables for a variety of architectures using only GNU make.

Highlights:

- manage multiple compilers, multiple build types, and multiple architectures
- streamline library compilation, linking, and include path management
- scale for large code bases that bring together multiple libraries

There are a variety of package management systems and root filesystem creation
tools available today that will build large systems from source code.  This is
not the focus of gnumake-mold. Instead, the focus is on creating complete,
fully linked executables that are compiled completely from source code, or that
are statically linked against a number of libraries that are also compiled from
source.

Features
========

Cross-compiling
---------------

- Cross-compile embedded applications for different architectures with a single Makefile
- Out-of-the-box support for gcc and clang x86/ARM
- Create libaraies with different complier options (e.g. ARM vs Thumb code)

Target generation
-----------------

- Build the same source file with different compiler settings
- Create multiple executable and library target outputs with ease
- Separate definition of program source and structure from creating rules

Large projects
--------------

- Manage large projects with multiple "sub-components"
- Combine components from different parts of a filesystem (re-use installed library source in multiple projects)
- Create release directory structures with libraries, binaries, and includes
- Build against libraries either from source or from released package

Languages
---------

The initial vision is to support core languages in the GCC/LLVM toolchains:

- C
- C++
- Objective C

A longer term vision might be to expand this support to include languages such as

- Rust
- Swift
- Go

Usability
---------

- Tab auto-completion works with targets defined by Mold
- Makes it easier to debug by dumping full commands with options
- Unit testable for validation and performance testing
- Work with in-place object files for the smallest projects
- Support for automatic dependency generation

Key Concepts
============

Toolchain
---------

A toolchain is a set of tools that are used to translate source code into binary form for execution on a specific architecture.

Code generated by one toolchain cannot be linked or combined with code generated for another toolchain.

Examples: gcc, clang, gcc-debug, clang-release, clang-debug.

While a toolchain must include all tools required to produce an executable binary or library output, the name of the toolchain is generally based upon the compiler.

Generally, debug versus release build variants are handled at the toolchain level due to the significant impact that may be present on generated code.

- `MOLD_TOOLS`: All available toolchains
- `MOLD_TOOL`: Current toolchain

Architecture
------------

A build architecture is defined by a specific CPU type and associated instruction set.

Examples: host, amd64, cortex-m4, cortex-r7, avr, mipsel, etc.

Code generated for different architectures does not interoperate and cannot be linked together.

- `MOLD_ARCHS`: All available architectures for current toolchain
- `MOLD_ARCH`: Current architecture for current toolchain

Good discussion on architecture naming here: http://clang.llvm.org/docs/CrossCompilation.html

Targets
-------

Three different types of target outputs are currently supported:

- Executable: a fully-linked binary executable
- Library: a library (either static or dynamic)
- Headers: a set of header files for stand-alone use or use with a library

Implementation
==============

All mold targets are generated from small units called spores. Spores may
produce one or more target output.

Spores
------

Mold manages a list of all spores through the variable `MOLD_SPORES`. If a new
spore is created, it must be added to this list.

All spores share the following variables:

- `<spore>_name` - base name used as part of output files
- `<spore>_depends` - other spores required as dependencies of this spore

Dependencies among spores is captured entirely through the above structure.
This structure is managed through a series of hidden touch files that are
created by mold.

Each spore has a number of variables that are specific to that spore and are
used during the build process.

Spore Generation
----------------

- `<spore>_exe` - name of executable output target
- `<spore>_lib` - name of library output target
- `<spore>_src` - source files used to generate executable and/or library
- `<spore>_include` - list of header files to stage for system include

All targets are placed in a target staging area once created. This keeps the
directory structure organized and enables cleaner integration when multiple
libraries are involved.

Target outputs are staged in the following directories:

	lib -> MOLD_LIB_DIR (e.g. $PREFIX/lib)
	include -> MOLD_HEADER_DIR (e.g. $PREFIX/include)
	exe -> MOLD_EXE_DIR (e.g. $PREFIX/bin)

Headers that are staged into `MOLD_HEADER_DIR` are automatically included as part
of the system include path. Libraries that are staged in `MOLD_LIB_DIR` are
included as part of the library search path. This means that using libraries
created with mold is identical to using libraries installed in the development
system.

Libraries
---------

TODO

Headers
-------

TODO

Executables
-----------

Executable targets make use of the following spore variables:

	<spore>_c_defines - preprocessor defines for C compiler
	<spore>_c_include - additional include paths for C compiler

	<spore>_cpp_defines - preprocessor defines for CPP compiler
	<spore>_cpp_include - additional include paths for CPP compiler

	<spore>_static_lib - static libraries linked into exe
	<spore>_dynamic_lib - dynamic libraries linked against exe

The libraries listed for linking must be part of the library search path
defined by either the tool chain or staged by a previous library target.

Embedded platforms may make use of some additional special targets:

	<spore>_memory_map - guidance for linker on how to place symbols
	<spore>_raw_image - raw read only code for execution out of flash
	<spore>_hex_image - hex output of raw flash image

Examples
--------

Example of a static library:

	freertos_headers = $(freertos_home)/include
	freertos_headers_prefix = freertos
	freertos_source = $(freertos_home)/src/*
	freertos_c_defines = ENABLE_MPU=1 MEMORY_MODEL=1
	freertos_c_include = targets/light
	freertos_static_lib = libfreertos

Ideally, producing an executable from libraries that are built from source
would be identical (or as close as possible) to producing the same executable
from an installed (pre-built) version of the library.

One way an executable might be built with dependencies:

	main_source = main.cpp
	main_name = main
	main_lib = freertos
	main_exe_memory_map = main_flash_binary.ldf

A file system example for this might be:

	/opt/mylib
		include
			freertos
			core
			crypto
		bin
			armv5.gcc
			armv5.gcc-debug
		lib
			armv5.gcc
					libfreertos.a
					libcore.a
					libcrypto.a
			armv5.gcc-debug
					libcore.a
					libcrypto.a

Object File Organization
========================

Object files may be built in place with the source tree or may be built out of
place in a separate `OBJ_DIR`. If compiled in place, the convention is

In-place object file naming convention:

	<SOURCE_PATH>/<SOURCE_BASENAME>.<MOLD_TARGET>.<MOLD_ARCH>.<MOLD_TOOL>.<MOLD_OBJ_EXT>

Out-of-place object file naming convention:

	<MOLD_OBJDIR>/<SOURCE_PATH>/<SOURCE_BASENAME>.<MOLD_TARGET>.<MOLD_ARCH>.<MOLD_TOOL>.<MOLD_OBJ_EXT>

Object file examples:

	my_file.core_bare.arm-cm3.gcc-debug.o
	my_file.core_bare.arm-cm3.llvm-debug.o
	my_file.core_bare.host.gcc.o
	my_file.core_freertos.cortex-m3.gcc-release.o


{'fraction_non_alphanumeric': 0.06769556865602437, 'fraction_numerical': 0.0019871497648539446, 'mean_word_length': 4.08521387672617, 'pattern_counts': {'":': 0, '<': 47, '<?xml version=': 0, '>': 53, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6452062', 'n_tokens_mistral': 4274, 'n_tokens_neox': 3976, 'n_words': 1913}