<commit_before>= Moss Implementation Considerations

This document is a journal of implementation considerations, tradeoffs, and decisions encountered during the implementation of Moss.

== Structure Definition

A significant challenge in writing large makefiles is structuring and scoping the large number of variable definitions required.

One approach to doing this right in make is the use of prefixes to create a namespace convention.

[source,makefile]
----
M.library += util
util.name = utility
util.source = util.c helper.c
util.depends = common
----

This approach leads to significant duplication and requires that the group is 'registered' using a global list (`M.spores` in this case).

Another approach is a more traditional structure definition that can automatically be expanded into individual makefile variables using pattern replacement.

[source,makefile]
----
define library/util
  @name: utility
  @source: util.c helper.c
  @depends: common
endef
----

This eliminates much of the duplication and is fairly readable.
No registration variable is required to register these structures.
They can simply be discovered by name using the path-style prefix.

Error handling may be more difficult or cryptic, since evaluating the definition won't allow a specific line to be cited if make encounters a syntax error.
It may be possible to overcome this issue by expanding each spore definition into a generated makefile that is then included or checked for syntax errors.
Syntax errors in the generated makefile would be easier to and trace back to the original definition.

The most flexible approach is to actually use a domain-specific language to define the structural description of the software.
For exmaple, Lua's table definition and function invocation syntax can be combined to make something that is highly readable.

[source,lua]
----
util = library "utility" {
    source = {
        'util.c',
        'helper.c',
    },
    includes = common
}
----

Some string-to-table helpers could make this even more compact.

[source,lua]
----
util = library "utility" {
    source = [[ util.c helper.c ]],
    includes = common
}
----

This approach is the most promising since the full expressive power of Lua can be harnessed to create a DSL specifically for Moss.
Syntax checking is provided by the Lua compiler itself. Additional checks can be built through careful creation of the DSL definition functions.

== Dependency Generation

The most reliable way to get dependencies right with minimal maintenance is to use the compiler iteself with the same options as an actual build.
Otherwise, there is a risk that preprocessor macros will not be evaluated correctly.

Reference build performance with no dependencies:

	Not parallel: 14.229s
	Parallel (-j4): 0m4.096s

Alternatives:

1. Generate dependencies first, one at a time

Not parallel: 0m21.822s
Parallel (-j4): 0m6.318s

2. Generate dependencies after compile, one at a time
Not parallel: 0m19.663s
Parallel (-j4): 0m6.329s

3. Generate dependencies during compile, one at a time
Not parallel: 0m14.578s
Parallel (-j4): 0m4.217s

3. Generate bulk dependencies for spore first: Not feasable without extra
   post-processing due to the fact that each target needs custom name
<commit_msg>Add YAML concept for definitions<commit_after>= Moss Implementation Considerations

This document is a journal of implementation considerations, tradeoffs, and decisions encountered during the implementation of Moss.

== Table Definition

A significant challenge in writing large makefiles is structuring and scoping the large number of variable definitions required.

One approach to doing this right in make is the use of prefixes to create a namespace convention.

[source,makefile]
----
M.library += util
util.name = utility
util.source = util.c helper.c
util.depends = common
----

This approach leads to significant duplication and requires that the group is 'registered' using a global list (`M.spores` in this case).

Another approach is a more traditional table definition that can automatically be expanded into individual makefile variables using pattern replacement.

[source,makefile]
----
define library/util
  @name: utility
  @source: util.c helper.c
  @depends: common
endef
----

This eliminates much of the duplication and is fairly readable.
No registration variable is required to register these tables.
They can simply be discovered by name using the path-style prefix.

Error handling may be more difficult or cryptic, since evaluating the definition won't allow a specific line to be cited if make encounters a syntax error.
It may be possible to overcome this issue by expanding each spore definition into a generated makefile that is then included or checked for syntax errors.
Syntax errors in the generated makefile would be easier to and trace back to the original definition.

The most flexible approach is to actually use a domain-specific language to define the structural description of the software.
For exmaple, Lua's table definition and function invocation syntax can be combined to make something that is highly readable.

[source,lua]
----
util = library "utility" {
    source = {
        'util.c',
        'helper.c',
    },
    includes = common
}
----

Some string-to-table helpers could make this even more compact.

[source,lua]
----
util = library "utility" {
    source = [[ util.c helper.c ]],
    includes = common
}
----

This approach is the most promising since the full expressive power of Lua can be harnessed to create a DSL specifically for Moss.
Syntax checking is provided by the Lua compiler itself. Additional checks can be built through careful creation of the DSL definition functions.

As an alternative, YAML files could be used as a way to create table and variable definitions with a very compact and readable syntax.
A special gnumake extension (e.g. `$(yaml ...)`) could then be created to load YAML files as namespaced variable definitions.
Table and template definition could then be expressed almost fully in YAML files.
Final table composition for artifact definition would still take place in makefiles.

== Dependency Generation

The most reliable way to get dependencies right with minimal maintenance is to use the compiler iteself with the same options as an actual build.
Otherwise, there is a risk that preprocessor macros will not be evaluated correctly.

Reference build performance with no dependencies:

	Not parallel: 14.229s
	Parallel (-j4): 0m4.096s

Alternatives:

1. Generate dependencies first, one at a time

Not parallel: 0m21.822s
Parallel (-j4): 0m6.318s

2. Generate dependencies after compile, one at a time
Not parallel: 0m19.663s
Parallel (-j4): 0m6.329s

3. Generate dependencies during compile, one at a time
Not parallel: 0m14.578s
Parallel (-j4): 0m4.217s

3. Generate bulk dependencies for spore first: Not feasable without extra
   post-processing due to the fact that each target needs custom name

{'fraction_non_alphanumeric': 0.054104477611940295, 'fraction_numerical': 0.014638346727898967, 'mean_word_length': 4.150776053215077, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7355424', 'n_tokens_mistral': 1787, 'n_tokens_neox': 1677, 'n_words': 969}