<commit_before>Espruino Internals: Build Process
---------------------------------

Directories and Files
-------------------

* `ChangeLog`:          What's new
* `TODO`:               List of things to do
* `boards/`:            Information on boards, used to auto-generate a lot of the code
* `gen/`:               Auto-Generated Source Files
* `libs/`:              Optional libraries to include in Espruino (Math, Filesystem, Graphics, etc)
* `misc/`:              Other useful things
* `scripts/`:           Scripts for generating files in gen, and for analysing code/compilation/etc
* `src/`:               Main source code
* `targetlibs/`:        Libraries for targeted architectures
* `targets/`:           Specific code for targeted architectures
* `tests/`:             JavaScript Testcases
* `benchmark/`:         JavaScript Benchmarks
* `dist_*`:             files to be copied into distribution zip file
* `Makefile`:           the Makefile which controls the build process

Build Commands
--------------

Building for an Espruino board is generally done by typing:

```
BOARDNAME=1 make
```

And for release, by typing:

```
RELEASE=1 BOARDNAME=1 make
```

Valid board names are in comments at the head of the [`Makefile`](https://github.com/espruino/Espruino/blob/master/Makefile)..

Release builds have all the assertions removed, saving space. Normal builds still have the assertions, and debug builds (with `DEBUG=1` have more debug info in the assertions, as well as the debug info needed for proper GDB debugging).

Build Process
-------------

* Most build logic is in the [`Makefile`](https://github.com/espruino/Espruino/blob/master/Makefile).
* An `ifdef BOARDNAME` in the Makefile sets up some specific build options, as well as what gets included (eg. `USE_NET=1`). The `BOARD` variable references a [Board Definition File](#boarddefinition) in `boards/`.
* `boards/$BOARD.py` is read, and various information (chip family, part number, binary name, whether it has a bootloader) is read out into variables in the Makefile. **Note:** in the future more of the build options should be inferred from the `BOARD.py` file,
* The `Makefile` adds source files to `$SOURCES`, and 'wrapper' source files to `$WRAPPERSOURCES`. [Wrapper files](#wrapperfiles) are files that contain functions that are exposed to JS.
* The script [`scripts/build_platform_config.py`](scripts/build_platform_config.py) is run which generates `gen/platform_config.h` from `boards/$BOARD.py` - this contains information like the amount of RAM, as well as buffer sizes and the amount of variables that will be stored.
* The script [`scripts/build_pininfo.py`](scripts/build_pininfo.py) creates [the pin definitions](#pindefinitions) and puts them in `gen/jspininfo.c`
* The script [`scripts/build_jswrapper.py`](scripts/build_jswrapper.py) is then run on `$WRAPPERSOURCES` - it generates `gen/jswrapper.c` - a hard-coded symbol table of built-in functions - from the [Wrapper files](#wrapperfiles).
* Source files are built
* Everything is linked using link time optimisation (where possible) - this helps to inline code that wouldn't otherwise have been inlined, and generally makes for a much more efficient binary.
* The `elf` file is converted with `objdump` to:
  * `bin` (for uploading to Espruino board)
  * `hex` (for uploading to mbed/etc)
  * `lst` (assembler listing - used for debugging)
* [`scripts/check_size.sh`](scripts/check_size.sh) does a sanity check of the `bin` file's size against what's described in the [board definition](#boarddefinition), and fails if it won't fit into available flash memory.

Oddities
--------

* `malloc` and `free` are not included, because they're not needed and by default they use a large pool of valuable RAM to speed up memory allocation.
* `strcpy`, `memcpy`, `memset` are all reimplemented in [`jsutils.h`](src/jsutils.h) - again because of initial issues with high flash memory usage
* Espruino on STM32 uses its own [Maths library](lib/maths) for double arithmetic, because when building initially the libraries included by GCC dynamically allocated memory - which pulled in `malloc`. If we could move back to GCC libraries at some point it'd be a huge bonus


Board Definition File  <a name="boarddefinition">
-------------------------------------------------

The board definition files are `.py` files that reside in the [`boards`](boards) folder (for example [the Pico one](boards/PICO_R1_3.py)). They're used all over the place:

* To work out the binary's name
* To help choose what gets built in the Makefile
* To create `gen/platform_config.h` via [`scripts/build_playform_config.py`](scripts/build_playform_config.py)
* To create [the pin definitions](#pindefinitions)
* To create the HTML page on each individual board (referenced from [the top of the Reference](http://www.espruino.com/Reference)) using [`scripts/build_board_docs.py`](scripts/build_board_docs.py)
* Converted to JSON (along with the pin declarations) and put [on espruino.com](http://www.espruino.com/json/PICO_R1_3.json) to help with autocomplete in the Web IDE (and auto-population of dropdowns in the Blockly editor). Converted by [`scripts/build_board_json.py`](scripts/build_board_json.py)

These contain:

### info

* Board name and link (for the HTML file)
* Default console device, pins, and baus rate (the device Espruino goes to when USB is unplugged or not built in)
* `variables` - The number of variables to use (this depends on the amount of RAM available). Less than 1023 vars use 12 bytes per var, more uses 16 bytes. You have to adjust this such that there is spare room for the stack and static variables (on Espruino boards this means leaving around 16kB free, but on smaller boards it can be reduced a lot)
* `bootloader` - whether the binary image needs compiling with a special USB-VCP bootloader (Espruino boards only)
* `binary_name` - the name of the binary that'll be produced
* `binaries` - available binaries - this is used by the Web IDE to allow the user to choose which binary to upload

### chip

* `part` - Chip part number (this is defined in the compiler - eg if the part is `STM32F401CDU6`, `-DSTM32F401CDU6` is put on the GCC command-line)<commit_msg>incl linker<commit_after>Espruino Internals: Build Process
---------------------------------

Directories and Files
-------------------

* `ChangeLog`:          What's new
* `TODO`:               List of things to do
* `boards/`:            Information on boards, used to auto-generate a lot of the code
* `gen/`:               Auto-Generated Source Files
* `libs/`:              Optional libraries to include in Espruino (Math, Filesystem, Graphics, etc)
* `misc/`:              Other useful things
* `scripts/`:           Scripts for generating files in gen, and for analysing code/compilation/etc
* `src/`:               Main source code
* `targetlibs/`:        Libraries for targeted architectures
* `targets/`:           Specific code for targeted architectures
* `tests/`:             JavaScript Testcases
* `benchmark/`:         JavaScript Benchmarks
* `dist_*`:             files to be copied into distribution zip file
* `Makefile`:           the Makefile which controls the build process

Build Commands
--------------

Building for an Espruino board is generally done by typing:

```
BOARDNAME=1 make
```

And for release, by typing:

```
RELEASE=1 BOARDNAME=1 make
```

Valid board names are in comments at the head of the [`Makefile`](https://github.com/espruino/Espruino/blob/master/Makefile)..

Release builds have all the assertions removed, saving space. Normal builds still have the assertions, and debug builds (with `DEBUG=1` have more debug info in the assertions, as well as the debug info needed for proper GDB debugging).

The Make system has the following targets:

* `clean` - Clean the build
* `all` (or unspecified) - Build, including binaries needed for flashing the firmware
* `flash` - Generate the firmware files and flash to the board
* `serialflash` - Generate the firmware files and flash to the board via serial bootloader (used for STM32, especially the Original Espruino and Espruino Pico)

Build Process
-------------

* Most build logic is in the [`Makefile`](https://github.com/espruino/Espruino/blob/master/Makefile).
* An `ifdef BOARDNAME` in the Makefile sets up some specific build options, as well as what gets included (eg. `USE_NET=1`). The `BOARD` variable references a [Board Definition File](#boarddefinition) in `boards/`.
* `boards/$BOARD.py` is read, and various information (chip family, part number, binary name, whether it has a bootloader) is read out into variables in the Makefile. **Note:** in the future more of the build options should be inferred from the `BOARD.py` file,
* The `Makefile` adds source files to `$SOURCES`, and 'wrapper' source files to `$WRAPPERSOURCES`. [Wrapper files](#wrapperfiles) are files that contain functions that are exposed to JS.
* The script [`scripts/build_platform_config.py`](scripts/build_platform_config.py) is run which generates `gen/platform_config.h` from `boards/$BOARD.py` - this contains information like the amount of RAM, as well as buffer sizes and the amount of variables that will be stored.
* The script [`scripts/build_pininfo.py`](scripts/build_pininfo.py) creates [the pin definitions](#pindefinitions) and puts them in `gen/jspininfo.c`
* The script [`scripts/build_jswrapper.py`](scripts/build_jswrapper.py) is then run on `$WRAPPERSOURCES` - it generates `gen/jswrapper.c` - a hard-coded symbol table of built-in functions - from the [Wrapper files](#wrapperfiles).
* Source files are built
* On embedded targets, a linker file `gen/linker.ld` is auto-generated by [`scripts/build_linker.py`](scripts/build_linker.py) based on the [board definition](#boarddefinition) (available flash, ram, bootloader etc). In some targets (especially non-STM32) a pre-made linker file will be used instead.
* Everything is linked using link time optimisation (where possible) - this helps to inline code that wouldn't otherwise have been inlined, and generally makes for a much more efficient binary.
* The `elf` file is converted with `objdump` to:
  * `bin` (for uploading to Espruino board)
  * `hex` (for uploading to mbed/etc)
  * `lst` (assembler listing - used for debugging)
* [`scripts/check_size.sh`](scripts/check_size.sh) does a sanity check of the `bin` file's size against what's described in the [board definition](#boarddefinition), and fails if it won't fit into available flash memory.

Oddities
--------

* `malloc` and `free` are not included, because they're not needed and by default they use a large pool of valuable RAM to speed up memory allocation.
* `strcpy`, `memcpy`, `memset` are all reimplemented in [`jsutils.h`](src/jsutils.h) - again because of initial issues with high flash memory usage
* Espruino on STM32 uses its own [Maths library](lib/maths) for double arithmetic, because when building initially the libraries included by GCC dynamically allocated memory - which pulled in `malloc`. If we could move back to GCC libraries at some point it'd be a huge bonus


Board Definition File  <a name="boarddefinition">
-------------------------------------------------

The board definition files are `.py` files that reside in the [`boards`](boards) folder (for example [the Pico one](boards/PICO_R1_3.py)). They're used all over the place:

* To work out the binary's name
* To help choose what gets built in the Makefile
* To create `gen/platform_config.h` via [`scripts/build_playform_config.py`](scripts/build_playform_config.py)
* To generate the linker file `gen/linker.ld` using [`scripts/build_linker.py`](scripts/build_linker.py). Mainly STM32-based targets only.
* To create [the pin definitions](#pindefinitions)
* To create the HTML page on each individual board (referenced from [the top of the Reference](http://www.espruino.com/Reference)) using [`scripts/build_board_docs.py`](scripts/build_board_docs.py)
* Converted to JSON (along with the pin declarations) and put [on espruino.com](http://www.espruino.com/json/PICO_R1_3.json) to help with autocomplete in the Web IDE (and auto-population of dropdowns in the Blockly editor). Converted by [`scripts/build_board_json.py`](scripts/build_board_json.py)

These contain:

### info

* Board name and link (for the HTML file)
* Default console device, pins, and baus rate (the device Espruino goes to when USB is unplugged or not built in)
* `variables` - The number of variables to use (this depends on the amount of RAM available). Less than 1023 vars use 12 bytes per var, more uses 16 bytes. You have to adjust this such that there is spare room for the stack and static variables (on Espruino boards this means leaving around 16kB free, but on smaller boards it can be reduced a lot)
* `bootloader` - whether the binary image needs compiling with a special USB-VCP bootloader (Espruino boards only)
* `binary_name` - the name of the binary that'll be produced
* `binaries` - available binaries - this is used by the Web IDE to allow the user to choose which binary to upload

### chip

* `part` - Chip part number (this is defined in the compiler - eg if the part is `STM32F401CDU6`, `-DSTM32F401CDU6` is put on the GCC command-line)
{'fraction_non_alphanumeric': 0.10644236572118286, 'fraction_numerical': 0.005431502715751358, 'mean_word_length': 3.822480902146235, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 4, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5862740', 'n_tokens_mistral': 3728, 'n_tokens_neox': 3532, 'n_words': 1657}