<commit_before>Some information about how to debug/build this lexer/parser
===========================================================
I wrote this file to remember how I am doing to debug the parser: how to build
it "by hand" using ocamlbuild, how understand the automaton behaviour... I hope
it will help!

## Build and install ##
There are two ways of building: with with pre-generated setup.ml (and
./configure and Makefile) or directly with the more "low-level" ocamlbuild.

1. Using setup.ml. You basically have to make and make install. The
`make install` step is not mandatory as it will only move the compiled touist
to the place you indicated in --bindir.

        ./configure --enable-debug --bindir . --override is_native false
        make
        make install

2. Using ocamlbuild (which is used by the 1). See section [Run the debugger](#run-the-debugger).


## Running touist ##
After having make & make install using `./configure --bindir .`, you should be
able to run `./touist`:

    echo '$a' | ./touist - --sat


## Edit how the build is made ##
If you want to modify how the build is made, you must know that the files
`Makefile` and `setup.ml` are generated by `oasis setup`. Oasis reads the
file `_oasis`. Using oasis should only happen when `_oasis` is modified.
Whenever you want to regenerate the build scripts:

    oasis setup


## Run the debugger ##

If you want to run ocamldebug, you must build touist as a .byte instead of
.native. So I propose to use directly ocamlbuild instead of what oasis
has generated (`make`). You also need to add to the ocamlbuild command:
- the option `-tag debug` -> will generate the debug symbols for ocamldebug
- the option `-menhir "--trace"` -> will show what the automaton does.

Here is the full command (you must be in `src/`):

    ocamlbuild -use-ocamlfind -use-menhir -menhir "menhir --trace --table --inspection \
    -v -la 2" -package fileutils,str,menhirLib touist.byte -tag debug

Now with the debugger. Note that I installed `rlwrap` to be able to use the
arrows to go through the command history.

    rlwrap ocamldebug touist.byte --sat test/atleast.touistl

The commands I used:
- `r` to run, `f` to finish and be able to re-run
- `break @eval 366` to set a breakpoint on src/eval.ml at line 366
- `s` to step forward when you reached a breakpoint
- `b` or `backstep` to do a backward step when you what to understand what
  happened just before the crash/exception



The syntax errors in parser.messages
====================================
One of the new features of the touist compiler is to be able to give real
and understandable syntax errors (when you forget to close a parenthesis...).
To do so, we use a feature of menhir that allows to pick a message among a
list of hand-written error messages. The thing is that there are more than
180 of these messages...

So developers can improve the error syntax messages by editing
`parser.messages` and then re-generating `parser_messages.ml`.


## Re-building `parser_messages.ml` using `parser.messages` ##

I wrote the makefile TopMakefile to do the whole thing. You can just run

    make -f TopMakefile
    make -f TopMakefile clean

and everything will be compiled. But if you want to do it by hand:

Most of the time, the only command you want to use is the 3.

1. To generate the `parser.messages` for the first time, use:

        menhir parser.mly --list-errors > parser.messages

2. To update the existing `parser.messages` whenever you modify the `parser.mly`:

        menhir parser.mly --update-errors parser.messages > tmp && mv tmp parser.messages

3. To regenerate the final file `parser_messages.ml` whenever you edit the error
messages in `parser.messages`:

        menhir --compile-errors parser.messages parser.mly > parser_messages.ml

4. And to recompile parser_messages.ml and then touist:

        menhir --compile-errors parser.messages parser.mly > parser_messages.ml && ocamlbuild -use-ocamlfind -use-menhir -menhir "menhir --trace --table --inspection -v -la 2" -package menhirLib -package fileutils,str touist.byte -tag debug -r

## Missing error cases in parser.messages
When updating parser.mly, you might sometimes create new error states
that do not appear in your already-written parser.messages.
This often leads to the error message:
```
1:14: syntax error after ',' and before 'a'.
This is an unknown syntax error (92).
Please report this problem to the compiler vendor.
```
meaning that the state 92 isn't in parser_messages.ml, and thus not in
parser.messages. To fix that, anytime you modify parser.mly, check that
no new errors have been intruduced:

```
menhir --list-errors src/parser.mly > parser.messages_updated
menhir --compare-errors parser.messages_updated --compare-errors src/parser.messages --list-errors src/parser.mly
```

## Testing your hand-written messages ##
There are two ways to test if the messages actually work. The first one
is simply to launch `./touist` with a wrong syntax touistl file. If you built
using ocamlbuild and the option --trace in -menhir "" block, it will show you
the steps of the automaton. It is REALLY helpful to understand why an expression
is not parsed as expected. For example (WARNING: this is the old syntax with
begin formula end formula):

```
./touist.byte --sat /dev/stdin -o /dev/stdout --table /dev/stdout << eof
begin
eof

State 0:
Lookahead token is now BEGIN (0-5)
Shifting (BEGIN) to state 1
State 1:
Lookahead token is now EOF (7-7)
Initiating error handling
/dev/stdin:2:1: syntax error after 'begin' and before ''.
You must specify (at least) a "formula" bloc,
or a "sets" bloc followed by a "formula" bloc.
Example:
    begin sets    ... end sets
    begin formula ... end formula
Debug: Automaton state: 1 (see src/parser.messages)
```

The second way is to focus on tokens (the capital letters words like BEGIN)
and to ask menhir to see if this sequence of tokens triggers an error:

```
menhir --interpret-error parser.mly << eof
BEGIN FORMULA ATLEAST LPAREN VAR COMMA RPAREN
eof
```


Prototyping ocaml functions using toplevel
==========================================
*toplevel* = the interpreter that prompts when you launch `ocaml` with no file.

Just launch utop. I wrote a .ocamlinit that should load everything fine.
Just make sure that everything is compiled (with make build) and that
you are compiling in bytecode; to enable that:

    ./configure --override is_native false<commit_msg>docs-dev: explain how to show the backtrace when exception is raised<commit_after>Some information about how to debug/build this lexer/parser
===========================================================
I wrote this file to remember how I am doing to debug the parser: how to build
it "by hand" using ocamlbuild, how understand the automaton behaviour... I hope
it will help!

## Build and install ##
There are two ways of building: with with pre-generated setup.ml (and
./configure and Makefile) or directly with the more "low-level" ocamlbuild.

1. Using setup.ml. You basically have to make and make install. The
`make install` step is not mandatory as it will only move the compiled touist
to the place you indicated in --bindir.

        ./configure --enable-debug --bindir . --override is_native false
        make
        make install

2. Using ocamlbuild (which is used by the 1). See section [Run the debugger](#run-the-debugger).


## Running touist ##
After having make & make install using `./configure --bindir .`, you should be
able to run `./touist`:

    echo '$a' | ./touist - --sat


## Edit how the build is made ##
If you want to modify how the build is made, you must know that the files
`Makefile` and `setup.ml` are generated by `oasis setup`. Oasis reads the
file `_oasis`. Using oasis should only happen when `_oasis` is modified.
Whenever you want to regenerate the build scripts:

    oasis setup


## Run the debugger ##

If you want to run ocamldebug, you must build touist as a .byte instead of
.native. So I propose to use directly ocamlbuild instead of what oasis
has generated (`make`). You also need to add to the ocamlbuild command:
- the option `-tag debug` -> will generate the debug symbols for ocamldebug
- the option `-menhir "--trace"` -> will show what the automaton does.

Here is the full command (you must be in `src/`):

    ocamlbuild -use-ocamlfind -use-menhir -menhir "menhir --trace --table --inspection \
    -v -la 2" -package fileutils,str,menhirLib touist.byte -tag debug

Now with the debugger. Note that I installed `rlwrap` to be able to use the
arrows to go through the command history.

    rlwrap ocamldebug touist.byte --sat test/atleast.touistl

The commands I used:
- `r` to run, `f` to finish and be able to re-run
- `break @eval 366` to set a breakpoint on src/eval.ml at line 366
- `s` to step forward when you reached a breakpoint
- `b` or `backstep` to do a backward step when you what to understand what
  happened just before the crash/exception



The syntax errors in parser.messages
====================================
One of the new features of the touist compiler is to be able to give real
and understandable syntax errors (when you forget to close a parenthesis...).
To do so, we use a feature of menhir that allows to pick a message among a
list of hand-written error messages. The thing is that there are more than
180 of these messages...

So developers can improve the error syntax messages by editing
`parser.messages` and then re-generating `parser_messages.ml`.


## Re-building `parser_messages.ml` using `parser.messages` ##

I wrote the makefile TopMakefile to do the whole thing. You can just run

    make -f TopMakefile
    make -f TopMakefile clean

and everything will be compiled. But if you want to do it by hand:

Most of the time, the only command you want to use is the 3.

1. To generate the `parser.messages` for the first time, use:

        menhir parser.mly --list-errors > parser.messages

2. To update the existing `parser.messages` whenever you modify the `parser.mly`:

        menhir parser.mly --update-errors parser.messages > tmp && mv tmp parser.messages

3. To regenerate the final file `parser_messages.ml` whenever you edit the error
messages in `parser.messages`:

        menhir --compile-errors parser.messages parser.mly > parser_messages.ml

4. And to recompile parser_messages.ml and then touist:

        menhir --compile-errors parser.messages parser.mly > parser_messages.ml && ocamlbuild -use-ocamlfind -use-menhir -menhir "menhir --trace --table --inspection -v -la 2" -package menhirLib -package fileutils,str touist.byte -tag debug -r

## Missing error cases in parser.messages
When updating parser.mly, you might sometimes create new error states
that do not appear in your already-written parser.messages.
This often leads to the error message:
```
1:14: syntax error after ',' and before 'a'.
This is an unknown syntax error (92).
Please report this problem to the compiler vendor.
```
meaning that the state 92 isn't in parser_messages.ml, and thus not in
parser.messages. To fix that, anytime you modify parser.mly, check that
no new errors have been intruduced:

```
menhir --list-errors src/parser.mly > parser.messages_updated
menhir --compare-errors parser.messages_updated --compare-errors src/parser.messages --list-errors src/parser.mly
```

## Testing your hand-written messages ##
There are two ways to test if the messages actually work. The first one
is simply to launch `./touist` with a wrong syntax touistl file. If you built
using ocamlbuild and the option --trace in -menhir "" block, it will show you
the steps of the automaton. It is REALLY helpful to understand why an expression
is not parsed as expected. For example (WARNING: this is the old syntax with
begin formula end formula):

```
./touist.byte --sat /dev/stdin -o /dev/stdout --table /dev/stdout << eof
begin
eof

State 0:
Lookahead token is now BEGIN (0-5)
Shifting (BEGIN) to state 1
State 1:
Lookahead token is now EOF (7-7)
Initiating error handling
/dev/stdin:2:1: syntax error after 'begin' and before ''.
You must specify (at least) a "formula" bloc,
or a "sets" bloc followed by a "formula" bloc.
Example:
    begin sets    ... end sets
    begin formula ... end formula
Debug: Automaton state: 1 (see src/parser.messages)
```

The second way is to focus on tokens (the capital letters words like BEGIN)
and to ask menhir to see if this sequence of tokens triggers an error:

```
menhir --interpret-error parser.mly << eof
BEGIN FORMULA ATLEAST LPAREN VAR COMMA RPAREN
eof
```


Prototyping ocaml functions using toplevel
==========================================
*toplevel* = the interpreter that prompts when you launch `ocaml` with no file.

Just launch utop. I wrote a .ocamlinit that should load everything fine.
Just make sure that everything is compiled (with make build) and that
you are compiling in bytecode; to enable that:

    ./configure --override is_native false

Unexpected exception: where does it come from?
=============================================
Backtrace for exceptions is disabled by default as it needs more computation.
To enable backtracing on exception, you just have to set to 'b' the
OCAMLRUNPARAM env variable:

    OCAMLRUNPARAM=b ./touist.native
{'fraction_non_alphanumeric': 0.09980371432885399, 'fraction_numerical': 0.005435603200966329, 'mean_word_length': 3.7344531808434596, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7012351', 'n_tokens_mistral': 3808, 'n_tokens_neox': 3642, 'n_words': 1884}