<commit_before>
## Lexer

### Goals

Okay, so the first phase of this whole compilation thing is _lexing._ The point is to take the _source code,_ which is just a long string of characters, and try to turn that into a sequence of tokens. _Tokens_ are really just substrings of the source text, and are always classified as a specific _kind_ of token. Having tokens makes the parser's life much, much easier, because it can always just consume one _token_ at a time, instead of one _character_ at a time.. 

As a practical example, the source code below

```
var x = other_var + 3
```

is to be lexed as follows

```
(VarKeyword, "var"), (Whitespace, " "), (Identifier, "x"), 
(Whitespace, " "), (Eq, "="), (Identifier, "other_var"), 
(Whitespace, " "), (Plus, "+"), (Whitespace, " "), 
(Integer, "3")
```

Note that the _lexer,_ the object that turns source code into tokens, has absolutely no notion of what the tokens it produces _mean._ That's up to the parsing and codegen phases. Gibberish like this

```
= + var
```

is perfectly fine from the lexer's perspective, which will produce the following stream of tokens

```
(Eq, "="), (Whitespace, " "), (Plus, "+"), 
(Whitespace, " "), (VarKeyword, "var")
```

Garbage in, garbage out. Finding out that these tokens make no sense in this order, is entirely the parser's problem.

### Implementation

<commit_msg>Updated Lexer.md<commit_after>
## Lexer

### A brief explanation of lexical analysis

The first phase of this whole compilation thing is [lexical analysis](https://en.wikipedia.org/wiki/Lexical_analysis). The plan is to take the _source code,_ which is just a long string of characters, and try to turn that into a sequence of [tokens](https://en.wikipedia.org/wiki/Lexical_analysis#Token). _Tokens_ capture a _string of related characters_ that is also present in the source text, and every token is classified as a specific _kind_ of token.

The primary merit of tokens is that they are useful to the parser, which will always consume one _token_ at a time, rather than one _character_ at a time. For example, when parsing `"some_name = 32"`, it makes perfect sense to consume token `"some_name"` of kind `Identifier` in one step. [Scannerless parsing](https://en.wikipedia.org/wiki/Scannerless_parsing), that is, parsing the identifier character-by-character without a lexical analysis phase, is not unprecedented, but typically complicates the parser's logic.

As a practical example, we'd like to lex the (Objective-Kaleidoscope) source code below

```
var x = other_var + 32
```

as

```
(VarKeyword, "var"), (Whitespace, " "), (Identifier, "x"),
(Whitespace, " "), (Eq, "="), (Identifier, "other_var"),
(Whitespace, " "), (Op, "+"), (Whitespace, " "),
(Integer, "32")
```

Note that the _lexer,_ the object that turns source code into tokens, has absolutely no notion of what the tokens it produces _mean._ That's up to the parsing and codegen phases. Gibberish like this

```
= + var
```

is perfectly fine from the lexer's perspective, which will produce the following stream of tokens

```
(Eq, "="), (Whitespace, " "), (Op, "+"),
(Whitespace, " "), (VarKeyword, "var")
```

Garbage in, garbage out. Finding out that these tokens make no sense in this order, is entirely the parser's problem.

### Implementation
{'fraction_non_alphanumeric': 0.11286887739580165, 'fraction_numerical': 0.002433830240340736, 'mean_word_length': 4.639794168096055, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7409309', 'n_tokens_mistral': 987, 'n_tokens_neox': 938, 'n_words': 448}