<commit_before># scala-lex
Scala lexical analysis without parsing literals and identifiers

## Installation

#### Requirements

* CMake 3.6+
* C++ Compiler with C++14 support (GCC 6+, Clang 3.4+)

#### Compilation

This project uses cmake as its build system. CMake automatically generates build files for target platform. If you have POSIX compatible system chances are you are going to use make. Below are the steps to compile program on POSIX system:
1. In project root, create "build" directory. Here all build files and resulting binaries are going to be located.
```
mkdir build
cd build
```

2. Now we need to generate build files. For this CMake needs file name "CMakeLists.txt". It is located in src/ directory, so to generate build files:
```
cmake ../src
```

3. After CMake has finished succesfully, build files are generated. This means that we can use whatever build system we have generated files for. By default CMake generates build files for make. So to build a final executable:
```
make
```
4. Now you have "scala-lex" exeсutable in your directory. You can run it as usual.


## Contribute
<commit_msg>Updated readme to show compiler version update<commit_after># scala-lex
Scala lexical analysis without parsing literals and identifiers

## Installation

#### Requirements

* CMake 3.6+
* C++ Compiler with C++17 support.(At least string_view)

#### Compilation

This project uses cmake as its build system. CMake automatically generates build files for target platform. If you have POSIX compatible system chances are you are going to use make. Below are the steps to compile program on POSIX system:
1. In project root, create "build" directory. Here all build files and resulting binaries are going to be located.
```
mkdir build
cd build
```

2. Now we need to generate build files. For this CMake needs file name "CMakeLists.txt". It is located in src/ directory, so to generate build files:
```
cmake ../src
```

3. After CMake has finished succesfully, build files are generated. This means that we can use whatever build system we have generated files for. By default CMake generates build files for make. So to build a final executable:
```
make
```
4. Now you have "scala-lex" exeсutable in your directory. You can run it as usual.


## Contribute

{'fraction_non_alphanumeric': 0.07152259763053971, 'fraction_numerical': 0.008336989907854322, 'mean_word_length': 4.217391304347826, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1709679', 'n_tokens_mistral': 640, 'n_tokens_neox': 590, 'n_words': 344}