<commit_before># Google Protocol Buffers
Google's protocol buffers are used in this project for serialization of data. Protocol buffers are used because they work in multiple languages, and have binary compatibilty between the different languages. Syntax version 2 of the protocol buffer protocol will be used for compatibility with the Unity Game Engine.

- [Unity](#unity)
  - [Setup](#unity-setup)
  - [Compiling](#compiling-to-c-code)
- [C++](#c-code)
  - [Setup](#c-setup)
  - [Compiling](#compiling-to-c-code-1)

## Unity 
Unity uses the mono framework for C# scripting. The used version does not contain all the .NET features used in the C# version of protocol buffers. This is why [protobuf-net](https://github.com/mgravell/protobuf-net) (version r668) is used.

### Unity setup
The protobuf-net library is already included in the Unity project. To generate C# code from .proto files you will need the protogen tool, which can be found in the [protobuf-net binary release](https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/protobuf-net/protobuf-net%20r668.zip). This folder contains a ProtoGen folder with an executable. You might want to add this directory to your PATH if you want to use the Protogen tool from the command line.

### Compiling to C# code
When the ProtoGen tool is in your PATH it can be used to generate C# code from .proto files:
```
protogen -i:input.proto -o:output.cs
```
Now output.cs can be used inside the Unity project

## C++ Code
### C++ Setup
### Compiling to C++ code<commit_msg>Added generation script to proto readme.<commit_after># Google Protocol Buffers
Google's protocol buffers are used in this project for serialization of data. Protocol buffers are used because they work in multiple languages, and have binary compatibilty between the different languages. Syntax version 2 of the protocol buffer protocol will be used for compatibility with the Unity Game Engine.

- [Unity](#unity)
  - [Setup](#unity-setup)
  - [Compiling](#compiling-to-c-code)
- [C++](#c-code)
  - [Setup](#c-setup)
  - [Compiling](#compiling-to-c-code-1)

## Unity 
Unity uses the mono framework for C# scripting. The used version does not contain all the .NET features used in the C# version of protocol buffers. This is why [protobuf-net](https://github.com/mgravell/protobuf-net) (version r668) is used.

### Unity setup
The protobuf-net library is already included in the Unity project. To generate C# code from .proto files you will need the protogen tool, which can be found in the [protobuf-net binary release](https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/protobuf-net/protobuf-net%20r668.zip). This folder contains a ProtoGen folder with an executable. You might want to add this directory to your PATH if you want to use the Protogen tool from the command line.

### Compiling to C# code
When the ProtoGen tool is in your PATH it can be used to generate C# code from .proto files:
```
protogen -i:input.proto -o:output.cs
```
Now output.cs can be used inside the Unity project
The `generate_for_unity.bat` script can also be used. This script will automatically generate the C# code inside the Unity folder.

## C++ Code
### C++ Setup
### Compiling to C++ code
{'fraction_non_alphanumeric': 0.09311616472034419, 'fraction_numerical': 0.006760909649661954, 'mean_word_length': 4.380165289256198, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2305968', 'n_tokens_mistral': 937, 'n_tokens_neox': 879, 'n_words': 425}