<commit_before># CUHKSZ C++ Library

[![Build Status](https://travis-ci.org/cuhkshenzhen/CUHKSZLib.svg?branch=master)](https://travis-ci.org/cuhkshenzhen/CUHKSZLib)

## Requirements
1. cmake >= 2.8
2. make
3. gcc/clang/msvc(2015+)

## Usage
### Compile
#### macOS and Linux
First clone the repository:
```
git clone --recursive https://github.com/cuhkshenzhen/CUHKSZLib.git
```
Then configure and compile:
```
mkdir build && cd build
cmake ..
make -j4
```
#### Windows (MSVC)
1. Download and install `cmake` from https://cmake.org/download/, make sure to choose the windows installer.
2. Make sure you have installed the Visual Studio with C++ component and it works.
3. Clone or download this repository 
4. Open Cmake GUI, choose this repository as source and `CUHKSZLib/build` as build. (TODO: details and pics to be added)
5. Click `Configure`, and then click `Open Project`
6. Build with Visual Studio.
<commit_msg>Add appveyor status<commit_after># CUHKSZ C++ Library

[![Build Status](https://travis-ci.org/cuhkshenzhen/CUHKSZLib.svg?branch=master)](https://travis-ci.org/cuhkshenzhen/CUHKSZLib)
[![Build status](https://ci.appveyor.com/api/projects/status/634a6wrw9xheiu0e/branch/master?svg=true)](https://ci.appveyor.com/project/alesiong/cuhkszlib-cmxvy/branch/master)

## Requirements
1. cmake >= 2.8
2. make
3. gcc/clang/msvc(2015+)

## Usage
### Compile
#### macOS and Linux
First clone the repository:
```
git clone --recursive https://github.com/cuhkshenzhen/CUHKSZLib.git
```
Then configure and compile:
```
mkdir build && cd build
cmake ..
make -j4
```
#### Windows (MSVC)
1. Download and install `cmake` from https://cmake.org/download/, make sure to choose the windows installer.
2. Make sure you have installed the Visual Studio with C++ component and it works.
3. Clone or download this repository 
4. Open Cmake GUI, choose this repository as source and `CUHKSZLib/build` as build. (TODO: details and pics to be added)
5. Click `Configure`, and then click `Open Project`
6. Build with Visual Studio.

{'fraction_non_alphanumeric': 0.13607125185551708, 'fraction_numerical': 0.01880257298367145, 'mean_word_length': 4.53972602739726, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 10, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5651205', 'n_tokens_mistral': 726, 'n_tokens_neox': 688, 'n_words': 220}