<commit_before># Learn Chinese
Simple game to learn how to form Chinese words

The project utilises a java web crawler to get our database of questions.
The words are obtained by crawling [Weibo](www.weibo.com) daily.
New words are then added to our database.
The words are then extracted and obfuscated to produce questions.
Users then simply pick correct words that fit into sentences or phrases presented to them.

# Dependencies
We have some libraries that we are dependent upon. 
The required jars have been included in the repository.
The following is the list of dependencies.

#### Apache HTTPComponents
Used for connecting to the websites
If there are issues with the jars provided in the repo, you may get the latest version [here](https://hc.apache.org/downloads.cgi).

#### JSoup
Used to manipulate the HTTP pages that we have crawled
If there are issues with the jars provided in the repo, you may get the latest version [here](https://jsoup.org/download).

#### dom4J
Used to parse the HTTP pages that we have crawled
If there are issues with the jars provided in the repo, you may get the latest version [here](https://github.com/dom4j/dom4j/releases).<commit_msg>Housekeeping — Updated readme<commit_after># Learn Chinese
Simple game to learn how to form Chinese words

The project utilises a java web crawler to get our database of questions.
The words are obtained by crawling [Weibo](www.weibo.com) daily.
New words are then added to our database.
The words are then extracted and obfuscated to produce questions.
Users then simply pick correct words that fit into sentences or phrases presented to them.

# Structure
The project is split into its front end, back end as well as the crawler.
The crawler is a standalone java app. The app will be run daily by the server to renew the database. 
The front end contains all the code pertaining to how the interface for the game will look to the users.
The back end handles all the server side logic for the game. This involve pulling phrases or words from the database and turning them into questions.

# Dependencies
We have some libraries that we are dependent upon. 
The required jars have been included in the repository.
The following is the list of dependencies.

#### Apache HTTPComponents
Used for connecting to the websites
If there are issues with the jars provided in the repo, you may get the latest version [here](https://hc.apache.org/downloads.cgi).

#### JSoup
Used to manipulate the HTTP pages that we have crawled
If there are issues with the jars provided in the repo, you may get the latest version [here](https://jsoup.org/download).

#### dom4J
Used to parse the HTTP pages that we have crawled
If there are issues with the jars provided in the repo, you may get the latest version [here](https://github.com/dom4j/dom4j/releases).
{'fraction_non_alphanumeric': 0.05145493257629524, 'fraction_numerical': 0.0021291696238466998, 'mean_word_length': 4.390057361376673, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 6, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6282076', 'n_tokens_mistral': 713, 'n_tokens_neox': 680, 'n_words': 436}