<commit_before>## requirements

- python 3.x.
    - tested and developed on 3.3 and 3.5.
- install dependencies in requirements.txt:
    - `pip install -r requirements.txt`.
    - you may need `sudo` if you are not using virtualenv.
    - lxml may failed to install on osx, if so try:
        - `STATIC_DEPS=true LIBXML2_VERSION=2.9.2 pip install lxml==3.5.0`
        - see also:
            - http://louistiao.me/posts/installing-lxml-on-mac-osx-1011-inside-a-virtualenv-with-pip/
            - https://bugs.launchpad.net/lxml/+bug/1503807
            - http://lxml.de/installation.html#source-builds-on-macos-x<commit_msg>added lxml info to README.md.<commit_after>## requirements

- python 3.x.
    - tested and developed on 3.3 and 3.5.
- install dependencies in requirements.txt:
    - `pip install -r requirements.txt`.
    - you may need `sudo` if you are not using virtualenv.
    - lxml
        - on rhel/centos you will need:
            - `yum install python3-devel.x86_64 libxml2-devel.x86_64 libxslt-devel.x86_64`
            - `yum groupinstall 'Development Tools'`
        - equivalent packages should also work on other distros.
        - lxml may failed to install on osx, if so try:
            - `STATIC_DEPS=true LIBXML2_VERSION=2.9.2 pip install lxml==3.5.0`
            - see also:
                - http://louistiao.me/posts/installing-lxml-on-mac-osx-1011-inside-a-virtualenv-with-pip/
                - https://bugs.launchpad.net/lxml/+bug/1503807
                - http://lxml.de/installation.html#source-builds-on-macos-x
{'fraction_non_alphanumeric': 0.11886304909560723, 'fraction_numerical': 0.03875968992248062, 'mean_word_length': 2.1872427983539096, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 13}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3239022', 'n_tokens_mistral': 568, 'n_tokens_neox': 513, 'n_words': 124}