<commit_before>NewsdiffHK
==========

This is the backend of Newsdiff HK.  The frontend lives [here](https://github.com/code4hk/Newsdiff-Frontend).

Inspired by and modified from similar projects in [Taiwan](https://github.com/ronnywang/newsdiff) and [US](https://github.com/ecprice/newsdiffs).

Dev Environment Setup
---------------------
To create a virtual environment and install the dependencies:

    $ virtualenv -p python3 newsdiff
    $ source newsdiff/bin/activate
    $ pip3 install -r requirements.txt

To deactivate the virtual environment after use:

    $ deactivate
   <commit_msg>Add details for running scraper<commit_after>NewsdiffHK
==========

This is the backend of Newsdiff HK.  The frontend lives [here](https://github.com/code4hk/Newsdiff-Frontend).

Inspired by and modified from similar projects in [Taiwan](https://github.com/ronnywang/newsdiff) and [US](https://github.com/ecprice/newsdiffs).

Dev Environment Setup
---------------------
To create a virtual environment and install the dependencies:

    $ virtualenv -p python3 newsdiff
    $ source newsdiff/bin/activate
    $ pip3 install -r requirements.txt

To run the scraper:
	
	$ python3 main.py &

To track the log:
	$ tail news_diff.log -f

To deactivate the virtual environment after use:

    $ deactivate
   
{'fraction_non_alphanumeric': 0.1276923076923077, 'fraction_numerical': 0.005384615384615384, 'mean_word_length': 3.486206896551724, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '32132', 'n_tokens_mistral': 391, 'n_tokens_neox': 372, 'n_words': 123}