<commit_before># Background

Let's compare the amount different types of crimes are reported in certain areas vs. the actual occurrence amount in those areas. Are some crimes under-represented in certain areas but over-represented in others? To accomplish this, we'll need to be able to extract a type-of-crime tag and geospatial data from news articles.

We meet every Tuesday at [Chi Hack Night](https://chihacknight.org/), and you can find out more about [this specific project here](https://github.com/chihacknight/breakout-groups/issues/61).

The [Chicago Justice Project](http://chicagojustice.org/) has been scraping RSS feeds of articles written by Chicago area news outlets for several years, allowing them to collect almost 300,000 articles. At the same time, an amazing group of [volunteers](http://chicagojustice.org/volunteer-for-cjp/) have helped them tag these articles. The tags include crime categories like "Gun Violence", "Drugs", "Sexual Assault", but also organizations such as "Cook County State's Attorney's Office", "Illinois State Police", "Chicago Police Department", and other miscellaneous categories such as "LGBTQ", "Immigration".

# Installation and Usage

## Requirements

To install this library, you will need at least the python packages [nltk](http://www.nltk.org/), [numpy](http://www.numpy.org/), [scikit-learn](http://scikit-learn.org/), and [pandas](http://pandas.pydata.org/). We recommend using [Anaconda](https://www.continuum.io/downloads):

```bash
$ # create a new anaconda environment with required packages
$ conda create -n article-tagging nltk numpy scikit-learn pandas
$ source activate article-tagging
(article-tagging) $ ...
```

## Installation

Download the code from git, `cd` into the directory, and run the setup.py file.

```bash
$ git clone <EMAIL>:chicago-justice-project/article-tagging.git
$ cd article-tagging
$ python setup.py install
```

As long as the nltk package is already installed, running the setup.py file should automatically download the required nltk corpora.

## Testing

You will additionally need `pytest` installed to run the tests.

TODO

## Usage

### Inside python

The main class is `newstag.crimetype.tag.Tagger`:

```python
>>> import newstag
>>> tagger = newstag.crimetype.tag.Tagger()
>>> article_text = 'This is an article about lots of crimes. Crimes about drugs.'
>>> tagger.relevant(article_text, prob_thresh=0.1)
True
>>> tagger.tagtext(article_text, prob_thresh=0.5)
['DRUG', 'CPD']
>>> tagger.tagtext_prob(article_text)
<pandas series>
```

### Command line interface

The installation comes with a command line interface, which without any arguments defaults to reading from the stdin.

```bash
$ python -m newstag.crimetype.cli
Go ahead and start typing. Hit ctrl-d when done.
<type here>
```

Or you can provide an article to tag.

```bash
$ python -m newstag.crimetype.cli sample-article.txt
$ cat sample-article.txt.tagged
GUNV, 0.9877
HOMI, 0.8765
...
```

Note that the `-m` flag is required.

# Areas of research

## Type-of-Crime Article Tagging<commit_msg>re-ordering readme sections<commit_after># Installation and Usage

## Requirements

To install this library, you will need at least the python packages [nltk](http://www.nltk.org/), [numpy](http://www.numpy.org/), [scikit-learn](http://scikit-learn.org/), and [pandas](http://pandas.pydata.org/). We recommend using [Anaconda](https://www.continuum.io/downloads):

```bash
$ # create a new anaconda environment with required packages
$ conda create -n article-tagging nltk numpy scikit-learn pandas pytest
$ source activate article-tagging
(article-tagging) $ ...
```

## Installation

Download the code from git, `cd` into the directory, and run the setup.py file. If you created an Anaconda environment, then make sure that environment is active before running the setup file.

```bash
$ git clone <EMAIL>:chicago-justice-project/article-tagging.git
$ cd article-tagging
$ python setup.py install
```

### nltk

As long as the nltk package is already installed, running the setup.py file should automatically download the required nltk corpora. If that does not work for some reason, then you will need to download the corpora manually. See the list `required_nltk_packages` in setup.py. Each corpus can be downloaded by running `nltk.download(corpus_name)`

## Testing

You will additionally need `pytest` installed to run the tests.

TODO

## Usage

### Inside python

The main class is `newstag.crimetype.tag.Tagger`:

```python
>>> import newstag
>>> tagger = newstag.crimetype.tag.Tagger()
>>> article_text = 'This is an article about lots of crimes. Crimes about drugs.'
>>> tagger.relevant(article_text, prob_thresh=0.1)
True
>>> tagger.tagtext(article_text, prob_thresh=0.5)
['DRUG', 'CPD']
>>> tagger.tagtext_prob(article_text)
<pandas series>
```

### Command line interface

The installation comes with a command line interface, which without any arguments defaults to reading from the stdin.

```bash
$ python -m newstag.crimetype.cli
Go ahead and start typing. Hit ctrl-d when done.
<type here>
```

Or you can provide an article to tag.

```bash
$ python -m newstag.crimetype.cli sample-article.txt
$ cat sample-article.txt.tagged
GUNV, 0.9877
HOMI, 0.8765
...
```

Note that the `-m` flag is required.

# Background

We want to compare the amount different types of crimes are reported in certain areas vs. the actual occurrence amount in those areas. Are some crimes under-represented in certain areas but over-represented in others? To accomplish this, we'll need to be able to extract a type-of-crime tag and geospatial data from news articles.

We meet every Tuesday at [Chi Hack Night](https://chihacknight.org/), and you can find out more about [this specific project here](https://github.com/chihacknight/breakout-groups/issues/61).

The [Chicago Justice Project](http://chicagojustice.org/) has been scraping RSS feeds of articles written by Chicago area news outlets for several years, allowing them to collect almost 300,000 articles. At the same time, an amazing group of [volunteers](http://chicagojustice.org/volunteer-for-cjp/) have helped them tag these articles. The tags include crime categories like "Gun Violence", "Drugs", "Sexual Assault", but also organizations such as "Cook County State's Attorney's Office", "Illinois State Police", "Chicago Police Department", and other miscellaneous categories such as "LGBTQ", "Immigration". The volunteer UI was also recently updated to allow highlighting of geographic information.

# Areas of research

## Type-of-Crime Article Tagging
{'fraction_non_alphanumeric': 0.10288753799392097, 'fraction_numerical': 0.006686930091185411, 'mean_word_length': 4.7526223776223775, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 45, 'https://': 6, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1282976', 'n_tokens_mistral': 2018, 'n_tokens_neox': 1894, 'n_words': 775}