<commit_before># Local News Engine

Local News Engine is prototype software by [Talk About Local](https://talkaboutlocal.org.uk/). You can find out more about the project on the [Talk About Local blog](https://talkaboutlocal.org.uk/local-news-engine-prototype/). Development has been funded by [Google Digital News Initiative](https://www.digitalnewsinitiative.com/). The software is being developed by [Open Data Services](http://opendataservices.coop).

LNE helps local and hyperlocal journalists and bloggers work with public data that, while often available to the public, is difficult to work with - especially for time-pressured journalists. 

At the prototype stage, LNE is a toolkit containing:
* Scrapers / downloaders for three public data sources - planning and licensing records for the London Boroughs of Camden and Islington
* A parser for magistrates courts lists
* Tools to create three HTML files:
  * **leads.html** - A list of names which appear in two or more sources, and which may therefore be a lead to a story
  * **explore.html** - An aggregate list of all records from all sources, with tools to filter by London wards

## Running Local News Engine

### Setup

You will need Python3 (with virtualenv), NPM and golang installed - e.g. on Ubuntu

```
sudo aptitude install python3-virtualenv npm golang
```

Install python requirements
```
python3 -m virtualenv --python=python3 .ve
source .ve/bin/activate
pip install -r requirements.txt
```

Install javascript requirements
```
npm install .
```

### Courts Data

Get a courts file put it in the data directory and use pdftotext on it with layout command.

```
pdftotext -layout courts_data/courts.pdf
```
Then run parse the data which creates courts.txt

```
python courts_parse.py
```

The courts data parser writes its output to the courts_data directory. 

### Licensing and planning data

The scrapers put their results in the data directory

```
python islington_license.py 
python islington_planning.py 
python camden_license.py 
```
In order to rerun these the files created must be removed or moved.

For Camden planning

```
cd data
wget https://opendata.camden.gov.uk/resource/mcgw-i4rx.json
cd ..
```

### Camden New Journal & Islington Tribune entity recognition

See the top of [camdennewjournal_ner.py](camdennewjournal_ner.py).


### Name matching<commit_msg>Update readme on downloading all data for camden planning<commit_after># Local News Engine

Local News Engine is prototype software by [Talk About Local](https://talkaboutlocal.org.uk/). You can find out more about the project on the [Talk About Local blog](https://talkaboutlocal.org.uk/local-news-engine-prototype/). Development has been funded by [Google Digital News Initiative](https://www.digitalnewsinitiative.com/). The software is being developed by [Open Data Services](http://opendataservices.coop).

LNE helps local and hyperlocal journalists and bloggers work with public data that, while often available to the public, is difficult to work with - especially for time-pressured journalists. 

At the prototype stage, LNE is a toolkit containing:
* Scrapers / downloaders for three public data sources - planning and licensing records for the London Boroughs of Camden and Islington
* A parser for magistrates courts lists
* Tools to create three HTML files:
  * **leads.html** - A list of names which appear in two or more sources, and which may therefore be a lead to a story
  * **explore.html** - An aggregate list of all records from all sources, with tools to filter by London wards

## Running Local News Engine

### Setup

You will need Python3 (with virtualenv), NPM and golang installed - e.g. on Ubuntu

```
sudo aptitude install python3-virtualenv npm golang
```

Install python requirements
```
python3 -m virtualenv --python=python3 .ve
source .ve/bin/activate
pip install -r requirements.txt
```

Install javascript requirements
```
npm install .
```

### Courts Data

Get a courts file put it in the data directory and use pdftotext on it with layout command.

```
pdftotext -layout courts_data/courts.pdf
```
Then run parse the data which creates courts.txt

```
python courts_parse.py
```

The courts data parser writes its output to the courts_data directory. 

### Licensing and planning data

The scrapers put their results in the data directory

```
python islington_license.py 
python islington_planning.py 
python camden_license.py 
```
In order to rerun these the files created must be removed or moved.

For Camden planning

```
cd data
curl 'https://opendata.camden.gov.uk/resource/mcgw-i4rx.json?$limit=100000' > mcgw-i4rx.json
cd ..
```

### Camden New Journal & Islington Tribune entity recognition

See the top of [camdennewjournal_ner.py](camdennewjournal_ner.py).


### Name matching
{'fraction_non_alphanumeric': 0.07993304038501779, 'fraction_numerical': 0.003557229545930111, 'mean_word_length': 4.437997724687144, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 4, 'https://': 8, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2574546', 'n_tokens_mistral': 1366, 'n_tokens_neox': 1282, 'n_words': 596}