<commit_before># Blacklist Name Matcher

## Description:
* Imports a file with a list of blacklisted names, cleans it against a file with irrelevant names and contents
* Then implements a strict find against a query name,
* If no results are found, a modified (partial) search is implemented

## Contents
### Components:
* importer.py (1 method)
* processor.py (1 method)
* cleaner.py (1 method)
* terrorist_finder.py (3 methods)
* main.py

### Data:
* blacklist.tsv, blacklist.txt, ...
* noisefile.tsv, noisefile.txt, ...

## Example command line process
```
tanel@tanel:~/Documents/pyBlacklist$ python main.py

Please enter name to search in the terrorist list 
> Obama Bin Laden
Please enter name of blacklist file in same path (e.g. blacklist.tsv) or absolute location 
> blacklist.tsv
Please enter name of noise file in same path (e.g. noisefile.tsv) or absolute location 
> noisefile.tsv
No strict match, looking for partial matches...
TERRORIST MATCHED!
Certainty:  33.33 %
Name:  Osama Laden

TERRORIST MATCHED!
Certainty:  66.67 %
Name:  Osama Bin Laden

TERRORIST MATCHED!
Certainty:  66.67 %
Name:  Bin Laden, Osama

TERRORIST MATCHED!
Certainty:  66.67 %
Name:  Laden Osama Bin

TERRORIST MATCHED!
Certainty:  66.67 %
Name:  osama bin laden

TERRORIST MATCHED!
Certainty:  66.67 %
Name:  osama and bin laden
```

## Program logic
![ProgramLogic](https://cloud.githubusercontent.com/assets/5417573/21487516/86a0d6f2-cbd3-11e6-9b9b-d49cf43d27da.png)

## Program setup:
![ProgramSetUp](https://cloud.githubusercontent.com/assets/5417573/21275441/85653a40-c3d5-11e6-866a-c3029029aace.png)

## Details
* file has one name in every row (no XML, JSON,.., formatting)
* common filetypes: txt, csv, tsv

## Tech used:
* Python 2.7
* Lubuntu 16.04
<commit_msg>logic uml added<commit_after># Blacklist Name Matcher

## Description:
* Imports a file with a list of blacklisted names, cleans it against a file with irrelevant names and contents
* Then implements a strict find against a query name,
* If no results are found, a modified (partial) search is implemented

## Contents
### Components:
* importer.py (1 method)
* processor.py (1 method)
* cleaner.py (1 method)
* terrorist_finder.py (3 methods)
* main.py

### Data:
* blacklist.tsv, blacklist.txt, ...
* noisefile.tsv, noisefile.txt, ...

## Example command line process
```
tanel@tanel:~/Documents/pyBlacklist$ python main.py

Please enter name to search in the terrorist list 
> Obama Bin Laden
Please enter name of blacklist file in same path (e.g. blacklist.tsv) or absolute location 
> blacklist.tsv
Please enter name of noise file in same path (e.g. noisefile.tsv) or absolute location 
> noisefile.tsv
No strict match, looking for partial matches...
TERRORIST MATCHED!
Certainty:  33.33 %
Name:  <NAME>

TERRORIST MATCHED!
Certainty:  66.67 %
Name:  <NAME>

TERRORIST MATCHED!
Certainty:  66.67 %
Name:  <NAME>, Osama

TERRORIST MATCHED!
Certainty:  66.67 %
Name:  <NAME>

TERRORIST MATCHED!
Certainty:  66.67 %
Name:  osama bin laden

TERRORIST MATCHED!
Certainty:  66.67 %
Name:  osama and bin laden
```

## Program logic
![ProgramLogic](https://cloud.githubusercontent.com/assets/5417573/21487556/15817dfe-cbd4-11e6-823d-45ebd9346ae6.png)

## Program setup:
![ProgramSetUp](https://cloud.githubusercontent.com/assets/5417573/21275441/85653a40-c3d5-11e6-866a-c3029029aace.png)

## Details
* file has one name in every row (no XML, JSON,.., formatting)
* common filetypes: txt, csv, tsv

## Tech used:
* Python 2.7
* Lubuntu 16.04

{'fraction_non_alphanumeric': 0.10971428571428571, 'fraction_numerical': 0.05942857142857143, 'mean_word_length': 4.377880184331797, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 13, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '698027', 'n_tokens_mistral': 1390, 'n_tokens_neox': 1262, 'n_words': 390}