<commit_before>
# Blacklist overlap test <img align="right", src="http://i.imgur.com/ln57AGi.png" width="386">

The blacklist overlap test provides a measure on the degree to which one feed is contained within some other feed. It is a metric that may be used to exclude blacklist feeds that is already contained in other feeds.

This project is inspired by the combine and tiq-test project (written in R) by [@alexcpsec](https://twitter.com/alexcpsec), [@kylemaxwell](https://twitter.com/kylemaxwell) and others. While the tiq-test project provides additional quality metrics and also supports enrichment, this project is pure Python and at this point only provides the overlap test. That said, if overlap test is the goal just running the Jupyter notebook or `overlap_test.py` will do the fetch data, extract IP addresses, calculate overlap between each pair, and build heatmap steps. It is designed to be standalone. 

If you prefer ggplot heatmaps, there's an option to save a .csv file with a format that is expected by tiq.test. See [Importing into tiq-test](#tiq-test) for an example. However, if [Seaborn](http://stanford.edu/~mwaskom/software/seaborn/) is your prefered choise of visualisation library and you're interested in overlap test then please continue reading.


## Purpose
The purpose of this project is to provide a means to measure the degree to which blacklists overlap in terms of IPv4 entries. 

Addresses are fetched from public and private (optional) blacklists and two plots are given as output, a barchart showing sizes per blacklist and a heatmap showing the degree of overlap between the blacklists. There is also support for importing blacklists from the local filesystem.

The Jupyter notebook found in notebooks directory gives transparancy into the code and plots as you run it, the [Standalone .py version](#standalone) saves the plots on disk and has a separate config file. But in terms of functionality the .py and notebook versions don't differ apart from the option to save the data in a .csv file if running the .py version.

URLs to feeds or paths to local files to be imported are all set within the notebook or in overlap.conf (if running the .py version). There are no assumptions on the formats of these feeds or local files, a regexp is used to search for IPv4 addresses line by line.


## Plots
Example output is shown below.

### Blacklist feed sizes
This barchart shows the blacklist feed sizes, with entry count on x-axis and the feeds (sources) on the y-axis.

![Barchart with sizes of blacklist feeds](http://i.imgur.com/sVf38Xs.png)

### Heatmaps displaying overlap between feeds 
The heatmap is used to show the overlap between pairs of feeds. The color of a cell reveals the degree to which a blacklist feed on the y-axis contains some feed on the x-axis.

An example is shown below. Here we note that e.g. *alienvault_reputation.generic* contains all of *openbl.org_base.txt*. We also note that *emergingthreats_emerging-Block-IPs.txt* cotains close to 100% of both *feodotracker_blocklists_ipblocklist* and *zeustracker_blocklist_badips*.

Setting ANNOTATE to *True* or *False* determines whether the actual value are shown in the cell or not.

**Regular**

![Heatmap showing overlap](http://i.imgur.com/709Ov5s.png)

**Annotated**

![Heatmap showing overlap - annoteated](http://i.imgur.com/1dVE22w.png)


## I/O
#### Input
- List of URLs that points to text files that contain IPv4 addresses assumed to be blacklisted.
- List of local text files that contain blacklisted IPv4 addresses.


#### Output
- Two plots: barchart and heatmap. Rendered inline if running Jupyiter notebook version or optionally saved to disk if running .py version.
- In the standalone .py version a .csv file that basically contains a concatenation of the input data may be saved in a format accepted by tiq-test.


## Usage

Make sure the [requirements](#requirements) are fullfilled. 

### Notebook
```bash
$ cd notebooks/
$ jupyter notebook overlap_test.ipynb
```
All configuration are done within the notebook.

### <a name="standalone"></a>Standalone .py version
```bash
$ cd src/scripts/
$ python overlap_test.py
```

A partial output from the execution of the standalone version is shown below. The only differences to the notebook versions are the possibility to save the plots to disk and that the config is located in a separate file.


```bash
(ml)ola-lx3:ola 08:26 ~/projs/blacklist_overlap_test/src/scripts develop > python overlap_test.py 
2016-05-23 08:26:54,698 main          : INFO     ------------------++++++++BEGIN+++++++++++----------------------

2016-05-23 08:26:54,709 GetData       : INFO     >>>> Fetching public inbound blacklisted IPv4 addresses from URLs <<<<<commit_msg>Spelling errors<commit_after>
# Blacklist overlap test <img align="right", src="http://i.imgur.com/ln57AGi.png" width="386">

The blacklist overlap test provides a measure on the degree to which one feed is contained within some other feed. It is a metric that may be used to exclude blacklist feeds that is already contained in other feeds.

This project is inspired by the combine and tiq-test project (written in R) by [@alexcpsec](https://twitter.com/alexcpsec), [@kylemaxwell](https://twitter.com/kylemaxwell) and others. While the tiq-test project provides additional quality metrics and also supports enrichment, this project is pure Python and at this point only provides the overlap test. That said, if overlap test is the goal just running the Jupyter notebook or `overlap_test.py` will do the fetch data, extract IP addresses, calculate overlap between each pair, and build heatmap steps. It is designed to be standalone. 

If you prefer ggplot heatmaps, there's an option to save a .csv file with a format that is expected by tiq.test. See [Importing into tiq-test](#tiq-test) for an example. However, if [Seaborn](http://stanford.edu/~mwaskom/software/seaborn/) is your prefered choise of visualisation library and you're interested in overlap test then please continue reading.


## Purpose
The purpose of this project is to provide a means to measure the degree to which blacklists overlap in terms of IPv4 entries. 

Addresses are fetched from public and private (optional) blacklists and two plots are given as output, a barchart showing sizes per blacklist and a heatmap showing the degree of overlap between the blacklists. There is also support for importing blacklists from the local filesystem.

The Jupyter notebook found in notebooks directory gives transparancy into the code and plots as you run it, the [Standalone .py version](#standalone) saves the plots on disk and has a separate config file. But in terms of functionality the .py and notebook versions don't differ apart from the option to save the data in a .csv file if running the .py version.

URLs to feeds or paths to local files to be imported are all set within the notebook or in overlap.conf (if running the .py version). There are no assumptions on the formats of these feeds or local files, a regexp is used to search for IPv4 addresses line by line.


## Plots
Example output is shown below.

### Blacklist feed sizes
This barchart shows the blacklist feed sizes, with entry count on x-axis and the feeds (sources) on the y-axis.

![Barchart with sizes of blacklist feeds](http://i.imgur.com/sVf38Xs.png)

### Heatmaps displaying overlap between feeds 
The heatmap is used to show the overlap between pairs of feeds. The color of a cell reveals the degree to which a blacklist feed on the y-axis contains some feed on the x-axis.

An example is shown below. Here we note that e.g. *alienvault_reputation.generic* contains all of *openbl.org_base.txt*. We also note that *emergingthreats_emerging-Block-IPs.txt* cotains close to 100% of both *feodotracker_blocklists_ipblocklist* and *zeustracker_blocklist_badips*.

Setting ANNOTATE to *True* or *False* determines whether the actual value are shown in the cell or not.

**Regular**

![Heatmap showing overlap](http://i.imgur.com/709Ov5s.png)

**Annotated**

![Heatmap showing overlap - annoteated](http://i.imgur.com/1dVE22w.png)


## I/O
#### Input
- List of URLs that points to text files that contain IPv4 addresses assumed to be blacklisted.
- List of local text files that contain blacklisted IPv4 addresses.


#### Output
- Two plots: barchart and heatmap. Rendered inline if running Jupyter notebook version or optionally saved to disk if running .py version.
- In the standalone .py version a .csv file that basically contains a concatenation of the input data may be saved in a format accepted by tiq-test.


## Usage

Make sure the [requirements](#requirements) are fullfilled. 

### Notebook
```bash
$ cd notebooks/
$ jupyter notebook overlap_test.ipynb
```
All configuration are done within the notebook.

A static version of the notebook rendered by GitHub is found here: [overlap_test.ipynb](https://github.com/mrola/blacklist_overlap_test/blob/master/notebooks/overlap_test.ipynb)

### <a name="standalone"></a>Standalone .py version
```bash
$ cd src/scripts/
$ python overlap_test.py
```

A partial output from the execution of the standalone version is shown below. The only differences to the notebook versions are the possibility to save the plots to disk and that the config is located in a separate file.


```bash
(ml)ola-lx3:ola 08:26 ~/projs/blacklist_overlap_test/src/scripts develop > python overlap_test.py 
2016-05-23 08:26:54,698 main          : INFO     ------------------++++++++BEGIN+++++++++++----------------------

2016-05-23 08:26:54,709 GetData       : INFO     >>>> Fetching public inbound blacklisted IPv4 addresses from URLs <<<<
{'fraction_non_alphanumeric': 0.07968911917098445, 'fraction_numerical': 0.012642487046632124, 'mean_word_length': 4.2536744692433315, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 19, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '694703', 'n_tokens_mistral': 2764, 'n_tokens_neox': 2546, 'n_words': 1324}