<jupyter_start><jupyter_text>Unsupervised Anomaly Detection: Schools Performance 2016-17 ObjectiveThe focus of this notebook/project is to investigate the use of **unsupervised anomaly detection** methods on **Schools Performance 2016-17** data. We are particularly interested in spotting anomalous pupil destination data.  DefinitionsThe following terms used in this project are defined below:+ **Unsupervised data:** When your data is not labelled. *e.g. If you have income data but you do not know if this data is correct*+ **Anomaly data:** Unusual and typically wrong observations/data points in your data.+ **Unsupervised anomaly detection:** Spotting anomaly observations in your data without any labelled data to tell you whether your predictions are correct. DataThe data for this project comes from the [Schools Performance 2016-17](https://www.compare-school-performance.service.gov.uk/). The data is here very rich and detailed. For the sake of demonstration and simplification., we will only consider a subset of this information, defined below:+ **URN** | Unique reference number for schools+ **LEA** | Local authority, as a code+ **ESTAB** | Establishment number    - *Note that combining the LA and ESTAB fields to give LAESTAB will give us another unique identifer for schools, akin to our URN field*+ **SCHNAME** | School name+ **NFTYPE** | School type    - AC = Sponsored academy    - ACC = Academy converter – mainstream    - AC1619 = Academy 16-19 sponsor led    - ACC1619 = Academy 16-19 converter    - ACCS = Academy converter - special school    - ACS = Sponsored special academy, CTC=City technology college    - CY = Community school    - CYS = Community special school    - F = Free school – mainstream    - FESI = Further Education Sector Institution    - FD = Foundation school    - FDS = Foundation special school    - FS = Free school – special    - FSS = Studio school    - FUTC = UTC (university technical college)    - F1619 = Free school - 16-19    - IND = Independent school    - INDSPEC = Independent special school    - MODFC = College funded by Ministry of Defence    - NMSS = Non-maintained special school    - VA = Voluntary aided school,    - VC = Voluntary controlled school+ **OVERALL_DESTPER** | Percentage of pupils who have been in a sustained education or employment destination for the first two terms, October 2015 to March 2016. + **PTEALGRP2** | Percentage of eligible pupils with English-as-(an)-language (EAL)+ **PTMOBN** | Percentage of pupils classified as non-mobile+ **PTRWM_EXP** | Percentage of pupils reaching the expected stnadard in reading, writing and maths+ **PSENELST** | Percentage of eligible pupils with special-education-needs (SEN)+ **PTFSM6CLA1A_16** | Percentage of KS2 disdvantaged pupils+ **PNUMFSM** | Percentage of pupils for free-school-mealsThe anomalous pupil destination data that we are interested in spotting is therefore found within the **OVERALL_DESTPER** field. Note that this information is *provisional* so it is **unlabelled**.  Set-upNeed to set-up our Jupyter notebook so that it has the required libraries and an environment is set-up so that when sharing this noteboook, others can use the same environment as we did here.By default, the use of `!conda install` will install the package to the environment location that we are running the kernel in. This would be the environment from which we started Jupyter notebook from, but we can check this by looking at some of the system variables from the sys module.<jupyter_code>import sys
sys.executable
# Allows multiple outputs to show from same cell
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = 'all'

# Show the two dataframes side-by-side by creating a function
from IPython.display import display_html
def display_side(*args):
    html_str = ''
    for df in args:
        html_str += df.to_html()
    display_html(html_str.replace('table', 'table style = "display:inline"'),raw = True)

# Use pip freeze to examine installed packages and versions within our Jupyter session
!pip freeze
# Install libraries not already above
!conda install seaborn -y<jupyter_output>Solving environment: ...working... done

## Package Plan ##

  environment location: C:\Users\a_vis\Anaconda3

  added / updated specs: 
    - seaborn


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    seaborn-0.8.1              |   py35hc73483e_0         338 KB
    openssl-1.0.2o             |       h8ea7d77_0         5.4 MB
    ------------------------------------------------------------
                                           Total:         5.7 MB

The following NEW packages will be INSTALLED:

    seaborn:         0.8.1-py35hc73483e_0            

The following packages will be UPDATED:

    ca-certificates: 2018.1.18-0          conda-forge --> 2018.03.07-0     
    openssl:         1.0.2n-vc14_0        conda-forge [vc14] --> 1.0.2o-h8ea7d77_0


Downloading and Extracting Packages
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transact[...]<jupyter_text>Having downloaded all our required libraries, and set-up our environment as we wanted, we will save this environment so others taking this Jupyter notebook can use the same environment as us in order to replicate our results, and not go through the same hassle of going through the parts above this comment block. They will download this environment within their own Anaconda Navigator.Creating this environemnt, can upload it to my Anaconda Cloud account, which would allow other users to download and install it as a new environment within their own Anaconda Navigator.<jupyter_code># Export our environment, "NewEnv" and save it as "anomaly-detection.yml"
!conda env export -n NewEnv -f anomaly-detection.yml<jupyter_output><empty_output><jupyter_text>Data Load, Consolidation and WranglingIn this section, we will load in our data, join our different data sources together, and manipulate our joine ddataframe so that it is in a format for us to use various **anomaly detection** algorithms on.<jupyter_code># Check working directory to ensure user notebook is easily transferable
import os
os.getcwd()
# Import required libraries
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

# Display plots in Jupyter notebooks
% matplotlib inline

# Set random seed so we can reproduce our outputs
rng = np.random.RandomState(123)

# Load data
data_provisional_destination = pd.read_csv('Data 2016-2017/england_pupdestprovisional.csv', encoding = 'UTF-8')
data_final_ks2 = pd.read_csv('Data 2016-2017/england_ks2final.csv', encoding = 'UTF-8')
# Take only relevant columns

# View data
display_side(data_provisional_destination.head(), data_final_ks2.head())<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08387563268257411, 'fraction_numerical': 0.021836587129428778, 'mean_word_length': 3.598404255319149, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 14, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '586316', 'license': 'MIT', 'n_tokens_mistral': 1972, 'n_tokens_neox': 1774, 'n_words': 815, 'path': 'Detect-Anomalous-School-Performance.ipynb', 'repo_name': 'avisionh/Detect-Anomalous-Pupil-Destination', 'size': '6915'}