<jupyter_start><jupyter_text>Table of Contents1&nbsp;&nbsp;Load pandas, matplotlib, and the StandardScaler and IsolationForest modules from scikit-learn2&nbsp;&nbsp;Create a standardized analysis data frame3&nbsp;&nbsp;Run an Isolation Forest model to detect outliers4&nbsp;&nbsp;Create outlier and inlier data frames5&nbsp;&nbsp;Plot the outliers and inliers  Load pandas, matplotlib, and the StandardScaler and IsolationForest modules from scikit-learn<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import IsolationForest
from mpl_toolkits.mplot3d import Axes3D
# pd.set_option('display.width', 80)
# pd.set_option('display.max_rows', 20)
# pd.set_option('display.max_columns', 7)
import watermark
%load_ext watermark

%watermark -n -v -iv
covidtotals = pd.read_csv('data/covidtotals.csv')
covidtotals.set_index('iso_code', inplace=True)<jupyter_output><empty_output><jupyter_text>Create a standardized analysis data frame<jupyter_code>analysisvars = [
    'location', 'total_cases_pm', 'total_deaths_pm', 'pop_density',
    'median_age', 'gdp_per_capita'
]
standardizer = StandardScaler()
covidtotals.isnull().sum()
covidanalysis = covidtotals.loc[:, analysisvars].dropna()
covidanalysisstand = standardizer.fit_transform(covidanalysis.iloc[:, 1:])<jupyter_output><empty_output><jupyter_text>Run an Isolation Forest model to detect outliers<jupyter_code>clf = IsolationForest(n_estimators=100,
                      max_samples='auto',
                      contamination=0.1,
                      max_features=1.0)
clf.fit(covidanalysisstand)
covidanalysis['anomaly'] = clf.predict(covidanalysisstand)
covidanalysis['scores'] = clf.decision_function(covidanalysisstand)
covidanalysis['anomaly'].value_counts()<jupyter_output><empty_output><jupyter_text>Create outlier and inlier data frames<jupyter_code>inlier, outlier = covidanalysis.loc[
    covidanalysis['anomaly'] == 1], covidanalysis.loc[covidanalysis['anomaly']
                                                      == -1]
outlier.info()
outlier[[
    'location', 'total_cases_pm', 'total_deaths_pm', 'median_age',
    'gdp_per_capita', 'scores'
]].sort_values(['scores']).head(10)<jupyter_output><empty_output><jupyter_text>Plot the outliers and inliers<jupyter_code>ax = plt.subplots(figsize=(12, 9))
ax = plt.axes(projection='3d')
ax.set_title('Isolation Forest Anomaly Detection')
ax.set_zlabel('Cases Per Million')
ax.set_xlabel('GDP Per Capita')
ax.set_ylabel('Median Age')
ax.scatter3D(inlier['gdp_per_capita'],
             inlier['median_age'],
             inlier['total_cases_pm'],
             label='inliers',
             c='blue')
ax.scatter3D(outlier['gdp_per_capita'],
             outlier['median_age'],
             outlier['total_cases_pm'],
             label='outliers',
             c='red')
ax.legend()
plt.tight_layout()
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12044913235794488, 'fraction_numerical': 0.010207553589656346, 'mean_word_length': 5.074380165289257, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '296404', 'license': 'MIT', 'n_tokens_mistral': 1039, 'n_tokens_neox': 971, 'n_words': 172, 'path': 'src/datacleaning/Chapter 4/7_isolation_forest.ipynb', 'repo_name': 'vidyabhandary/DataScienceNotebooks', 'size': '2939'}