<jupyter_start><jupyter_text>Table of Contents1&nbsp;&nbsp;Load pandas, pyod, and scikit-learn, along with the Covid case data2&nbsp;&nbsp;Create a standardized data frame of the analysis columns3&nbsp;&nbsp;Run the KNN model and generate anomaly scores4&nbsp;&nbsp;Show the predictions from the model5&nbsp;&nbsp;Show the COVID data for the outliers  Load pandas, pyod, and scikit-learn, along with the Covid case data<jupyter_code>import pandas as pd
from pyod.models.knn import KNN
from sklearn.preprocessing import StandardScaler
# pd.set_option('display.width', 80)
# pd.set_option('display.max_columns', 7)
# pd.set_option('display.max_rows', 20)
# pd.options.display.float_format = '{:,.2f}'.format
import watermark
%load_ext watermark

%watermark -n -v -iv
covidtotals = pd.read_csv('data/covidtotals.csv')
covidtotals.set_index('iso_code', inplace=True)<jupyter_output><empty_output><jupyter_text>Create a standardized data frame of the analysis columns<jupyter_code>standardizer = StandardScaler()
analysisvars = [
    'location', 'total_cases_pm', 'total_deaths_pm', 'pop_density',
    'median_age', 'gdp_per_capita'
]
covidanalysis = covidtotals.loc[:, analysisvars].dropna()
covidanalysis.tail(2)
covidanalysisstand = standardizer.fit_transform(covidanalysis.iloc[:, 1:])<jupyter_output><empty_output><jupyter_text>Run the KNN model and generate anomaly scores<jupyter_code>clf_name = 'KNN'
clf = KNN(contamination=0.1)
clf.fit(covidanalysisstand)
y_pred = clf.labels_
y_pred[:3]
y_scores = clf.decision_scores_
y_scores[:3]<jupyter_output><empty_output><jupyter_text>Show the predictions from the model<jupyter_code>pred = pd.DataFrame(zip(y_pred, y_scores),
                    columns=['outlier', 'scores'],
                    index=covidanalysis.index)
pred.sample(10, random_state=1)
pred['outlier'].value_counts()
pred.groupby(['outlier'])[['scores']].agg(['min', 'median', 'max'])<jupyter_output><empty_output><jupyter_text>Show the COVID data for the outliers<jupyter_code>covidanalysis.join(pred).loc[
    pred['outlier'] == 1,
    ['location', 'total_cases_pm', 'total_deaths_pm', 'scores']].sort_values(
        ['scores'], ascending=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13120728929384967, 'fraction_numerical': 0.009567198177676537, 'mean_word_length': 6.956521739130435, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '574006', 'license': 'MIT', 'n_tokens_mistral': 790, 'n_tokens_neox': 753, 'n_words': 151, 'path': 'src/datacleaning/Chapter 4/6_outiers_knn.ipynb', 'repo_name': 'vidyabhandary/DataScienceNotebooks', 'size': '2195'}