<jupyter_start><jupyter_text>Exercises- Within your codeup-data-science directory, create a new repo named anomaly-detection-exercises. This will be where you do your work for this module. Create a repository on GitHub with the same name, and link your local repository to GitHub.- Save this work in your anomaly-detection-exercises repo. Then add, commit, and push your changes.- continuous_probabilistic_methods.py or continuous_probabilistic_methods.ipynb 1. Define a function named get_lower_and_upper_bounds that has two arguments. The first argument is a pandas Series. The second argument is the multiplier, which should have a default argument of 1.5. 1) Using lemonade.csv dataset and focusing on continuous variables:      - Use the IQR Range Rule and the upper and lower bounds to identify the lower outliers of each column of lemonade.csv, using the multiplier of 1.5. Do these lower outliers make sense?Which outliers should be kept?      - Use the IQR Range Rule and the upper and lower bounds to identify the upper outliers of each column of lemonade.csv, using the multiplier of 1.5. Do these lower outliers make sense?Which outliers should be kept?   - Using the multiplier of 3, IQR Range Rule, and the lower and upper bounds, identify the outliers below the lower bound in each colum of lemonade.csv. Do these lower outliers make sense?Which outliers should be kept?   - Using the multiplier of 3, IQR Range Rule, and the lower and upper bounds, identify the outliers above the upper_bound in each colum of lemonade.csv. Do these upper outliers make sense? Which outliers should be kept?<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
df = pd.read_csv('lemonade.csv')
df.head()
df.shape
df.tail()
df.Date = pd.to_datetime(df.Date)
df = df.set_index('Date')
df['Month'] = df.index.month_name()
df.head()
sns.pairplot(df)
plt.scatter(df.Month, df.Rainfall)
plt.xticks(rotation = 90)
plt.show()
for col in df.columns:
    if np.issubdtype(df[col].dtype, np.number):
        df[col].hist()
        print(col)
        plt.show()
# Let's use IQR for the dataset
def get_lower_and_upper_bounds(col, multiplier = 1.5):
    """
    Positional arguments:
    col: a pandas series
    keyword arguments:
    mult: default 1.5, the magnutidue specified for the IQR rule
    
    returns:
    lower_bound and upper)bound: two float values representing the f`ence values requested.
    """

    q1 = col.quantile(.25)
    q3 = col.quantile(.75)
    iqr = q3 - q1
    upper_bound = q3 + (multiplier * iqr)
    lower_bound = q1 - (multiplier * iqr)

    return upper_bound, lower_bound
outliers = {}
for col in df.columns:
    if np.issubdtype(df[col].dtype, np.number):
        lower_bound, upper_bound = get_lower_and_upper_bounds(df[col], multiplier = 1.5)
        outliers[col] = {}
        outliers[col]['bounds'] = {'upper' : upper_bound, 'lower' : lower_bound}
outliers
df_stats = df.describe()
df_stats.Temperature
z_scores = pd.Series((df['Temperature'] - df['Temperature'].mean()) / df['Temperature'].std())
df['z_scores_inf'] = z_scores
df[df['z_scores_inf'].abs() >= 2]
df[(df['Temperature'] < lower_bound) | (df['Temperature'] > upper_bound)]<jupyter_output><empty_output><jupyter_text>2) Identify if any columns in lemonade.csv are normally distributed. For normally distributed columns:        - Use a 2 sigma decision rule to isolate the outliers.       - Do these make sense?       - Should certain outliers be kept or removed? 3) Now use a 3 sigma decision rule to isolate the outliers in the normally distributed columns from lemonade.csv<jupyter_code>outliers2 = {}
for col in df.columns:
    if np.issubdtype(df[col].dtype, np.number):
        lower_bound, upper_bound = get_lower_and_upper_bounds(df[col], multiplier = 3)
        outliers2[col] = {}
        outliers2[col]['bounds'] = {'upper' : upper_bound, 'lower' : lower_bound}
outliers2 = pd.DataFrame(outliers)
outliers2.head()
z_scores = pd.Series((df['Temperature'] - df['Temperature'].mean()) / df['Temperature'].std())
z_scores = pd.DataFrame(z_scores)
z_scores
z_scores[z_scores.Temperature.abs() >= 2]
z_scores[z_scores.Temperature.abs() >= 3]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09883858734297227, 'fraction_numerical': 0.009717942640436123, 'mean_word_length': 4.235732009925559, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '627765', 'license': 'MIT', 'n_tokens_mistral': 1373, 'n_tokens_neox': 1268, 'n_words': 460, 'path': 'continuous_probabilistic_methods.ipynb', 'repo_name': 'ljackson707/anomaly-detection-exercises', 'size': '4219'}