<jupyter_start><jupyter_text>Objective* Vetting each of the categorical and numerical columns for missing values andoutliers.<jupyter_code>import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
%%javascript 
//Disable autoscrolling to see entire graph
IPython.OutputArea.prototype._should_scroll = function(lines) {
    return false;
}<jupyter_output><empty_output><jupyter_text>1. Segregate the categorical attributes in a separate dataframe to analyze:* Figure out the categorical columns from the data type.* Find out the number of missing values in each column.* Get the number of labels in each variable.<jupyter_code># Read Rain in Australia Dataset
df_weather = pd.read_pickle("../data/df_weather_2.pkl")
df_weather.columns
# Divide columns by numerical and categorical
numerical_columns = df_weather._get_numeric_data().columns
categorical_columns = list(set(df_weather.columns) - set(numerical_columns))
df_weather_categorical = df_weather[categorical_columns]
df_weather_categorical.shape
df_weather_categorical.info()
# Determine number of missing values in each categorical variable
df_weather_categorical.isna().sum()
# Display number of possible values in each column, left justify column name for easier reading
for column in categorical_columns:
    number_of_possible_values = len(df_weather[column].value_counts())    
    print(f"{column.ljust(12,' ')}: {number_of_possible_values:,}")
# Get frequency distribution and unique labels
df_weather['RainToday'].value_counts()<jupyter_output><empty_output><jupyter_text>RainToday has 1,406 missing elements as indicated by the -1  2. Now, segregate the numerical attributes to analyze them separately:* Figure out the numerical columns using the data type of the columns.* Find out the number of missing values in each column.<jupyter_code># Dataframe with numerical variables
df_weather_numerical = df_weather[numerical_columns]
# Sample 5 rows of numerical variables
df_weather_numerical.sample(5)
# Display data types for each variable
df_weather_numerical.info()
# Determine number of missing values in each numerical variables
df_weather_numerical.isna().sum()<jupyter_output><empty_output><jupyter_text>3. Identify the potential columns for outliers:* Look at the summary statistics to figure out the potential columns foroutliers.* Visualize the data points for the shortlisted columns using box plots.<jupyter_code># Get frequency distribution and unique labels, Transpose for easier reading
df_weather_numerical.describe().T<jupyter_output><empty_output><jupyter_text>At first glance, Rainfall has a max of 371 with a 75% of 0.80. Sunshine has a max of 135 with a 75% of 48.WindGustSpeed has a max of 135 with a 75% percentile of 48. Similarly WindSpeed9am has a percentile of 75% go 19 but a max of 130. These are potential outliers<jupyter_code># Temperature boxplots
df_weather_numerical[['MinTemp','MaxTemp','Temp9am','Temp3pm']].boxplot()
# Rainfall boxplots
df_weather_numerical[['Rainfall']].boxplot()
# Evaporation boxplots
df_weather_numerical[['Evaporation']].boxplot()
# Sunshine boxplots
df_weather_numerical[['Sunshine']].boxplot()
# Sunshine boxplots
df_weather_numerical[['WindGustSpeed','WindSpeed9am','WindSpeed3pm']].boxplot()
# Humidity boxplots
df_weather_numerical[['Humidity9am','Humidity3pm']].boxplot()
# Presure boxplots
df_weather_numerical[['Pressure9am','Pressure3pm']].boxplot()
# Presure boxplots
df_weather_numerical[['Cloud9am','Cloud3pm']].boxplot()<jupyter_output><empty_output><jupyter_text>**Based on these boxplots, it looks like these variables have outliers:*** Rainfall* Evaporation* WindGustSpeed* WindSpeed9am* WindSpeed3pm  4. Calculate the IQR(Interquartile Range) for the shortlisted columns:* Print the lower bound and upper bound for each of the columns.<jupyter_code>df_weather_numerical['Rainfall'].describe().T
# Calculate Rainfall IQR
column = 'Rainfall'
quantile_25 = df_weather[column].quantile(.25)
quantile_75 = df_weather[column].quantile(.75)
iqr = quantile_75 - quantile_25
print(f" {column} IQR: {iqr}")
print(f" {column} 25% quantile: {quantile_25}")
print(f" {column} 75% quantile: {quantile_75}")
# Calculate Evaporation IQR
column = 'Evaporation'
quantile_25 = df_weather[column].quantile(.25)
quantile_75 = df_weather[column].quantile(.75)
iqr = quantile_75 - quantile_25
print(f" {column} IQR: {iqr}")
print(f" {column} 25% quantile: {quantile_25}")
print(f" {column} 75% quantile: {quantile_75}")
# Calculate WindGustSpeed IQR
column = 'WindGustSpeed'
quantile_25 = df_weather[column].quantile(.25)
quantile_75 = df_weather[column].quantile(.75)
iqr = quantile_75 - quantile_25
print(f" {column} IQR: {iqr}")
print(f" {column} 25% quantile: {quantile_25}")
print(f" {column} 75% quantile: {quantile_75}")
# Calculate WindSpeed9am IQR
column = 'WindSpeed9am'
quantile_25 = df_weather[column].quantile(.25)
quantile_75 = df_weather[column].quantile(.75)
iqr = quantile_75 - quantile_25
print(f" {column} IQR: {iqr}")
print(f" {column} 25% quantile: {quantile_25}")
print(f" {column} 75% quantile: {quantile_75}")
# Calculate WindSpeed3pm IQR
column = 'WindSpeed3pm'
quantile_25 = df_weather[column].quantile(.25)
quantile_75 = df_weather[column].quantile(.75)
iqr = quantile_75 - quantile_25
print(f" {column} IQR: {iqr}")
print(f" {column} 25% quantile: {quantile_25}")
print(f" {column} 75% quantile: {quantile_75}")<jupyter_output>WindSpeed3pm IQR: 11.0
 WindSpeed3pm 25% quantile: 13.0
 WindSpeed3pm 75% quantile: 24.0
{'fraction_non_alphanumeric': 0.10624315443592552, 'fraction_numerical': 0.0321285140562249, 'mean_word_length': 7.27643504531722, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '594888', 'license': 'MIT', 'n_tokens_mistral': 1957, 'n_tokens_neox': 1744, 'n_words': 500, 'path': 'part_one/Rain Prediction - Milestone - 3.ipynb', 'repo_name': 'arielgamino/rain-prediction', 'size': '5478'}