<jupyter_start><jupyter_text>Water Potability Data Exploration Notebook<jupyter_code>import sys
sys.path.append('/home/jovyan/work')

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

import pyspark
import pyspark.sql.functions as F
import pyspark.sql.types as T
from pyspark.sql import SparkSession, DataFrame

%matplotlib inline
spark = SparkSession.builder.getOrCreate()
spark.sparkContext.setLogLevel('ERROR')<jupyter_output>WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/usr/local/spark-3.1.2-bin-hadoop3.2/jars/spark-unsafe_2.12-3.1.2.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
21/07/27 18:33:56 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).<jupyter_text>Load the Dataset<jupyter_code>df = spark.read.csv('../data/water_potability.csv',inferSchema=True, header=True)
record_count = df.count()
print('Total record count: {}'.format(record_count))
# Show a Sample
df.show(5)<jupyter_output>+-----------------+------------------+------------------+-----------------+------------------+------------------+------------------+------------------+------------------+----------+
|               ph|          Hardness|            Solids|      Chloramines|           Sulfate|      Conductivity|    Organic_carbon|   Trihalomethanes|         Turbidity|Potability|
+-----------------+------------------+------------------+-----------------+------------------+------------------+------------------+------------------+------------------+----------+
|             null| 204.8904554713363|20791.318980747026|7.300211873184757|368.51644134980336| 564.3086541722439|  10.3797830780847|  86.9909704615088|2.9631353806316407|         0|
| 3.71608007538699|129.42292051494425|18630.057857970347|   6.635245883862|              null| 592.8853591348523|15.180013116357259| 56.32907628451764| 4.500656274942408|         0|
|8.099124189298397|224.23625939355776|19909.541732292393|9.275883602694089|              n[...]<jupyter_text>Exploratory Data Analysis<jupyter_code># Show the Schema and Check Types
df.printSchema()
# Convert to Pandas and View Summary Statistics
df_pandas = df.toPandas()
df_pandas.describe().transpose()<jupyter_output><empty_output><jupyter_text>Explore Missing Values<jupyter_code># Check for Missing Data
null_counts = df_pandas.isnull().sum()
null_counts
print('ph null percentage: {}'.format(null_counts['ph']/record_count*100))
print('Sulfate null percentage: {}'.format(null_counts['Sulfate']/record_count*100))
print('Trihalomethanes null percentage: {}'.format(null_counts['Trihalomethanes']/record_count*100))
df_nulls = df_pandas[['ph', 'Sulfate', 'Trihalomethanes']]
potable = df_pandas.query('Potability == 0')
not_potable = df_pandas.query('Potability == 1')

fig = plt.figure(figsize=(16,8))
num_cols = 3

for ax,column in enumerate(df_nulls.columns[:df_nulls.shape[1]]):
    plt.subplot(1, num_cols, ax+1)
    sns.kdeplot(x=not_potable[column],label='Not Potable')
    sns.kdeplot(x=potable[column],label='Potable')
    plt.legend(prop=dict(size=df_pandas.shape[1]))
    plt.ylabel('')<jupyter_output><empty_output><jupyter_text>For trihalomethane levels, they are roughly the same for potable and nonpotable water. We shoould be able to impute the missing data with the mean, and it should not change our model prediction much.However the ph level is lower in potable water, and the sulfate level is higher in potable water. So, we need to think about what to do with those values.<jupyter_code>df_pandas.corr().abs()['ph'].sort_values(ascending = False)
df_pandas.corr().abs()['Sulfate'].sort_values(ascending = False)<jupyter_output><empty_output><jupyter_text>Unfortunately, there are any strong correlations with either feature that we can use to impute the value with stratification.<jupyter_code>sns.catplot(data=df_pandas, x='Potability', y='ph', kind='box');
sns.catplot(data=df_pandas, x='Potability', y='Sulfate', kind='box');<jupyter_output><empty_output><jupyter_text>The mean value does appear to be roughtly the same for potable and non-potable water.  Target Distribution<jupyter_code># Distribution of Target Column
sns.set(rc={"figure.figsize":(8, 8)})
sns.countplot(x=df_pandas['Potability']);<jupyter_output><empty_output><jupyter_text>This is an unbalanced dataset, so when we build out our pipeline, we will want to undersample the majority class to balance things out. I will create and test out a function to do that in the next notebook.  Correlations<jupyter_code>corrs = df_pandas.corr()
sns.set(rc={"figure.figsize":(16, 12)})
sns.heatmap(corrs, square=True, annot=True, cbar=True, cmap="gist_heat");<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.17243335224049916, 'fraction_numerical': 0.07090187180941576, 'mean_word_length': 4.033301617507136, 'pattern_counts': {'":': 2, '<': 31, '<?xml version=': 0, '>': 31, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '21672', 'license': 'Apache-2.0', 'n_tokens_mistral': 1943, 'n_tokens_neox': 1657, 'n_words': 397, 'path': 'notebooks/1_water_potability_data_exploration.ipynb', 'repo_name': 'bryanfree66/udemy_mlflow_course', 'size': '5289'}