<jupyter_start><jupyter_text>Context---  Data Definitions--- The water_potability.csv file contains water quality metrics for 3276 different bodies of water.1. __pH value__ * pH is an important parameter in evaluating the acid–base balance of water. It is also the indicator of acidic or alkaline condition of water status. The World Health Organization (WHO) recommends a pH of 6.5 to 8.5.2. __Hardness__  * Hardness is mainly caused by calcium and magnesium salts. These salts are dissolved from geologic deposits through which water travels. The length of time water is in contact with hardness producing material helps determine how much hardness there is in raw water.3. __Solids__  * Water has the ability to dissolve a wide range of inorganic and some organic minerals or salts such as potassium, calcium, sodium, bicarbonates, chlorides, magnesium, sulfates etc. These minerals produce unwanted taste and diluted color in appearance of water. A desirable limit for total dissolved solids is 500 milligrams per liter (mg/L) and maximum limit is 1000mg/L which prescribed for drinking purpose.4. __Chloramines__  * Chlorine and chloramine are the major disinfectants used in public water systems. Chloramines are most commonly formed when ammonia is added to chlorine to treat drinking water. Chlorine levels up to 4mg/L or 4 parts per million (ppm) are considered safe in drinking water.5. __Sulfate__  * Sulfates are naturally occurring substances that are found in minerals, soil, and rocks. Sulfate concentration in seawater is about 2,700mg/L. It ranges from 3 to 30mg/L in most freshwater supplies, although much higher concentrations (1000mg/L) are found in some geographic locations.6. __Conductivity__  * Pure water is not a good conductor of electric current. The amount of dissolved solids in water determines the electrical conductivity (EC) which measures the ionic process of a solution that enables it to transmit current. According to WHO standards, the EC value of water should not exceeded 400μS/cm.7. __Organic_carbon__  * Total organic carbon (TOC) in source waters comes from decaying natural organic matter as well as synthetic sources. TOC is a measure of the total amount of carbon in organic compounds in pure water. According to the U.S. Environmental Protection Agency (EPA) there should be less than 2mg/L of TOC in treated/drinking water.8. __Trihalomethanes__  * Trihalomethanes (THM) are chemicals which may be found in water treated with chlorine. The concentration of THM in drinking water varies according to the level of organic material in the water, the amount of chlorine required to treat the water, and the temperature of the water that is being treated. THM levels up to 80ppm is considered safe in drinking water.9. __Turbidity__  * The turbidity of water depends on the quantity of solid matter present in a suspended state. It is a measure of light emitting properties of water and the test is used to indicate the quality of waste discharge with respect to colloidal matter.10. __Potability__  * Water's potability indicates if the water is safe for human consumption. [1][1]: https://www.kaggle.com/adityakadiwal/water-potability ---  Preprocessing<jupyter_code># Imports
import graphviz
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import accuracy_score
from sklearn import metrics
from sklearn import tree 

plt.figure(figsize=(10,10))
df = pd.read_csv("/content/drive/MyDrive/Fall 2021/CS 634/Project/water_potability.csv")
df.columns = df.columns.str.lower()
df.head()
sns.histplot(df.ph,kde = True)
sns.histplot(df.hardness,kde = True)
sns.histplot(df.solids,kde = True)
sns.histplot(df.chloramines,kde = True)
sns.histplot(df.sulfate,kde = True)
sns.histplot(df.conductivity,kde = True)
sns.histplot(df.organic_carbon,kde = True)
sns.histplot(df.trihalomethanes,kde = True)
sns.histplot(df.turbidity,kde = True)
df.describe().T
df.isnull().sum()
df[df['ph'].isnull()]
df[df['sulfate'].isnull()]
df[df['trihalomethanes'].isnull()]
df['ph']=df['ph'].fillna(df.groupby(['potability'])['ph'].transform('mean'))
df['sulfate']=df['sulfate'].fillna(df.groupby(['potability'])['sulfate'].transform('mean'))
df['trihalomethanes']=df['trihalomethanes'].fillna(df.groupby(['potability'])['trihalomethanes'].transform('mean'))
df.isnull().sum()
sc = StandardScaler()
df_sc = sc.fit_transform(df.iloc[:, :-1])
df_scaled = pd.DataFrame(df_sc, columns = df.columns[:-1])
df_scaled['potability'] = df['potability']
df_scaled.head()
sns.heatmap(df_scaled.corr(),annot=True)<jupyter_output><empty_output><jupyter_text>It is apparent that there is little correlation between features and therefore all will be utilized for the model.  Modeling---  K Nearest Neighbors<jupyter_code>x = df_scaled.drop("potability", axis=1)
y = df_scaled.potability
X_train , X_test , y_train , y_test = train_test_split(x , y, test_size=0.2, random_state=68)
for n in range(2,10):
  knn = KNeighborsClassifier(n_neighbors=n)
  knn.fit(X_train, y_train)
  y_pred = knn.predict(X_test)
  print("Accuracy for "+str(n)+" neighbors:",metrics.accuracy_score(y_test, y_pred))<jupyter_output>Accuracy for 2: 0.6310975609756098
Accuracy for 3: 0.6128048780487805
Accuracy for 4: 0.6310975609756098
Accuracy for 5: 0.6158536585365854
Accuracy for 6: 0.6371951219512195
Accuracy for 7: 0.635670731707317
Accuracy for 8: 0.6280487804878049
Accuracy for 9: 0.6280487804878049<jupyter_text>Decision Trees<jupyter_code>model = tree.DecisionTreeClassifier(criterion='gini', splitter='best', max_depth=5, class_weight=None, min_samples_leaf=200, random_state=37)
clf = model.fit(X_train, y_train)
pred_labels_tr = model.predict(X_train)
pred_labels_te = model.predict(X_test)
print('Classes: ', clf.classes_)
print('Tree Depth: ', clf.tree_.max_depth)
print('Leaves: ', clf.tree_.n_leaves)
print('Features: ', clf.n_features_in_)
tree_data = tree.export_graphviz(clf, out_file=None, 
                            feature_names=x.columns, 
                            class_names=[str(list(clf.classes_)[0]), str(list(clf.classes_)[1])],
                            filled=True, 
                            rounded=True, 
                            ) 
graph = graphviz.Source(tree_data)
graph
score = model.score(X_test, y_test)
print('Accuracy Score: ', score)<jupyter_output>Accuracy Score:  0.7545731707317073
{'fraction_non_alphanumeric': 0.08961614925829638, 'fraction_numerical': 0.036091145435081816, 'mean_word_length': 5.095060577819199, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '475271', 'license': 'CC0-1.0', 'n_tokens_mistral': 2226, 'n_tokens_neox': 1960, 'n_words': 686, 'path': 'Final_Project.ipynb', 'repo_name': 'joedemcher/water-potability', 'size': '6539'}