<jupyter_start><jupyter_text>DATA ANALYTICS HW02 Importing all the packages required<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
from sklearn import preprocessing
from scipy.spatial import distance_matrix
from sklearn.cluster import KMeans
from scipy.spatial.distance import cdist
import matplotlib.pyplot as plt
import gc
from sklearn import metrics
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import classification_report,confusion_matrix,precision_score,recall_score,f1_score,r2_score,roc_curve,roc_auc_score
from sklearn.model_selection import train_test_split, GridSearchCV, cross_val_score, cross_validate, ShuffleSplit, LeaveOneOut
import re
from impyute.imputation.cs import mice
from scipy.cluster.hierarchy import ward, dendrogram
from sklearn.cluster import KMeans,AgglomerativeClustering
import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline
matplotlib.style.use('ggplot')<jupyter_output><empty_output><jupyter_text>Reading both the training and testing data<jupyter_code>df=pd.read_csv("C:/Users/rtd91/OneDrive/Syracuse University/IST 707/Homework2/Weather Forecast Training.csv")
tempdf=df
df1=pd.read_csv("C:/Users/rtd91/OneDrive/Syracuse University/IST 707/Homework2/Weather Forecast Testing.csv")
tempdf1=df1
df.head(5) #Training Dataset
df1.head(5) #Testing Dataset
df.shape
df1.shape<jupyter_output><empty_output><jupyter_text>Data Preparation: Converting the categorical variables to numeric by label encoding for the processing of algorithms<jupyter_code>le = preprocessing.LabelEncoder()
obj_df = df.select_dtypes(include=['object']).copy()
obj_df.shape
for i in obj_df:
    le.fit(obj_df[i][obj_df[i].notna()].values)
    obj_df[i][obj_df[i].notna()] = le.transform(obj_df[i][obj_df[i].notna()].values)
le = preprocessing.LabelEncoder()
obj_df1 = df1.select_dtypes(include=['object']).copy()
obj_df1.shape
for i in obj_df1:
    le.fit(obj_df1[i][obj_df1[i].notna()].values)
    obj_df1[i][obj_df1[i].notna()] = le.transform(obj_df1[i][obj_df1[i].notna()].values)
obj_df = obj_df.apply(pd.to_numeric,axis =1)
obj_df1 = obj_df1.apply(pd.to_numeric,axis =1)<jupyter_output><empty_output><jupyter_text>Dropping the categorical values and later adding the numeric version of these columns for algorithm processing.<jupyter_code>tempdf=df
tempdf=tempdf.drop(["Location","WindGustDir","WindDir","RainToday","RainTomorrow"],axis=1)
tempdf.info()
tempdf1=df1
tempdf1=tempdf1.drop(["Location","WindGustDir","WindDir","RainToday"],axis=1)
tempdf1.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 12994 entries, 0 to 12993
Data columns (total 12 columns):
ID               12994 non-null int64
MinTemp          12947 non-null float64
MaxTemp          12976 non-null float64
Rainfall         12833 non-null float64
Evaporation      7478 non-null float64
Sunshine         6900 non-null float64
WindGustSpeed    12074 non-null float64
WindSpeed        12733 non-null float64
Humidity         12645 non-null float64
Pressure         11701 non-null float64
Cloud            7974 non-null float64
Temp             12727 non-null float64
dtypes: float64(11), int64(1)
memory usage: 1.2 MB<jupyter_text>Checking which columns has how many null values ***There were a lot of missing values in the dataset so replacing it directly with mean/mode/median could just have made the data bias so instead I tried using knn imputation and mice imputation, MICE imputation gave better accuracy results so I decided to go with MICE. It takes a while for computation but handles the missing data appropriately. ***<jupyter_code>df.isnull().sum() #checking which columns has how many null values<jupyter_output><empty_output><jupyter_text>Applying MICE imputation for Data Cleaning The mice package implements a method to deal with missing data. The package creates multiple imputations (replacement values) for multivariate missing data. The method is based on Fully Conditional Specification, where each incomplete variable is imputed by a separate model. The MICE algorithm can impute mixes of continuous, binary, unordered categorical and ordered categorical data.In addition, it can also process the two-level data.<jupyter_code># start the MICE training
imputed_training=mice(obj_df)
# start the MICE training
imputed_testing=mice(obj_df1)<jupyter_output>C:\Users\rtd91\Anaconda3\lib\site-packages\impyute\util\preprocess.py:52: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.
  args[0] = args[0].as_matrix()<jupyter_text>Since MICE imputation makes the columns label to numeric in order(0,1,2..).Renaming the columns for better understanding<jupyter_code>imputed_training=imputed_training.rename(columns={0: "Location", 1: "WindGustDir",2: "WindDir",3: "RainToday",4: "RainTomorrow"})
imputed_training
imputed_testing=imputed_testing.rename(columns={0: "Location", 1: "WindGustDir",2: "WindDir",3: "RainToday"})
imputed_testing<jupyter_output><empty_output><jupyter_text>Storing the imputed values back into the dataframes's new collumns for easier processing<jupyter_code>tempdf["Locationint"]=imputed_training["Location"]
tempdf["WindGustDirint"]=imputed_training["WindGustDir"]
tempdf["WindDirint"]=imputed_training["WindDir"]
tempdf["RainTodayint"]=imputed_training["RainToday"]
tempdf["RainTomorrowint"]=imputed_training["RainTomorrow"]
tempdf1["Locationint"]=imputed_testing["Location"]
tempdf1["WindGustDirint"]=imputed_testing["WindGustDir"]
tempdf1["WindDirint"]=imputed_testing["WindDir"]
tempdf1["RainTodayint"]=imputed_testing["RainToday"]<jupyter_output><empty_output><jupyter_text>Implementing MICE again on the entire dataframe to remove the NA values other than the object values which we had selected before.<jupyter_code># start the MICE training
tempdf=mice(tempdf)
tempdf1=mice(tempdf1)<jupyter_output>C:\Users\rtd91\Anaconda3\lib\site-packages\impyute\util\preprocess.py:52: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.
  args[0] = args[0].as_matrix()
C:\Users\rtd91\Anaconda3\lib\site-packages\impyute\util\preprocess.py:52: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.
  args[0] = args[0].as_matrix()<jupyter_text>Exploratory Data Analysis:  Taking care of Outliers: Visualizing some Outliers:<jupyter_code>sns.boxplot(x=df['MinTemp'])
sns.boxplot(x=df['MaxTemp'])
sns.boxplot(x=df['Rainfall'])<jupyter_output><empty_output><jupyter_text>After seeing these few plots, we see that just approximating some boundary for the outlier and removing it would not give accurate results.Since normalization plays a huge role in the algorithms. If we do not handle outliers properly, the normalization process could go wrong.So, to deal with outliers,I have used the IQR score function to detect and replace such values. Fixing outliers by the function created which helps to detect outliers by the value of IQR score.The values which are less than (Q1 - 1.5 * IQR) i.e. min and greater than (Q3 + 1.5* IQR) i.e max are outliers and these values are replaced by min and max respectively.<jupyter_code>def fixoutliers(x):
##Get all the column name from the input dataframe x
    xColumnNames=x.columns
    print(xColumnNames)
    #for j in df2ColumnNames:

    for j in xColumnNames:
        try:
            print("colnames ",j)
            xy=x[j]    
            mydata=pd.DataFrame()
            #print(xy)
            updated=[]
            Q1,Q3=np.percentile(xy,[25,75])
            IQR=Q3-Q1
            minimum=Q1-1.5*IQR
            maximum=Q3+1.5*IQR
            for i in xy:
                if(i>maximum):
                    print("Entering maxim")
                    i=maximum
                    updated.append(i)
                elif(i<minimum):
                    print("enterinf minimum")
                    i=minimum
                    updated.append(i)
                else:
                    updated.append(i)
            x[j]=updated
        except:
            continue
    return x
tempdf=fixoutliers(tempdf)
tempdf1=fixoutliers(tempdf1)<jupyter_output>RangeIndex(start=0, stop=16, step=1)
colnames  0
enterinf minimum
Entering maxim
enterinf minimum
enterinf minimum
enterinf minimum
enterinf minimum
enterinf minimum
enterinf minimum
enterinf minimum
enterinf minimum
Entering maxim
enterinf minimum
enterinf minimum
enterinf minimum
enterinf minimum
enterinf minimum
enterinf minimum
enterinf minimum
enterinf minimum
enterinf minimum
enterinf minimum
enterinf minimum
enterinf minimum
enterinf minimum
enterinf minimum
enterinf minimum
enterinf minimum
enterinf minimum
enterinf minimum
enterinf minimum
enterinf minimum
enterinf minimum
colnames  1
enterinf minimum
enterinf minimum
Entering maxim
Entering maxim
enterinf minimum
Entering maxim
Entering maxim
Entering maxim
enterinf minimum
enterinf minimum
enterinf minimum
enterinf minimum
enterinf minimum
enterinf minimum
enterinf minimum
Entering maxim
enterinf minimum
enterinf minimum
Entering maxim
enterinf minimum
enterinf minimum
enterinf minimum
enterinf minimum
Entering maxim
Enterin[...]<jupyter_text>Renaming the columns for better understanding..<jupyter_code>tempdf=tempdf.rename(columns={0: "MinTemp", 1: "MaxTemp",2: "Rainfall",3: "Evaporation",4:"Sunshine",5:"WindGustSpeed",
                      6:"WindSpeed",7:"Humidity",8:"Pressure",9:"Cloud",10:"Temp",11:"LocationInt",12:"WindGustDirInt",
                      13:"WindDirInt",14:"RainTodayInt",15:"RainTomorrowInt"})
tempdf1=tempdf1.rename(columns={0: "ID", 1: "MinTemp",2: "MaxTemp",3: "Rainfall",4:"Evaporation",5:"Sunshine",
                      6:"WindGustSpeed",7:"WindSpeed",8:"Humidity",9:"Pressure",10:"Cloud",11:"Temp",12:"LocationInt",
                      13:"WindGustDirInt",14:"WindDirInt",15:"RainTodayInt"})
tempdf1=tempdf1.drop(["ID"],axis=1)<jupyter_output><empty_output><jupyter_text>***Here we can visualize the correlation matrix which helps us to identify how each variable is affecting our decision variable RainTomorrowInt.***<jupyter_code>plt.figure(figsize=(20,10))
c= tempdf.corr()
sns.heatmap(c,cmap="BrBG",annot=True)
c<jupyter_output><empty_output><jupyter_text>Normalizing the values in both the training as well as the testing dataset as per algorithm's requirements. This function considers all the columns and normalizes it using inbuilt function<jupyter_code>min_max_scaler = preprocessing.MinMaxScaler()
np_scaled = min_max_scaler.fit_transform(tempdf)
df_normalized = pd.DataFrame(np_scaled, columns=tempdf.columns, index=tempdf.index)
df_normalized.describe()
min_max_scaler1 = preprocessing.MinMaxScaler()
np_scaled1 = min_max_scaler1.fit_transform(tempdf1)
df_normalized1 = pd.DataFrame(np_scaled1, columns=tempdf1.columns, index=tempdf1.index)
df_normalized1.describe()<jupyter_output><empty_output><jupyter_text>K-Means Clustering  HyperparametersThe K-Means model returned the best results with a recall of 0.77 and an accuracy of 0.73. ***number of clusters= 2 because the model is used for binary classification.n_init parameter=number of times the algorithm will run with different centroidsmax_iter parameter=maximum number of iterations of the k-means algorithm for a single run*** Using Kmeans function to implement the clustering algorithm<jupyter_code>tempdf=tempdf.rename(columns={0: "MinTemp", 1: "MaxTemp",2: "Rainfall",3: "Evaporation",4:"Sunshine",5:"WindGustSpeed",
                      6:"WindSpeed",7:"Humidity",8:"Pressure",9:"Cloud",10:"Temp",11:"LocationInt",12:"WindGustDirInt",
                      13:"WindDirInt",14:"RainTodayInt",15:"RainTomorrowInt"})
tempdf
x = tempdf.drop(['RainTomorrowInt','RainTodayInt'],axis = 1)
y = tempdf['RainTomorrowInt']
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.3, random_state=101)
kmeans = KMeans(n_clusters=2, n_init=25, max_iter=100, random_state=6)
kmeans.fit(x_train)
kmeans.__dict__.keys()
kmeans.labels_
pred = kmeans.predict(x_test)
kmeans=pred
kmeans
print(classification_report(y_test,pred))
pd.crosstab(y_test, pred, rownames=['True'], colnames=['Predicted'], margins=True)<jupyter_output>precision    recall  f1-score   support

         0.0       0.75      0.67      0.71      7872
         1.0       0.70      0.77      0.73      7722

    accuracy                           0.72     15594
   macro avg       0.72      0.72      0.72     15594
weighted avg       0.72      0.72      0.72     15594<jupyter_text>**Finally we've plotted an ROC curve with an AUC of 0.727**<jupyter_code>fpr,tpr,_ = roc_curve(y_test,pred)
def plot_roc_curve(fpr, tpr, label=None):

    plt.plot(fpr, tpr, linewidth=2, label=label)
    plt.plot([0, 1], [0, 1], 'k--')
    plt.axis([0, 1, 0, 1])                                   
    plt.xlabel('False Positive Rate (Fall-Out)', fontsize=16)
    plt.ylabel('True Positive Rate (Recall)', fontsize=16)   
    plt.grid(True)  
    

plt.figure(figsize=(8, 6))                     
plot_roc_curve(fpr, tpr)
print("AUC:{}".format(roc_auc_score(y_test, pred)))
plt.show()<jupyter_output>AUC:0.7227969283990625<jupyter_text>Using the Elbow method to estimate the value of number of clusters required to get the best performance for kmeans. As we can see in the graph that the joint of the elbow is at 2. The idea behind the elbow method is to implement k-means clustering on a given dataset for a range of values of k and for each value of k, calculate the sum of squared errors (SSE). Here, the optimal value for k is 2.<jupyter_code>distortions = []
K = range(1, 10)
for k in K:
    kmeanModel = KMeans(n_clusters=k).fit(tempdf)
    kmeanModel.fit(tempdf)
    distortions.append(sum(np.min(cdist(tempdf, kmeanModel.cluster_centers_, 'euclidean'), axis=1)) / tempdf.shape[0])
    
plt.plot(K, distortions, 'bx-')
plt.xlabel('k')
plt.ylabel('Distortion')
plt.title('The Elbow Method to Identify Optimal k')
plt.show()<jupyter_output><empty_output><jupyter_text>Using Hierarchical Agglomerative Clustering with the help of dendograms for explanation Hierarchical clustering is where you build a cluster tree (a dendrogram) to represent data, where each group(node) links to two or more successor groups.  Implementing the Dendogram on Training dataset:A dendogram is a type of tree diagram showing hierarchical clustering relationships between similar sets of data. They are frequently used in biology to show clustering between genes or samples, but they can represent any type of grouped data.<jupyter_code>from scipy.cluster.hierarchy import ward, dendrogram

linkage_matrix = ward(tempdf)

fig, ax = plt.subplots(figsize=(20, 10))
ax.grid(False)
ax.set_title('Cluster Dendrogram', fontsize = 25)
ax = dendrogram(linkage_matrix, orientation='top', labels=tempdf.index)
plt.xticks(fontsize=15)
plt.show()   
tempdf=tempdf.rename(columns={0: "MinTemp", 1: "MaxTemp",2: "Rainfall",3: "Evaporation",4:"Sunshine",5:"WindGustSpeed",
                      6:"WindSpeed",7:"Humidity",8:"Pressure",9:"Cloud",10:"Temp",11:"LocationInt",12:"WindGustDirInt",
                      13:"WindDirInt",14:"RainTodayInt",15:"RainTomorrowInt"})
tempdf
tempdf1.head(5)<jupyter_output><empty_output><jupyter_text>Implementing the Dendogram on Testing dataset<jupyter_code>linkage_matrix1 = ward(tempdf1)

fig, ax = plt.subplots(figsize=(20, 10))
ax.grid(False)
ax.set_title('Cluster Dendrogram', fontsize = 25)
ax = dendrogram(linkage_matrix1, orientation='top', labels=tempdf1.index)
plt.xticks(fontsize=15)
plt.show()  
linkage_matrix1 = ward(tempdf1)

fig, ax = plt.subplots(figsize=(20, 10))
ax.grid(False)
ax.set_title('Cluster Dendrogram', fontsize = 25)
ax = dendrogram(linkage_matrix1, orientation='top',truncate_mode="lastp",labels=tempdf1.index,p=2)
plt.xticks(fontsize=15)
plt.show()<jupyter_output><empty_output><jupyter_text>The ROC curve for HAC can be seen below, It has an auc of 0.647<jupyter_code>x = tempdf.drop(['RainTomorrowInt','RainTodayInt'],axis = 1)
y = tempdf['RainTomorrowInt']
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.3, random_state=101)
clustering = AgglomerativeClustering(distance_threshold=None, n_clusters=2,linkage = "ward").fit(x)

HAC=clustering.labels_<jupyter_output><empty_output><jupyter_text>We can see that we get recall value of 0.90 with an accuracy of 0.637<jupyter_code># Accuracy
lab = 1 - clustering.labels_
((lab == y).sum())/(lab == y).count()
print(confusion_matrix(y,lab))
print(classification_report(y,lab))<jupyter_output>[[12230 14229]
 [ 3588 21931]]
              precision    recall  f1-score   support

         0.0       0.77      0.46      0.58     26459
         1.0       0.61      0.86      0.71     25519

    accuracy                           0.66     51978
   macro avg       0.69      0.66      0.64     51978
weighted avg       0.69      0.66      0.64     51978<jupyter_text>We can see that the ROC curve has an AUC value of 0.642<jupyter_code>def plot_roc_curve(fpr, tpr, label=None):

    plt.plot(fpr, tpr, linewidth=2, label=label)
    plt.plot([0, 1], [0, 1], 'k--')
    plt.axis([0, 1, 0, 1])                                   
    plt.xlabel('False Positive Rate (Fall-Out)', fontsize=16)
    plt.ylabel('True Positive Rate (Recall)', fontsize=16)   
    plt.grid(True)  
    
fpr,tpr,_ = roc_curve(y,lab)
plt.figure(figsize=(8, 6))                     
plot_roc_curve(fpr, tpr)
print("AUC:{}".format(roc_auc_score(y, lab)))
plt.show()
tempdf.head(5)<jupyter_output><empty_output><jupyter_text>Decision Tree Classifer Decision tree learning is one of the predictive modeling approaches used in statistics, data mining and machine learning. It uses a decision tree to go from observations about an item to conclusions<jupyter_code>x=tempdf.drop(["RainTomorrowInt"],axis=1)
y=tempdf.RainTomorrowInt
clf=DecisionTreeClassifier()
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.3, random_state=1)
print(f"train data size is {x_train.shape}")
clf.fit(x_train, y_train)
y_pred = clf.predict(x_test)
clf.tree_.max_depth
y_pred
print(f"Accuracy: {round(metrics.accuracy_score(y_test, y_pred)*100)}%")
param_grid = {'criterion': ['gini', 'entropy'],
              'min_samples_split': [2, 10, 20],
              'max_depth': [5, 10, 20, 25, 30],
              'min_samples_leaf': [1, 5, 10],
              'max_leaf_nodes': [2, 5, 10, 20]}
grid = GridSearchCV(clf, param_grid, cv=10, scoring='accuracy')
grid.fit(x_train, y_train)
print(grid.best_score_)
for hps, values in grid.best_params_.items():
  print(f"{hps}: {values}")
num_vars = tempdf.select_dtypes(['int64', 'float64']).columns
x = tempdf[num_vars]
x=tempdf.drop(["RainTomorrowInt"],axis=1)
y = tempdf.RainTomorrowInt
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.3, random_state=16)

clf = DecisionTreeClassifier()
clf.fit(x_train, y_train)
y_pred = clf.predict(x_test)
DT=y_pred<jupyter_output><empty_output><jupyter_text>Performance Reporting<jupyter_code>print(f"Accuracy: {round(metrics.accuracy_score(y_test, y_pred)*100, 2)}%")
df_confusion = pd.crosstab(y_test, y_pred)
df_confusion.columns.name = "Pred"
df_confusion
print(metrics.classification_report(y_test, y_pred))
param_grid = {'criterion': ['gini', 'entropy'],
              'min_samples_split': [2, 10, 20, 30],
              'max_depth': [4, 5, 6, 10, 15, 20],
              'min_samples_leaf': [ 1, 5, 10],
              'max_leaf_nodes': [2, 5, 10, 20]}
grid = GridSearchCV(clf, param_grid, cv=5)
grid.fit(x_train, y_train)
bstrap = ShuffleSplit(n_splits=10, test_size=0.3, random_state=16)
grid_bstrap = GridSearchCV(clf, param_grid, cv=bstrap)
grid_bstrap.fit(x_train, y_train)<jupyter_output><empty_output><jupyter_text>Data testing, Prediction and Interpretation  Hyperparameters for best performing model<jupyter_code>print(f"Accuracy: {round(grid_bstrap.best_score_*100, 2)}%")
for key, value in grid_bstrap.best_params_.items():
  print(f"Hyperparameter: {key}; Value: {value}")
all_predictions=pd.DataFrame(list(zip(df1["ID"],kmeans,HAC,DT)), 
               columns =['ID', 'kmeans','HAC','DT']) 
all_predictions.to_csv("Predictions1.csv",index=False)<jupyter_output><empty_output><jupyter_text>Table for Reporting:<jupyter_code>#Model Comparison 
data = {'Model':['Manual tuning', 'Grid Search CV', 'Bootstrap resampling'],
        'Max Depth':[26,10,10],
        'min_samples_split':[2,2,2],
        'min_samples_leaf':[1,1,1],
        'max_leaf_nodes': [None,20,20],
        'Accuracy(%)':[79.00,79.35,79.31]
       }
pd.DataFrame(data)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1126011898956403, 'fraction_numerical': 0.03545303813517995, 'mean_word_length': 4.874248066456603, 'pattern_counts': {'":': 0, '<': 100, '<?xml version=': 0, '>': 100, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '26', 'dir': 'jupyter-structured-clean-dedup', 'id': '314368', 'license': 'MIT', 'n_tokens_mistral': 7458, 'n_tokens_neox': 6784, 'n_words': 1673, 'path': 'WeatherPrediction.ipynb', 'repo_name': 'rdx910/Predicting-weather', 'size': '20506'}