<jupyter_start><jupyter_text>Semester Project - Machine LearningProject made by <NAME>SI - 2021  Dataset descriptionThe current dataset contains parameters about weather in Szeged, a city located in Hungary, between 2006 and 2016. It contains the following columns: - **Formatted Date**- **Summary**  (weather forecast by hour)- **Precip Type**- **Temperature** (in Celsius Degrees)- **Apparent Temperature** (in Celsius Degrees) - It is the  temperature perceived by humans- **Humidity**- **Wind Speed** (in km/h)- **Wind Bearing** (degrees)- **Visibility** (km)- **Loud Cover**- **Pressure** (in millibars)- **Daily Summer** (weather forecast by day)All the data contained into the dataset were taken every hour.Also, there are some columns which summarize how was the weather at a precise hour.Available at: [*Weather in Szeged*](https://www.kaggle.com/budincsevity/szeged-weather/discussion).  Preliminar operations Making sure not losing the job done<jupyter_code>%autosave 25<jupyter_output><empty_output><jupyter_text>Load the cell below to import all dependencies needed to run the notebook<jupyter_code>#Various imports
import numpy as np
import pandas as pd

from sklearn.linear_model import LinearRegression
from sklearn.linear_model import Ridge
from sklearn.linear_model import Lasso
from sklearn.linear_model import ElasticNet

from sklearn.neighbors import KNeighborsClassifier

from sklearn.svm import SVC

from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import train_test_split

from sklearn.metrics import mean_squared_error
from sklearn.metrics import confusion_matrix
from sklearn.metrics import classification_report
from sklearn.metrics import roc_curve
from sklearn.metrics import auc

from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import LabelEncoder
from sklearn.preprocessing import OneHotEncoder

import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
#Defining portion for test pahse
testPortion = 0.4<jupyter_output><empty_output><jupyter_text>First step: we have to import the dataset into the notebook and save it into a Dataframe. Then, we show the first rows just to get a first view of it.<jupyter_code>#Load the dataset from memory 
dfPath = "./resources/weatherHistory.csv"
resPath = "./results/"
df = pd.read_csv(dfPath)
#Print dataset's head      
df.head()
#Printing features' information
df.info()
#Getting rows and columns' number
df.shape
#Printing some feature's information
df.describe()<jupyter_output><empty_output><jupyter_text>Prepocessing phase Before starting doing some analyses, it is important to prepare the dataset during the preprocessing phase. For example, we can delete irrlevant columns (more details below) and transform discrete features into a more adaptable format: There are some columns which are particulary useless. For instance, **Loud Cover** data are full of 0's and that makes this column so irrelevant. The same principle is valid for the string(**Daily Summary** and **Summary**) that tells how it's the weather every hour. \<jupyter_code>df = df.drop(['Daily Summary', 'Summary', 'Loud Cover'], axis = 1)
df = df.rename(columns = {"Temperature (C)":"Temperature",
                          "Wind Speed (km/h)":"Wind Speed",
                           "Apparent Temperature (C)":"Apparent Temperature",
                            "Visibility (km)":"Visibility",
                            "Wind Bearing (degrees)":"Wind Bearing",
                            "Pressure (millibars)":"Pressure"})

df.columns<jupyter_output><empty_output><jupyter_text>Now, we are going to take a quick look about data distribution and how features are correlated with each other. Seeing the table below we can start thinking what kinds of analyses could be interesting to make.<jupyter_code>#Print data distribution
df.hist(bins = 50, figsize = (10,25))
plt.show()
df.corr()<jupyter_output><empty_output><jupyter_text>In order to avoid to have **nan**, we convert this values in column **Precip Type** into other<jupyter_code>df = df.replace(np.nan, 'other', regex=True)
set(df["Precip Type"].values)<jupyter_output><empty_output><jupyter_text>For further utility we print the box plot of the precitapion types taken hourly<jupyter_code>sns.boxplot(x =  df["Humidity"],y = df["Precip Type"])
plt.title("PrecipatioN Type Box Plot")
plt.show()
#Optional: parse the date into a proper format and extract the respective Year,Month,Day and HouR
df['Formatted Date'] = pd.to_datetime(df['Formatted Date'], format='%Y-%m-%d %H:%M:%S.%f %z') 
df['Year'] = df['Formatted Date'].apply(lambda x: x.year)
df['Month'] = df['Formatted Date'].apply(lambda x: x.month)
df['Day'] = df['Formatted Date'].apply(lambda x: x.day)
df['Hour'] = df['Formatted Date'].apply(lambda x: x.hour)
df<jupyter_output><empty_output><jupyter_text>The regressions and classificatons done below were done using **scikit-learn**, a powerful library in Python used for Machine Learnin and Statistics. Before see the results that I have obtained, it's necessary to split the dataset into two differents:- Train set: used to train all classifiers and regressors;- Test set: used to tests all classifers and regressors prepared;<jupyter_code>#Splitting the dataset into train, and test dataset
[dfTrain,dfTest] = train_test_split(df.drop(['Formatted Date','Year','Month','Day'],axis=1),random_state=1234,test_size=testPortion)<jupyter_output><empty_output><jupyter_text>Predicting Temperature  basing on Humidity First of all let's start doing a simple Linear Regression. Our aim is to minimize as soon as possible the **RMSE** (Residual  Minimum Squared Error) computed as the mean of the differences between real and predicted values. Let's find out if **Temperature** and **Humidity** are correllated each other.<jupyter_code>xTrain = dfTrain["Humidity"].values
xTest  = dfTest["Humidity"].values
yTrain = dfTrain["Temperature"].values
yTest  = dfTest["Temperature"].values

xTrain = np.reshape(xTrain,(-1,1))
xTest = np.reshape(xTest,(-1,1))
linReg = LinearRegression()
linReg.fit(xTrain,yTrain)
#Print reg params
print(f"Intercepts:  {linReg.intercept_}")
print(f"Coefficients:  {linReg.coef_}")
yTestPredicted = linReg.predict(xTest)
plt.scatter(xTrain,yTrain)
plt.xlabel("Humidity")
plt.ylabel("Temperature")
plt.title("Correlation between Temperature And Humidity")
plt.plot(xTest,yTestPredicted,color="orange")
plt.show()
print("RMSE test set: ",np.sqrt(mean_squared_error(yTest,yTestPredicted)))
print("R^2 score: ",linReg.score(xTest,yTest))<jupyter_output>RMSE test set:  7.393057970127397
R^2 score:  0.3990707303279296<jupyter_text>The $R^2$, is the proportion of the variance in the dependent variable that is predictable from the independent variable(s). It is a statist that tell us the quality of a regressorThe chart above shows the best "line" that approxes all the points. Unfortunatley we have a score of 40%, so it is not a good regression. It seems that **Humidity** pass perfectly trough the points, but this is an illusion because the chart is fitted with a lot of points.  Predicting Apparent Temperature every hourThe apparent temperature is the temperature perceived by humans, caused by the combined effect of air temperature, relative temperature and wind speed. Mostly, it is applied on outdoor perceived temperature. Is it that these features have a great correlation? \Let's find out.<jupyter_code>#Take data needed
xTrain = dfTrain[["Temperature","Humidity","Wind Speed"]].values
yTrain = dfTrain["Apparent Temperature"].values

xTest = dfTest[["Temperature","Humidity","Wind Speed"]].values
yTest = dfTest["Apparent Temperature"].values
#Prepare Linear Regressior - Printig the best values found
linReg = LinearRegression()
linReg.fit(xTrain,yTrain)
print(f"Intercepts:  {linReg.intercept_}")
print(f"Coefficients:  {linReg.coef_}")
#Checking how regressior behaviours with dfTest
yTrainPredicted = linReg.predict(xTrain)
RMSETrain = np.sqrt(mean_squared_error(yTrain,yTrainPredicted))
print(f"RMSE train set: {RMSETrain}")
yTestPredicted = linReg.predict(xTest)
#Show errors distribution
errors = np.abs(yTestPredicted - yTest)
plt.figure()
plt.title("Error distribution - Apparent Temperature")
plt.hist(x = errors, bins = 50)
plt.show()
#Show trend between real and predicted (based on a sample of 1500 units)
plt.figure(figsize=(14, 4))
plt.title("Comparison between real and estimate data - Apparent Temperature")
plt.plot(yTest[0:1500], label='Real')
plt.plot(yTestPredicted[0:1500], label='Prediction')
plt.legend(bbox_to_anchor=(1.02, 1), loc='upper left', borderaxespad=0.)
plt.show()
#Try to predict "real" data
RMSETest = np.sqrt(mean_squared_error(yTest,yTestPredicted))
R2Test = linReg.score(xTest,yTest)
print(f"RMSE score test: {RMSETest}")
print(f"R2 score test: {R2Test}")<jupyter_output>RMSE score test: 1.080129952197552
R2 score test: 0.989762960917948<jupyter_text>In conclusion, it's a evidence that they have a great correlation. The regression is abled to explain the 98% of all instances. So, there is no reason to try improving its quality through other algorithms. We can also save result in a txt file executing the cell below.<jupyter_code>#Save prediction's data
np.savetxt(resPath + "Apparent Temperature - Predictions",yTestPredicted)<jupyter_output><empty_output><jupyter_text>Predicting Atmospheric Pressure - Ridge - Lasso - Elastic Net RegressionLet's try predicting Pressure using all relevant columns. This predictions will be done using Ridge and Lasso regression.<jupyter_code>targetColumn = ["Pressure"]
predictors = list(set(list(df.drop(['Formatted Date','Month','Day','Year','Hour','Precip Type'],axis = 1))) 
                  - set(targetColumn))
alpha = [1e-15, 1e-10, 1e-8, 1e-4, 1e-3,1e-2, 1, 5, 10]
params = {'alpha': alpha}

Xs = dfTrain[predictors].values
Ys = dfTrain[targetColumn].values

xTest = dfTest[predictors].values
yTest = dfTest["Pressure"].values<jupyter_output><empty_output><jupyter_text>We define the number of folds used for **GridSearchCV** . In thsi case we define *cv* as 5.<jupyter_code>cv = 5<jupyter_output><empty_output><jupyter_text>Ridge Regression<jupyter_code>ridge = Ridge()
ridgeReg = GridSearchCV(ridge,params,scoring='neg_mean_squared_error', cv = cv, verbose=1)
ridgeReg.fit(Xs,Ys)
ridgeReg.best_params_
ridgeReg.best_score_
yTestRidgePredicted = ridgeReg.predict(xTest)
ridgeError = np.sqrt(mean_squared_error(yTest,yTestRidgePredicted))
print(f"RMSE score: {ridgeError}")<jupyter_output>RMSE score: 116.52281301076783<jupyter_text>Lasso Regression<jupyter_code>lasso = Lasso()
lassoReg = GridSearchCV(lasso,params,scoring='neg_mean_squared_error', cv= cv, verbose=1)
lassoReg.fit(Xs,Ys)
lassoReg.best_params_
lassoReg.best_score_
yTestLassoPredicted = lassoReg.predict(xTest)
lassoError = np.sqrt(mean_squared_error(yTest,yTestLassoPredicted))
print(f"RMSE score: {lassoError}")<jupyter_output>RMSE score: 116.52288548521467<jupyter_text>We can observe that Ridge and Lasso via **GridSearchCV** obtain a sligthly different RMSE score (**Lasso** regression has an alpha less than **Ridge** Regression)  Elastic Net Regression The **Elastic Net** is a regularized regression method that linearly combines the L1 and L2 penalties of the lasso and ridge methods<jupyter_code>elasticNet = ElasticNet()
elasticReg = GridSearchCV(elasticNet,params,scoring="neg_mean_squared_error",cv = cv, verbose = 1)
elasticReg.fit(Xs,Ys)
elasticReg.best_params_
elasticReg.best_score_
yTestElasticNetPredicted = elasticReg.predict(xTest)
#Print ElasticNet error
elasticError = np.sqrt(mean_squared_error(yTest,yTestElasticNetPredicted))
print(f"RMSE score: {elasticError}")<jupyter_output>RMSE score: 116.52265910153984<jupyter_text>Via **ElasticNet** regolarization we obtain a small alpha than **Lasso**, but bigger than **Ridge**  Predicting Precip type basing on Humidity - K NeighborsClassifier We start with classification analysis. Classification is a supervised learning concept which basically categorizes a set of data into classes.<jupyter_code>#Drop Precip Type columns with other value cause we have few samples tho predict
df = df.drop(df[df["Precip Type"] == "other"].index)
set(df["Precip Type"])
#Divide again the dataset int train and test
dfTrain,dfTest = train_test_split(df,random_state=1234,test_size= testPortion)
xTrain = dfTrain["Humidity"].values
yTrain = dfTrain["Precip Type"].values
xTest = dfTest["Humidity"].values
yTest = dfTest["Precip Type"].values
#Normalize Humidity values
scaler = StandardScaler()

xTrain = np.reshape(xTrain,(-1,1))
xTest = np.reshape(xTest,(-1,1))

scaler.fit(xTrain)
xTrainScaled = scaler.transform(xTrain)
xTestScaled = scaler.transform(xTest)
weatherClassifier =  KNeighborsClassifier(n_neighbors = 3)
weatherClassifier.fit(xTrainScaled,yTrain)
yWeatherPredicted = weatherClassifier.predict(xTest)<jupyter_output><empty_output><jupyter_text>Looking the confusion matrix we can see that we have an high prediction about  **rain** category. Instead **snow** category is not predicted so good cause we haven't got enough samples. Now, we want to see another statistic: the **ROC** curve. An ROC curve (receiver operating characteristic curve) is a graph showing the performance of a classification model at all classification thresholds. This curve plots two parameters:    True Positive Rate    False Positive Rate    But before, via **LabelEncoder** we have to trasform the categoric variable(Precip Type) into discrete values.<jupyter_code>#Printing some statistic like Confusion Matrix and a Classification Report
cf = confusion_matrix(yTest,yWeatherPredicted)
print("Confusion matrix: ")
print(cf)
print(classification_report(yTest,yWeatherPredicted))
print(f"Accuracy: {weatherClassifier.score(xTestScaled,yTest)}")
sns.heatmap(cf, annot = True)
le = LabelEncoder()
yTestDiscreteValues = le.fit_transform(yTest)
yTestDiscreteValues<jupyter_output><empty_output><jupyter_text>After the conversion done above, the variabile becomes binary with the following meanings:- 0 stands for **rain**- 1 stands for **snow**Referring also to **ROC curve** it is alsto interesting look the **AUC**.Instead, **AUC** stands for "Area under the ROC Curve." That is, AUC measures the entire two-dimensional area underneath the entire ROC curve (think integral calculus) from (0,0) to (1,1).<jupyter_code>#Compute ROC and AUC
probs =  weatherClassifier.predict_proba(xTest)
preeds = probs[:,1]
fpr,tpr,threshold = roc_curve(yTestDiscreteValues,preeds)

rocAuc = auc(fpr,tpr)
#Plot ROC curve
plt.title('Receiver Operating Characteristic with K =3')
plt.plot(fpr, tpr, 'b', label = 'AUC = %0.2f' % rocAuc)
plt.legend(loc = 'lower right')
plt.plot([0, 1], [0, 1],'r--')
plt.xlim([0, 1])
plt.ylim([0, 1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.show()<jupyter_output><empty_output><jupyter_text>The chart shows that the relaiabilty of the classifier is around 50%.\Now we try to predict  the best case, unless  if there would be another one. We start computing the classifier error for each *K* until 34<jupyter_code>#CELL A BIT SLOW TO RUN
errorsClassifiers = list()
for i in range (1,35):
    weatherClassifier =  KNeighborsClassifier(n_neighbors=i)
    weatherClassifier.fit(xTrainScaled,yTrain)
    yPredict = weatherClassifier.predict(xTest)
    errorsClassifiers.append(np.mean(yTest != yPredict))
#Plot error classifier  for each K
plt.figure(figsize =(14,6))
plt.plot(range(1,35), errorsClassifiers, color = "green", linestyle="dashed", marker="o", 
         markerfacecolor='blue', markersize=9)
plt.title("Error Rate K Value")
plt.xlabel("K value")
plt.ylabel("K mean error")<jupyter_output><empty_output><jupyter_text>Observing the chart we can figure out that the best *K* is 12, but anyway it is not a good classifier. So, now we want trying improve our classificator using SVM.  Predicting Precip type basing on Humidity - SVM At last, we want to improve the analyse done before using **SVM** (support vector machine). They are supervised learning models with associated learning algorithms that analyze data for classification and regression analysis. Before, we define a functions that compute the bet iperparamters using **GridSearchCV**. To make a comparison we want to use two different kernels:- Linear- RBF Instead, an other function plot a confusion matrix for each **SVM** that we use. .Execute the cell to load the two functions<jupyter_code>def compute_SVM(classfier,params,n_folds,Xs,Ys):
    gscv = GridSearchCV(classfier,params,cv = n_folds)
    gscv.fit(Xs,Ys)
    print("Params combination: \n",gscv.cv_results_['params'])
    print('Avg accuracy per combination:\n', gscv.cv_results_['mean_test_score'])
    print('Best combination:\n', gscv.best_params_)
    print('Avg accuracy of best combination: {}'.format(gscv.best_score_))
    return gscv
def print_confusion_matrix(classifier,yTest,yTestPredicted):
    cf = confusion_matrix(yTest,yTestPredicted)
    print(f"Confusion matrix:")
    print(cf)
    sns.heatmap(cf,annot = True)
#Prepare SVC's 
n_folds = 3
clsLinear = SVC()
clsRBF = SVC()

paramGridLinearSVC = [{'kernel': ['linear'], 'C': [1, 5, 10]}]
paramGridRBFSVC =  [{'kernel': ['rbf'], 'C': [1, 5, 10], 'gamma': [0.1, 0.01]}]<jupyter_output><empty_output><jupyter_text>Linear Clustering<jupyter_code>#CELL A BIT SLOW TO RUN
gscvLinear = compute_SVM(clsLinear,paramGridLinearSVC,n_folds,xTrain,yTrain)
yTestDiscreteValues = np.reshape(yTestDiscreteValues,(-1,1))
yLinearPredicted = gscvLinear.predict(yTestDiscreteValues)
print_confusion_matrix(gscvLinear,yTest,yLinearPredicted)<jupyter_output>Confusion matrix:
[[34051     0]
 [ 4324     0]]<jupyter_text>RBF Clustering<jupyter_code>#CELL A BIT SLOW TO RUN 
gscvRBF = compute_SVM(clsRBF,paramGridRBFSVC,n_folds,xTrain,yTrain)
yRBFPredicted = gscvRBF.predict(xTest)
print_confusion_matrix(gscvLinear,yTest,yRBFPredicted)<jupyter_output>Confusion matrix:
[[34051     0]
 [ 4324     0]]
{'fraction_non_alphanumeric': 0.10559456146216427, 'fraction_numerical': 0.016215312604480107, 'mean_word_length': 6.245458215583367, 'pattern_counts': {'":': 6, '<': 87, '<?xml version=': 0, '>': 87, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '381319', 'license': 'MIT', 'n_tokens_mistral': 6033, 'n_tokens_neox': 5564, 'n_words': 1753, 'path': 'weather_analysis.ipynb', 'repo_name': 'nikgreg99/weather-analysis-machine-learning', 'size': '17959'}