<jupyter_start><jupyter_text>EDA1. Impact of province on temperature using histogram (to understand trend of temperature per year in each province)==> It seems unusual dip of mean temperature in RJ and SP for the year 2004. Other than that, mean tempaerature seems to be distributed along the same range (approximately 20 degree celcius) through out four provinces.More than Province, it seems year and month has a more significant relationship with Temperature.2. Impact of elevation,lattitude and logitude on temperature (using scatter plot for upward or downward trend or random trend)==> It seems negative trend between elevation and temperature. It seems lattitude and logitude do not make much impact on temperature as mean values are. Temperature clustered around a particular range of lattitude and longitude.3. Impact of humidity and precipitation on temperature (using scatter plot for upward or downward or no trend)==> It seems a negative trend between humidity and temperature. Temprature more or less remains around 20 degree celcius even with significant increase in precipitation.4. Impact of Wind Speed and Wind Gust on Temperature. The analysis might also help to understand significance of Wind Direction on Temperature.==> It seems curvilinear relation between Wind Speed and Wind Gust and Temperature Distribution. Temperature clustered around a particular range for any level of increase in Wind Direction.5. Top 20 Temperature recorded weather station id wise for hr-day-month-yr wise. Bottom 20 Temperature recorded province wise and hr-day-month-yr wise.==> Just to have a look at Weather Station Id where extreme high and low temperatures are recorded along with it's year and month.6. Distribution of Outliers.==> For the columns where I could see some level outliers (above 5%), I might not worry at this moment basis the assumption at that very instance might have created that genuine spike. At a later point if error in prediction is high we can revisit this point of Outliers Treatment.<jupyter_code>#Teperature Distribution Province wise per year

dataset = weather_data_samp[weather_data_samp['prov_ES'] == 1].copy()
unique = dataset['yr'].unique()

mean_yr = []
for i in unique:
    mean_yr.append(round((dataset[dataset['yr'] == i]['temp'].mean()),2))
    
plt.bar(unique,mean_yr)
plt.title("For Provice ES Year Wise Mean Temperature Distribution")
plt.xlabel('Year')
plt.ylabel('Mean temperature for the year')
plt.show()
dataset = weather_data_samp[weather_data_samp['prov_MG'] == 1].copy()
unique = dataset['yr'].unique()

mean_yr = []
for i in unique:
    mean_yr.append(round((dataset[dataset['yr'] == i]['temp'].mean()),2))
    
plt.bar(unique,mean_yr)
plt.title("For Provice MG Year Wise Mean Temperature Distribution")
plt.xlabel('Year')
plt.ylabel('Mean temperature for the year')
plt.show()
dataset = weather_data_samp[weather_data_samp['prov_RJ'] == 1].copy()
unique = list(dataset['yr'].unique())

mean_yr = []
for i in unique:
    mean_yr.append(round((dataset[dataset['yr'] == i]['temp'].mean()),2))
    
plt.bar(unique,mean_yr)
plt.title("For Provice RJ Year Wise Mean Temperature Distribution")
plt.xlabel('Year')
plt.ylabel('Mean temperature for the year')
plt.show()
dataset = weather_data_samp[weather_data_samp['prov_SP'] == 1].copy()
unique = dataset['yr'].unique()

mean_yr = []
for i in unique:
    mean_yr.append(round((dataset[dataset['yr'] == i]['temp'].mean()),2))
    
plt.bar(unique,mean_yr)
plt.title("For Provice SP Year Wise Mean Temperature Distribution")
plt.xlabel('Year')
plt.ylabel('Mean temperature for the year')
plt.show()
#Overall Temperature Disctriution month wise across all provinces.

unique = dataset['yr'].unique()

mean_yr = []
for i in unique:
    mean_yr.append(round((dataset[dataset['yr'] == i]['temp'].mean()),2))
    
plt.bar(unique,mean_yr)
plt.title("Overall Year Wise Mean Temperature Distribution")
plt.xlabel('Year')
plt.ylabel('Mean temperature for the year')
plt.show()
#Teperature Distribution Province wise per month across years

dataset = weather_data_samp[weather_data_samp['prov_ES'] == 1].copy()
unique = dataset['mo'].unique()

mean_yr = []
for i in unique:
    mean_yr.append(round((dataset[dataset['mo'] == i]['temp'].mean()),2))
    
plt.bar(unique,mean_yr)
plt.title("For Provice ES Month Wise Mean Temperature Distribution")
plt.xlabel('Month')
plt.ylabel('Mean temperature for the month')
plt.show()
dataset = weather_data_samp[weather_data_samp['prov_MG'] == 1].copy()
unique = dataset['mo'].unique()

mean_yr = []
for i in unique:
    mean_yr.append(round((dataset[dataset['mo'] == i]['temp'].mean()),2))
    
plt.bar(unique,mean_yr)
plt.title("For Provice MG Month Wise Mean Temperature Distribution")
plt.xlabel('Month')
plt.ylabel('Mean temperature for the month')
plt.show()
dataset = weather_data_samp[weather_data_samp['prov_RJ'] == 1].copy()
unique = dataset['mo'].unique()

mean_yr = []
for i in unique:
    mean_yr.append(round((dataset[dataset['mo'] == i]['temp'].mean()),2))
    
plt.bar(unique,mean_yr)
plt.title("For Provice RJ Month Wise Mean Temperature Distribution")
plt.xlabel('Month')
plt.ylabel('Mean temperature for the month')
plt.show()
dataset = weather_data_samp[weather_data_samp['prov_SP'] == 1].copy()
unique = dataset['mo'].unique()

mean_yr = []
for i in unique:
    mean_yr.append(round((dataset[dataset['mo'] == i]['temp'].mean()),2))
    
plt.bar(unique,mean_yr)
plt.title("For Provice SP Month Wise Mean Temperature Distribution")
plt.xlabel('Month')
plt.ylabel('Mean temperature for the month')
plt.show()
#Overall Temperature Disctriution month wise across all provinces.

unique = dataset['mo'].unique()

mean_yr = []
for i in unique:
    mean_yr.append(round((dataset[dataset['mo'] == i]['temp'].mean()),2))
    
plt.bar(unique,mean_yr)
plt.title("Overall Month Wise Mean Temperature Distribution")
plt.xlabel('Month')
plt.ylabel('Mean temperature for the month')
plt.show()
# Impact of elevtion, lattitude and longitude on temperature

unique = weather_data_samp['elvt'].unique()

mean_temp = []
for i in unique:
    mean_temp.append(round((weather_data_samp[weather_data_samp['elvt'] == i]['temp'].mean()),2))


plt.scatter(unique,mean_temp)
plt.title("For Elevation and Temperature Distribution")
plt.xlabel('Elevation')
plt.ylabel('Temperature')
plt.show()
unique = weather_data_samp['lat'].unique()

mean_temp = []
for i in unique:
    mean_temp.append(round((weather_data_samp[weather_data_samp['lat'] == i]['temp'].mean()),2))


plt.scatter(unique,mean_temp)
plt.title("For Lattitude and Temperature Distribution")
plt.xlabel('Lattitude')
plt.ylabel('Temperature')
plt.show()
unique = weather_data_samp['lon'].unique()

mean_temp = []
for i in unique:
    mean_temp.append(round((weather_data_samp[weather_data_samp['lon'] == i]['temp'].mean()),2))


plt.scatter(unique,mean_temp)
plt.title("For Longitude and Temperature Distribution")
plt.xlabel('Longitude')
plt.ylabel('Temperature')
plt.show()
# Impact of Humidity and Precipitation on Temperature

unique = weather_data_samp['hmdy'].unique()

mean_temp = []
for i in unique:
    mean_temp.append(round((weather_data_samp[weather_data_samp['hmdy'] == i]['temp'].mean()),2))


plt.scatter(unique,mean_temp)
plt.title("For Humidity and Temperature Distribution")
plt.xlabel('Humidity')
plt.ylabel('Temperature')
plt.show()
unique = weather_data_samp['prcp'].unique()

mean_temp = []
for i in unique:
    mean_temp.append(round((weather_data_samp[weather_data_samp['prcp'] == i]['temp'].mean()),2))


plt.scatter(unique,mean_temp)
plt.title("For Precipitation and Temperature Distribution")
plt.xlabel('Precipitation')
plt.ylabel('Temperature')
plt.show()
# Impact of Wind Speed and Wind Gust on Temperature

unique = weather_data_samp['wdsp'].unique()

mean_temp = []
for i in unique:
    mean_temp.append(round((weather_data_samp[weather_data_samp['wdsp'] == i]['temp'].mean()),2))


plt.scatter(unique,mean_temp)
plt.title("For Wind Speed and Temperature Distribution")
plt.xlabel('Wind Speed')
plt.ylabel('Temperature')
plt.show()
unique = weather_data_samp['gust'].unique()

mean_temp = []
for i in unique:
    mean_temp.append(round((weather_data_samp[weather_data_samp['gust'] == i]['temp'].mean()),2))


plt.scatter(unique,mean_temp)
plt.title("For Wind Gust and Temperature Distribution")
plt.xlabel('Wind Gust')
plt.ylabel('Temperature')
plt.show()
unique = weather_data_samp['wdct'].unique()

mean_temp = []
for i in unique:
    mean_temp.append(round((weather_data_samp[weather_data_samp['wdct'] == i]['temp'].mean()),2))


plt.scatter(unique,mean_temp)
plt.title("For Wind Direction and Temperature Distribution")
plt.xlabel('Wind Direction')
plt.ylabel('Temperature')
plt.show()
# Top 20 highest temaperature recorded Weather Station wise for a aprticular year and month.

weather_data_samp[['wsid','yr','mo','temp']].sort_values('temp',axis=0,ascending=False).head(20)
# Bottom 20 highest temaperature recorded Weather Station wise for a aprticular year and month.

weather_data_samp[['wsid','yr','mo','temp']].sort_values('temp',axis=0,ascending=True).head(20)
# To identify outliers in the dataset

def get_outliers(self):
        a_list = []
        per_75 = np.percentile(self,75)
        per_25 = np.percentile(self,25)
        iqr = per_75-per_25
        tot_75 = per_75 + (1.5*iqr)
        tot_25 = per_25 - (1.5*iqr)
        a_list1 = np.array(self).tolist()
        for j in a_list1:
            if j > tot_75:
                a_list.append(j)
            if j < tot_25:
                a_list.append(j)
        return a_list

for i in weather_data_samp:
    outliers = get_outliers(weather_data_samp[i])
    print ("Total outliers in Column",weather_data_samp[i].name," =",len(outliers))
    print ("Percentatge of outliers in Column",weather_data_samp[i].name," =",round(((len(outliers)/weather_data_samp.shape[0])*100),2))
    print ("** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** **")<jupyter_output>Total outliers in Column wsid  = 2287
Percentatge of outliers in Column wsid  = 0.78
** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** **
Total outliers in Column elvt  = 0
Percentatge of outliers in Column elvt  = 0.0
** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** **
Total outliers in Column lat  = 4474
Percentatge of outliers in Column lat  = 1.53
** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** **
Total outliers in Column lon  = 2187
Percentatge of outliers in Column lon  = 0.75
** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** **
Total outliers in Column yr  = 513
Percentatge of outliers in Column yr  = 0.17
** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** **
Total outliers in Column mo  = 0
Percentatge of outliers in Column mo  = 0.0
** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** **
Total outliers in Column da  = 0
Percentatge of outliers in Column da  = 0.0
** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** **
Total outliers in Column hr  = 0
Percentatge of outliers in Column hr  = 0.0
[...]<jupyter_text>Feature Enginerring1. Converting categorical column to integer by multiplying with mean of temperature<jupyter_code>#Transforming categorical column into integer

weather_data_samp['wsid_temp_mean'] = weather_data_samp.groupby('wsid')['temp'].transform('mean')

#and dropping original wsid column

weather_data_samp.drop('wsid', axis=1, inplace=True)
weather_data_samp.columns<jupyter_output><empty_output><jupyter_text>Feature Selection<jupyter_code># Correlation with Temperature

weather_data_samp.corrwith(weather_data_samp['temp'])
# Drop features with multicolinearity

fig = plt.figure(figsize=(10, 10))
sns.heatmap(weather_data_samp[['elvt','lat','lon','yr','mo','da','hr']].corr(), annot=True)
plt.title('Correlation heatmap', fontsize=10, color='blue')
plt.show()

# longitude to drop since comparatively lattitude more corelated to temperature
fig = plt.figure(figsize=(10, 10))
sns.heatmap(weather_data_samp[['smax','smin','tmax','tmin','dmax','dmin','hmax','hmin']].corr(), annot=True)
plt.title('Correlation heatmap', fontsize=10, color='blue')
plt.show()

#between smin and smax, any one can be dropped hence I will drop smin
#between tmin and tmax, any one can be dropped hence I will drop tmin
#between dmin and dmax, any one can be dropped hence I will drop tmin
#but since even t, d and s are also highly corelated (greater than 0.6) will keep only tmax
#between hmin and hmax, any one can be dropped hence I will drop hmin
fig = plt.figure(figsize=(10, 10))
sns.heatmap(weather_data_samp[['prcp','stp','gbrd','dewp','hmdy','wdsp','wdct','gust','wsid_temp_mean']].corr(), annot=True)
plt.title('Correlation heatmap', fontsize=10, color='blue')
plt.show()

#between hmdy and dewp, I will keep dewp since it is more corelated with temperature and I will drop hmdy
# dropping columns

weather_data_samp.drop(['lon', 'smax', 'smin', 'dmax', 'tmin', 'dmin', 'hmin', 'hmdy'], axis=1, inplace=True)
weather_data_samp.columns
fig = plt.figure(figsize=(15, 15))
sns.heatmap(weather_data_samp[['elvt', 'lat', 'yr', 'mo', 'da', 'hr', 'prcp', 'stp', 'gbrd', 'dewp', 'tmax', 'hmax', 'wdsp', 'wdct', 'gust']].corr(), annot=True)
plt.title('Correlation heatmap', fontsize=10, color='blue')
plt.show()

#between wdsp and gust, I will drop wdsp
#will also remove dewp and stp
# dropping columns

weather_data_samp.drop(['wdsp', 'stp', 'dewp'], axis=1, inplace=True)
weather_data_samp.columns
fig = plt.figure(figsize=(15, 15))
sns.heatmap(weather_data_samp[['elvt', 'lat', 'yr', 'mo', 'da', 'hr', 'prcp', 'gbrd', 'tmax', 'hmax', 'wdct', 'gust']].corr(), annot=True)
plt.title('Correlation heatmap', fontsize=10, color='blue')
plt.show()
fig = plt.figure(figsize=(15, 15))
sns.heatmap(weather_data_samp[['prov_ES', 'prov_MG', 'prov_RJ', 'prov_SP']].corr(), annot=True)
plt.title('Correlation heatmap', fontsize=10, color='blue')
plt.show()
# Create X and y

X = weather_data_samp.drop(['temp'], axis=1)
y = weather_data_samp['temp']
# Use RFE for feature selection for Linear Regression Algorithm
from sklearn.feature_selection import RFE
lr = LinearRegression()
rfe = RFE(lr, 15)
rfe.fit(X, y)
sel_cols = []
for a, b, c in zip(rfe.support_, rfe.ranking_, X.columns):
    print(a, b, c)
    if b == 1:
        sel_cols.append(c)
print(sel_cols)
# Creating new DF of X

X_sel = X[sel_cols]<jupyter_output><empty_output><jupyter_text>Preparing ModelR-squared is a goodness-of-fit measure for linear regression models. This statistic indicates the percentage of the variance in the dependent variable that the independent variables explain collectively. R-squared measures the strength of the relationship between your model and the dependent variable on a convenient 0 – 100% scale. After fitting a regression model, you need to determine how well the model fits the data. The RMSE is the square root of the variance of the residuals. It indicates the absolute fit of the model to the data–how close the observed data points are to the model’s predicted values. Whereas R-squared is a relative measure of fit, RMSE is an absolute measure of fit. As the square root of a variance, RMSE can be interpreted as the standard deviation of the unexplained variance, and has the useful property of being in the same units as the response variable. Lower values of RMSE indicate better fit. RMSE is a good measure of how accurately the model predicts the response, and it is the most important criterion for fit if the main purpose of the model is prediction.<jupyter_code># Split data in to train and test

X_train, X_test, y_train, y_test = train_test_split(X_sel, y, train_size=0.7, random_state=7)
lr.fit(X_train, y_train)
y_pred = lr.predict(X_test)
print ("R squared value =",lr.score(X_train,y_train))
print ("RMSE value =",rmse(y_pred,y_test))
# Use RFE for feature selection for RandomForest Regression Algorithm
rf = RandomForestRegressor()
rfe = RFE(rf, 15)
rfe.fit(X, y)
sel_cols = []
for a, b, c in zip(rfe.support_, rfe.ranking_, X.columns):
    print(a, b, c)
    if b == 1:
        sel_cols.append(c)
print(sel_cols)
# Split data in to train and test

X_train, X_test, y_train, y_test = train_test_split(X_sel, y, train_size=0.7, random_state=7)
rf.fit(X_train, y_train)
y_pred = rf.predict(X_test)
print ("R squared value =",rf.score(X_train,y_train))
print ("RMSE value =",rmse(y_pred,y_test))<jupyter_output>/Users/pritigirishvankudre/anaconda3/lib/python3.7/site-packages/sklearn/ensemble/forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.
  "10 in version 0.20 to 100 in 0.22.", FutureWarning)
{'fraction_non_alphanumeric': 0.14448209099709583, 'fraction_numerical': 0.013068731848983543, 'mean_word_length': 5.4490831057354665, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 22, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '101815', 'license': 'MIT', 'n_tokens_mistral': 5783, 'n_tokens_neox': 5254, 'n_words': 1587, 'path': 'Hackathon_ML Sample Problems/Weather Dataset/Weather Dataset Prediction_Feature Engineering_Selection.ipynb', 'repo_name': 'girishvankudre/hackathon_ml_sample', 'size': '16528'}