<jupyter_start><jupyter_text>General EDAOur EDA presents our processed data by filtering out null values and breaking down the data into different regions. With that, we would like to point out that the data below is a summarized version of our individual EDA to simplify the results that we would like to highlight. For a more detailed version of each data, it is available on our individual EDA in our folders, and each section is specified on whose EDA you can visit to see a deeper breakdown of the data. --- Method Chain 1 (Loading and Filtering Data)Done by JamesThe method chain loads the necessary impports and dataset into our Notebook.<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pylab as plt
import seaborn as sns
import datetime as dt
import pandas_profiling as pdp<jupyter_output><empty_output><jupyter_text>*Cleaning and Filtering the Dataset*The code below loads the CSV file using Pandas and drops all the null values in the data to keep helpful information only.Furthermore, the first method chain code is used to replace the initial df value (dataset) with the new dataset.<jupyter_code>#Method Chain 1
def load_and_process(url_or_path_to_csv_file):
    # Method Chain 1 (Load data, deal with missing data, and making data readable)
    df1 = (
        pd.read_csv(url_or_path_to_csv_file)
        .dropna()
        .rename(columns ={"LOCAL_DATE":"DATE"})
        .sort_values("DATE", ascending=False)
        .reset_index(drop = True)
        
      )
    return df1
df = load_and_process('Canadian_climate_history (1970-2020).csv')
df<jupyter_output><empty_output><jupyter_text>Method Chain 2 (Grouping the Columns)Done by James *Grouping the different cities into regions to simplify the data*Our original data presented the data for each city, leaving the dataset to be heavy with information. To simplify the dataset, we decided to group the cities into six different regions: - British Columbia - The Prairie Provinces- Ontario - Quebec - The Atlantic Provinces - The Northern ProvincesBy doing so, we will reduce the number of columns to make our data analysis easier. We combined the cities into their regions and found the region's average temperature and precipitation by combining data from the respective cities. Afterwards, we updated the new df value to the grouped columns for subsequent use.<jupyter_code>#Method Chain 2 (Creating new columns for different regions of Canada)
def group_columns(df):
    df2 = (
        #create new columns that take mean temperature and percipitation from Atlantic Provinces 
        df
        .assign(TEMPERATURE_ATLANTIC = (df.iloc[:, [5, 7, 17]].mean(axis=1)).round(decimals=1))
        .assign(PRECIPITATION_ATLANTIC = (df.iloc[:, [6, 8, 18]].mean(axis=1)).round(decimals=1))
        
        #create new columns that take mean temperature and percipitation from Prairie provinces 
        .assign(TEMPERATURE_PRAIRIES = (df.iloc[:, [1, 3, 15, 25]].mean(axis=1)).round(decimals=1))
        .assign(PRECIPITATION_PRAIRIES = (df.iloc[:, [2, 4, 16, 26]].mean(axis=1)).round(decimals=1))
        
        #create new columns that take mean temperature and percipitation from cities in Ontario and merge into single column
        .assign(TEMPERATURE_ONTARIO = (df.iloc[:, [11, 19]].mean(axis=1)).round(decimals=1))
        .assign(PRECIPITATION_ONTARIO = (df.iloc[:, [12, 20]].mean(axis=1)).round(decimals=1))
        
         #create new columns that take mean temperature and percipitation from cities in Quebec and merge into single column
        .assign(TEMPERATURE_QUEBEC = (df.iloc[:, [9, 13]].mean(axis=1)).round(decimals=1))
        .assign(PRECIPITATION_QUEBEC = (df.iloc[:, [10, 14]].mean(axis=1)).round(decimals=1))
        
        #dropping columns that were amalgimated into the means
        .drop(columns = ['MEAN_TEMPERATURE_CALGARY', 'TOTAL_PRECIPITATION_CALGARY', 'MEAN_TEMPERATURE_EDMONTON', 'TOTAL_PRECIPITATION_EDMONTON', 
                         'MEAN_TEMPERATURE_HALIFAX', 'TOTAL_PRECIPITATION_HALIFAX', 'MEAN_TEMPERATURE_MONCTON', 'TOTAL_PRECIPITATION_MONCTON',
                        'MEAN_TEMPERATURE_SASKATOON', 'TOTAL_PRECIPITATION_SASKATOON', 'MEAN_TEMPERATURE_STJOHNS', 'TOTAL_PRECIPITATION_STJOHNS',
                        'MEAN_TEMPERATURE_WINNIPEG', 'TOTAL_PRECIPITATION_WINNIPEG', 'MEAN_TEMPERATURE_OTTAWA', 'TOTAL_PRECIPITATION_OTTAWA',
                        'MEAN_TEMPERATURE_TORONTO', 'TOTAL_PRECIPITATION_TORONTO', 'MEAN_TEMPERATURE_MONTREAL', 'TOTAL_PRECIPITATION_MONTREAL',
                        'MEAN_TEMPERATURE_QUEBEC', 'TOTAL_PRECIPITATION_QUEBEC'])
        
        #renaming columns to meet new location based naming
        .rename(columns ={"MEAN_TEMPERATURE_VANCOUVER":"TEMPERATURE_BRITISH_COLUMBIA"})
        .rename(columns ={"TOTAL_PRECIPITATION_VANCOUVER":"PRECIPITATION_BRITISH_COLUMBIA"})
        .rename(columns ={"MEAN_TEMPERATURE_WHITEHORSE":"TEMPERATURE_NORTHERN"})
        .rename(columns ={"TOTAL_PRECIPITATION_WHITEHORSE":"PRECIPITATION_NORTHERN"})
    )
        
    return df2
df = group_columns(df)
df<jupyter_output><empty_output><jupyter_text>EDA for Temperature --- Filtering Temperature Data by MonthDone by Nadine *Changing the date value to simplify filter process*Here, we changed the date value to remove the time and used the melt() function to create a dataset to create the graph. Then we created a graph that presents the different temperatures of the different regions in each month from 1970 until 2020. The dataset below only shows the changed date format<jupyter_code>df = df.assign(DATE = lambda x : x.DATE)
df["DATE"] = pd.to_datetime(df["DATE"], utc = True).dt.date
df.sort_values(by=['DATE'], inplace=True)
df = df.assign(YEAR = lambda x : x.DATE)
df["YEAR"] = pd.to_datetime(df["YEAR"], utc = True).dt.year
df = df[df['YEAR'] < 2020]
df
#Splitting Up locations for Temperature
britishcolumbia = pd.melt(df, id_vars ='DATE', value_vars = 'TEMPERATURE_BRITISH_COLUMBIA')
northern = pd.melt(df, id_vars ='DATE', value_vars = 'TEMPERATURE_NORTHERN')
atlantic = pd.melt(df, id_vars ='DATE', value_vars = 'TEMPERATURE_ATLANTIC')
praries = pd.melt(df, id_vars ='DATE', value_vars = 'TEMPERATURE_PRAIRIES')
ontario = pd.melt(df, id_vars ='DATE', value_vars = 'TEMPERATURE_ONTARIO')
quebec = pd.melt(df, id_vars ='DATE', value_vars = 'TEMPERATURE_QUEBEC')
combined = [britishcolumbia, northern, atlantic, praries, ontario, quebec]
df2 = pd.concat(combined)
df2 = df2.rename(columns ={"variable":"LOCATION"}).rename(columns ={"value":"TEMPERATURE"})
df2 = df2.assign(YEAR = lambda x : x.DATE)
df2["YEAR"] = pd.to_datetime(df2["YEAR"], utc = True).dt.year
df2 = df2[df2['YEAR'] < 2020]
df2<jupyter_output><empty_output><jupyter_text>Analysis of Temperature---**Research Question:** Are there changes occuring in temperature patterns in our climate zones and which zones have the highest and lowest average temperatures? For the temperature graph, we only included the first, sixth, and last month of the graph in our final EDA to show the temperature trend throughout the year in Canada. For a more detailed breakdown of the different months, it is presented in Nadine's EDA. January Temperature GraphThe dataset and graph below show the temperature trend in the different regions in January from 1970 - 2020.<jupyter_code>df2['DATE'] = pd.to_datetime(df2['DATE'])
df_january_temp = df2[df2['DATE'].dt.month == 1]
sns.lineplot(data = df_january_temp,
           y = "TEMPERATURE",x = "YEAR", hue = "LOCATION",).set_title("Temperature in Canada on January (1970 - 2020)")
plt.legend(bbox_to_anchor = (1.1 , 1), borderaxespad = 0)<jupyter_output><empty_output><jupyter_text>June Temperature GraphThe dataset and graph below show the temperature trend in the different regions in June from 1970 - 2020.<jupyter_code>df2['DATE'] = pd.to_datetime(df2['DATE'])
df_january_temp = df2[df2['DATE'].dt.month == 6]
sns.lineplot(data = df_january_temp,
           y = "TEMPERATURE",x = "YEAR", hue = "LOCATION").set_title("Temperature in Canada on June (1970 - 2020)")
plt.legend(bbox_to_anchor = (1.1 , 1), borderaxespad = 0)<jupyter_output><empty_output><jupyter_text>December Temperature GraphThe dataset and graph below show the temperature trend in the different regions in December from 1970 - 2020.<jupyter_code>df2['DATE'] = pd.to_datetime(df2['DATE'])
df_january_temp = df2[df2['DATE'].dt.month == 12]
sns.lineplot(data = df_january_temp,
           y = "TEMPERATURE",x = "YEAR", hue = "LOCATION").set_title("Temperature in Canada on December (1970 - 2020)")
plt.legend(bbox_to_anchor = (1.1 , 1), borderaxespad = 0)<jupyter_output><empty_output><jupyter_text>EDA for Percipitation --- Filtering Precipitation Data by MonthDone by James *Changing the date value to simplify filter process*Here, we changed the date value to remove the time and used the melt() function to create a dataset to create the graph. Then we created a graph that presents the different percipitation of the different regions in each month from 1970 until 2020. The dataset below only shows the changed date format. A more detailed breakdown of precipitation levels in Canada throughout the years is available in James's EDA.<jupyter_code>#### Grouping Precipitation by Region
britishcolumbia_p = pd.melt(df, id_vars ='DATE', value_vars = 'PRECIPITATION_BRITISH_COLUMBIA')
northern_p = pd.melt(df, id_vars ='DATE', value_vars = 'PRECIPITATION_NORTHERN')
atlantic_p = pd.melt(df, id_vars ='DATE', value_vars = 'PRECIPITATION_ATLANTIC')
praries_p = pd.melt(df, id_vars ='DATE', value_vars = 'PRECIPITATION_PRAIRIES')
ontario_p = pd.melt(df, id_vars ='DATE', value_vars = 'PRECIPITATION_ONTARIO')
quebec_p = pd.melt(df, id_vars ='DATE', value_vars = 'PRECIPITATION_QUEBEC')
#British Columbia
britishcolumbia_p = britishcolumbia_p.assign(YEAR = lambda x : x.DATE)
britishcolumbia_p["YEAR"] = pd.to_datetime(britishcolumbia_p["YEAR"], utc = True).dt.year
bc = britishcolumbia_p.groupby(by = ['YEAR']).mean(['PRECIPITATION']).rename(columns={"value": "PRECIPITATION"})
bc['LOCATION'] = 'British Columbia'
#Northern Provinces
northern_p = northern_p.assign(YEAR = lambda x : x.DATE)
northern_p["YEAR"] = pd.to_datetime(northern_p["YEAR"], utc = True).dt.year
nor = northern_p.groupby(by = ['YEAR']).mean(['PRECIPITATION']).rename(columns={"value": "PRECIPITATION"})
nor['LOCATION'] = 'Northern Provinces'
#Atlantic Provinces 
atlantic_p = atlantic_p.assign(YEAR = lambda x : x.DATE)
atlantic_p["YEAR"] = pd.to_datetime(atlantic_p["YEAR"], utc = True).dt.year
atl = atlantic_p.groupby(by = ['YEAR']).mean(['PRECIPITATION']).rename(columns={"value": "PRECIPITATION"})
atl['LOCATION'] = 'Atlantic Provinces'
#Prairie Provinces 
praries_p = praries_p.assign(YEAR = lambda x : x.DATE)
praries_p["YEAR"] = pd.to_datetime(praries_p["YEAR"], utc = True).dt.year
pra = praries_p.groupby(by = ['YEAR']).mean(['PRECIPITATION']).rename(columns={"value": "PRECIPITATION"})
pra['LOCATION'] = 'Prairie Provinces'
#Ontario
ontario_p = ontario_p.assign(YEAR = lambda x : x.DATE)
ontario_p["YEAR"] = pd.to_datetime(ontario_p["YEAR"], utc = True).dt.year
ont = ontario_p.groupby(by = ['YEAR']).mean(['PRECIPITATION']).rename(columns={"value": "PRECIPITATION"})
ont['LOCATION'] = 'Ontario'
#Quebec
quebec_p = quebec_p.assign(YEAR = lambda x : x.DATE)
quebec_p["YEAR"] = pd.to_datetime(quebec_p["YEAR"], utc = True).dt.year
que = quebec_p.groupby(by = ['YEAR']).mean(['PRECIPITATION']).rename(columns={"value": "PRECIPITATION"})
que['LOCATION'] = 'Quebec'
combined = [bc, nor, atl, pra, ont, que]
canada_precipitation = pd.concat(combined)
canada_precipitation<jupyter_output><empty_output><jupyter_text>Analysis of Percipitation: ---**Research Question:** Are there changes occuring in percipitation patterns in our climate zones and which zone have the highest and lowest percipitation values. The summative graph shows the precipitation patterns in each climate zone throughout our selected time period.<jupyter_code>sns.lineplot(data = canada_precipitation, x="YEAR", y="PRECIPITATION", hue = "LOCATION")
plt.xticks(rotation='vertical')
plt.ylabel('Precipitation')
plt.xlabel('Year')
plt.legend(bbox_to_anchor = (1.1 , 1), borderaxespad = 0)
sns.despine()<jupyter_output><empty_output><jupyter_text>Looking into the data set:---<jupyter_code>df.describe()<jupyter_output><empty_output><jupyter_text>Pandas ProfilingWe wanted to use Pandas Profile to look for possible correlations in the data and see what if there were any issues. I placed the  symbol infront of dfprofile so it doesn't run automatically (because it takes a long time).<jupyter_code>dfprofile = pdp.ProfileReport(df)
#dfprofile<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11247233638950363, 'fraction_numerical': 0.014464116345241858, 'mean_word_length': 5.317024463305042, 'pattern_counts': {'":': 13, '<': 47, '<?xml version=': 0, '>': 45, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '329600', 'license': 'MIT', 'n_tokens_mistral': 4380, 'n_tokens_neox': 4137, 'n_words': 1174, 'path': 'milestone2.ipynb', 'repo_name': 'data301-2020-winter2/course-project-group_1012', 'size': '12652'}