<jupyter_start><jupyter_text>Final Project: Predictive Modeling of California Wildfires Using Weather Data **Members:** *<NAME> Bocalbos <NAME>*  Introduction: Although there is not really a specific trend about US wildfires in recent years, that is, whether it is increasing or decreasing in number, this incident has been costing the US government money not just because of the losses but also the cost of mitigating fires. In fact, for the past 10 years alone, losses from wildfires added up to \\$5.1 billion. In California alone, the total potential exposure to wildfire damage for the high and very high risk categories amounts to a total of $105.45 billion dollars which is about 44% of the total amount for 13 states such as Washington, Texas, Oklahoma, Colorado, Montana, Nevada, to mention a few. In 2017, California also ranked as the top state to be most wildfire prone based on number of households at risk but ranked fourth in terms of percentage. Last year, California ranked second as having the highest number of fires. California, however, ranked first in terms of the number of acres burned.Wildfires are caused by different factors. According to research, 90% are caused by humans such as unattended campfires, burning of debris, downed power lines, irresponsibly discarding cigarettes, and even intentional acts of arson. The remaining 10% is caused by some environmental factors like lightning or lava. Because of the information discussed above, this study will focus on the wildfires that happened in California and build a model that will predict the size of the area burned during wildfires based on historical weather information.  Research Questions and Significance of the Study: Using historical forest fires data in California and weather data during the start of the fire, the main question that we would like to answer for this research is to create a model which can predict the area burned during forest fires. Prior to this, we will be conducting some exploratory data analysis to analyze trends (e.g., number of fires that occurred in a specific season, year, and month and also the area burned in a specific season, year and month).As mentioned above, losses brought about by forest fires are costly. Thus, mitigating the impact of forest fires will be helpful to minimize not just the cost of the losses that can be incurred but also to minimize injuries or sometimes, even saving lives of people that can be involved in this type of incident. We believe that by being able to create a model to predict the area burned using the parameters involved, we will be able to optimize (that is, minimize) the days involved in stopping these forest fires. Since firemen will know that impact will be severe, the firemen will know ahead of time the number of people they need to deploy ahead of time and they can plan ahead as to the equipments/vehicles they need to stop the fire as fast as possible. By doing these, the damage brought about by forest fires will be minimized.  Data Source and Acquisition:  Data Sources: The research work and data that will be gathered in this study is based on this paper http://www3.dsi.uminho.pt/pcortez/fires.pdf and this dataset http://archive.ics.uci.edu/ml/datasets/Forest+Fires. This dataset lists forest fires in Portugal. However, for this research, we will be collecting our own data so we can be able to apply and build a model that would estimate the size of the fire using a dataset in California.For this project, we will be gathering data using the following websites:1. National Interagency Coordination Center (NICC) [link] https://www.nifc.gov/fireInfo/fireInfo_statistics.html: This website was used to gather historical wildfire annual summary report in the US. The files are in PDF version and the table that we are going to use only lists fires that are considered "significant". The definition of which is that the total burned area is equal or over 40,000 acres in size. 2. Dark Sky API [link] (https://darksky.net/dev/account): This website was used to gather information about the weather based on the location and date that the fire happened. Before starting to gather data, a research was conducted since the original dataset (Portugal Forest Fires) used the Fire Weather Index (FWI) System in their study. This is one of the limitations of our current study. We were not able to calculate the different FWI such as Fine Fuel Moisture Code (FFMC), Duff Moisture Code (DMC), Drought Code (DC), and Initial Spread Index (ISI). However, weather data gathered used the information to calculate these indeces such as Air Temperature, Relative Humidity, Wind Speed, and Rainfall. Furthermore, research said that the weather readings should be taken at noon standard time. Therefore, this time information was used during the API call.  Data Acquisition:  Part 1: Fire Dataset Upon checking the PDF file annual reports of the fires in the US, our group decided to just use the files from 2005 - 2018 because the reports prior to year 2005, i.e., 2000 - 2004, does not have the table that we have identified as containing the useful information we need for this project. The target table that we used contains the following columns:Name: The unique name used to identify the fireGACC: Geographic Area Coordination Center where the incident was reported and handledState: Name of the state where the fire occurredStart Date: The date when the fire startedContain or Last Report Date: The date when the fire is considered containedSize: (in acres) The size of the burned areaCause: The reason for the start of the fireEstimated Cost: Cost of mitigating the fire**Steps for Data Management and Collection:**1. Download all annual report PDF files from the above website.2. Save the PDF files in a folder in one of our local machines.3. Look for the pages where the target table can be found.4. Run code found below to read the PDF files and extract target table and save table as CSV files that will be merged as one CSV file and will be used for data analysis.**Prerequisites for running the code:**Since the researchers have no experience yet in reading PDF files in Python, we had to do some research on how we can extract the PDF file in Python. After some research, we found a fairly new library called Camelot that is able to read and extract PDF files into a CSV file [link] https://camelot-py.readthedocs.io/en/master/user/quickstart.htmlread-the-pdf. To do this, there are some things that is needed to be installed. These are the camelot library, cv2 library, and install Ghostscript. **Step 1: Read PDF file and extract to CSV file**<jupyter_code>import os
import camelot
import cv2

path = r'C:\Maridor\MS DAV\DAV 5400 - Analytics Programming\Final Project\fire'
for myfiles in os.listdir(path): #loops through the folder above to check for the PDF files
    if myfiles.endswith('.pdf'): #searches for only PDF file
        fname, fext = os.path.splitext(myfiles) #splits the filename and extension
        tables = camelot.read_pdf(path + '\\' + myfiles, pages='1,2,3,10,11,12') #looks for the pages where the target table is and reads it
        tables.export(r'C:\Maridor\MS DAV\DAV 5400 - Analytics Programming\Final Project\fire\text files' + '\\' + fname + '.csv', f='csv') #extracts and exports the file as a CSV file<jupyter_output><empty_output><jupyter_text>**Step 2: Combines all extracted CSV file in one CSV file**<jupyter_code>path1 = r'C:\Maridor\MS DAV\DAV 5400 - Analytics Programming\Final Project\fire\text files'+ '\\'
i = 0
file = ''
combinedfile = pd.DataFrame() #creates an empty dataframe to store the combined csv files
for f in os.listdir(path1): #loops through the directory to find the CSV file
    file = file + str(i)
    file = pd.read_csv(path1 + f) #reads each csv file
    combinedfile = pd.concat([file,combinedfile]) #merges the previous read file into one file
combinedfile.to_csv(r'C:\Maridor\MS DAV\DAV 5400 - Analytics Programming\Final Project\fire\text files\combinedfire.csv') #saves the file as CSV into the local machine<jupyter_output><empty_output><jupyter_text>Part 2: Weather Dataset To be able to make requests through the Dark Sky API website, you need to first sign up and get your unique key. The url requests also requires the following information longitude and latitude (location where the fire happened) and time you are inquiring about. Since our historical fire does not have this information, we needed to research on the location of each California wildfire which is found to be 61 records from 2005 - 2018. After researching on the location information, a CSV file was created and this is used for each API requests. Found below is the code that was used to do this: **Step 1: Read CSV file**<jupyter_code>import requests
import json

#read csv file that contains longitude, latitude, and date/time information to be used for the API call
fire_loc = pd.read_csv('https://raw.githubusercontent.com/doryaswi/DAV-5400/master/combinedfire_location.csv') 
fire_loc.rename(columns={'Start Date': 'StartDate'},inplace=True) 
fire_loc

#cleans Name column to remove additional whitespaces
fire_loc['Name']=fire_loc['Name'].str.replace('\r\r','') 
fire_loc['Name']=fire_loc['Name'].str.replace('\n',' ')
fire_df<jupyter_output><empty_output><jupyter_text>**Step 2: Loops through each record in the CSV file and makes an API call to get weather data**<jupyter_code>combinedweather = pd.DataFrame() #creates empty dataframe to store all weather data
for f, row in enumerate(fire_loc.values): #loops through records in above csv file
    url = 'https://api.darksky.net/forecast/0620e4ec181169aa22e751c09161e627/' + str(fire_loc.Latitude[f]) + ',' + str(fire_loc.Longitude[f]) + ',' + fire_loc.Date[f] #builds url needed to request a call through the API
    call = requests.get(url)
    weather = call.json() #transforms weather information as json data
    wdata = weather['currently'] #extracts needed json data
    wdata.update({'name':fire_loc.Name[f]})
    w = pd.DataFrame(wdata,index=[0]) #transforms json data as a dataframe
    combinedweather = pd.concat([w,combinedweather]) #merges each row of the dataframe created into one dataframe
combinedweather.to_csv(r'C:\Maridor\MS DAV\DAV 5400 - Analytics Programming\Final Project\fire\text files\combinedweather.csv') #saves the dataframe as a CSV file in the local machine<jupyter_output><empty_output><jupyter_text>Data Preparation for Analysis: There are two files collected that will be used for data analysis and modeling. One file contains all the significant fires that happened in the US between 2005 to 2018 and the other one contains weather data information for all the significant fires that happened in California from 2005 to 2018.Exploratory Data Analysis (EDA) will be done in two parts. The first EDA will analyze the overall trend of wildfires in the US while the second EDA will only focus on the wildfires that happened in California.  Part 1: Cleaning up the dataset for all US wildfires **Steps done to prepare first dataset for analysis:**1. Import and read CSV file containing all records with US wildfires.2. Drop additional index column created when CSV file was exported.3. Clean "Name" column to remove whitespaces and special characters.4. Remove commas (",") from numeric columns (such as Costs and Size) and change datatype to numeric.5. Transform Start Date and Contain Date columns from string to Date Time to be able to do some calculations.6. Calculate days between date fire started and contain date.7. Extract month and year of when the fire started. Replace month columns to the names of the actual months.8. Check if there are contain dates that are greater than date fire actually started do necessary steps to correct this.<jupyter_code>import numpy as np
import pandas as pd

fire_df = pd.read_csv('https://raw.githubusercontent.com/doryaswi/DAV-5400/master/combinedfire.csv')
fire_df.drop(columns = 'Unnamed: 0',inplace=True) #drops index column created during extract
fire_df.rename(columns={'Contain or Control Date': 'ControlDate', 
                        'Start Date': 'StartDate'},inplace=True) #renames columns to become shorter
fire_df.head()
fire_df['Name']=fire_df['Name'].str.replace('\r\r','') #looks for '\r\r' between Name values and replaces string as space
fire_df['Name']=fire_df['Name'].str.replace('\n',' ') #looks for '\n' on Name values and replaces string as space
fire_df.head()
fire_df['Size']=fire_df['Size'].str.replace(',','').astype(float) #removes commas from Size column and replaces it to space
fire_df.head()                                                    #then change datatype from string to float
fire_df['Costs']=fire_df['Costs'].str.replace(',','') #removes commas from Costs column and replaces it to space
fire_df['Costs']=fire_df['Costs'].str.replace('$','') #removes $ from costs column and replaces it to space
fire_df['Costs'].replace({'NR':np.nan},inplace=True) #replaces NR values as null values
fire_df['Costs'] = pd.to_numeric(fire_df['Costs']) #changes datatype from string to numeric
fire_df.head()
from datetime import date #imports datetime library and date module to transform Start Date and Contain Date from string to 
                          #date
fire_list = []
for d, row in enumerate(fire_df.values): #loops through the dataframe
    getMonth, getDay, getYear = fire_df.StartDate[d].split('/') #splits StartDate column to get month, day, and year
    startmonth = getMonth #extracts and stores month information
    startyear = getYear #extracts and stores year information
    startdate = date(int(getYear), int(getMonth), int(getDay)) #extracts and stores the date information for future calculation
    getMonth, getDay, getYear = fire_df.ControlDate[d].split('/') #splits ContainDate column to get month, day, and year
    controldate = date(int(getYear), int(getMonth), int(getDay)) #extracts and stores the date information for future calculation
    f_list = [fire_df.Name[d], startmonth, startyear, startdate, controldate] #gets Name value for each row, month, year, startdate, and controldate
    fire_list.append(f_list)
fire_list

fire_days = pd.DataFrame(fire_list, columns = ['Name1', 'StartMonth', 'StartYear','Start_Date', 'ContainDate']) #creates a new dataframe to store list created above
fire_days

fire_days['Days'] = fire_days['ContainDate'] - fire_days['Start_Date'] #calculates days between start date of fire and contain date
fire_days
fire_all = pd.concat([fire_df, fire_days],axis=1,sort=False) #concatenates original dataframe and dataframe with transformed date
fire_all.head()
fire_all.drop(columns=['StartDate','ControlDate','Name1'],inplace=True) #drops extra and unnecessary columns from new dataframe
fire_all.head()
fire_all.StartMonth.replace({'1':'January',
                             '2':'February',
                             '3':'March',
                             '4':'April',
                             '5':'May',
                             '6':'June',
                             '7':'July',
                             '8':'August',
                             '9':'September',
                             '10':'October',
                             '11':'November',
                             '12':'December'}, inplace = True) #give meaningful names to the months
fire_all.head()
#creates Season column determined by month of the year
# December to February = Winter
# March to May = Spring
# June to August = Summer
# September to November = Fall

fire_all['Season'] = fire_all['StartMonth'] #creates a copy of StartMonth column and add a new column called Season
fire_all.Season.replace({'January':'Winter',
                         'February':'Winter',
                         'March':'Spring',
                         'April':'Spring',
                         'May':'Spring',
                         'June':'Summer',
                         'July':'Summer',
                         'August':'Summer',
                         'September':'Fall',
                         'October':'Fall',
                         'November':'Fall',
                         'December':'Winter'}, inplace = True) #replaces value of months to the season it corresponds with
fire_all.head()
#checks whether there is a containdate greater than start date
fire_all['is_correct'] = fire_all.apply(lambda x: x['ContainDate'] > x['Start_Date'], axis=1)
fire_false = fire_all[(fire_all['is_correct']==False)]
fire_false<jupyter_output><empty_output><jupyter_text>**Note:** There were two rows found to have contain date greater than start date. Since there were only two records affected, the researchers decided to just filter out these rows.<jupyter_code>fire_true = fire_all[(fire_all['is_correct']==True)] #only stores data with valid contain date and start date values
fire_true.shape
fire_true.drop(columns='is_correct', inplace=True) #drops "is_correct" column (not needed anymore)
fire_true.head()<jupyter_output>C:\Users\Admin\Anaconda3\lib\site-packages\pandas\core\frame.py:3697: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  errors=errors)<jupyter_text>Part 2: Cleaning up the dataset for CA wildfires only **Steps done to prepare second dataset for analysis:**1. From the list of all US wildfires dataframe, get only those that happened in California.2. Read CSV file that contains California wildfires and its corresponding weather data.3. Merge data from Step 1 and Step 2.4. Check if there are contain dates that are greater than date fire actually started do necessary steps to correct this.<jupyter_code>CA_fires = fire_true[(fire_true['State']=='CA')] #filters California fires only from all US wildfires data frame
CA_fires
CA_weather = pd.read_csv('https://raw.githubusercontent.com/doryaswi/DAV-5400/master/combinedweather.csv')
#drops unnecessary columns
CA_weather.drop(columns=['Unnamed: 0','apparentTemperature','cloudCover','dewPoint','icon','precipProbability',
                         'precipType','pressure','uvIndex','visibility','windBearing','windGust','time'],inplace=True)
#rearranges column names
CA_weather = CA_weather[['name','summary','temperature','humidity','precipIntensity','windSpeed']]
#replaces column name for standardization
CA_weather.rename(columns={'name':'Name'},inplace=True)
CA_weather
CA_all = pd.merge(CA_fires,CA_weather,on=['Name']) #merges two dataframes in one to have all CA fires and weather info in one
CA_all
CA_all['is_correct'] = CA_all.apply(lambda x: x['ContainDate'] > x['Start_Date'], axis=1) #checks if there are invalid Contain Date values
CA_false = CA_all[(CA_all['is_correct']==False)]
CA_false<jupyter_output><empty_output><jupyter_text>**Note:** Based on this, there are no invalid value for Contain Date therefore, there is no action needed.<jupyter_code>CA_all.drop(columns='is_correct',inplace=True) #drops is_correct column
CA_all<jupyter_output><empty_output><jupyter_text>Exploratory Data Analysis: Exploratory Data Analysis will be conducted in two parts. First part is the entire US Wildfire dataset while the second part will only focus on the wildfires that happened in California. Part 1: US Wildfire Dataset **A. Trend Based on Number of Fires**1. Get descriptive statistics of numerical columns.2. Create charts of trend in number of fires per year, month, and season.<jupyter_code>fire_true.describe()<jupyter_output><empty_output><jupyter_text>**Findings:** *Size:* In terms of the size of the burned area, the mean is about 107,113 acres. Since the median at 68,500 acres is less than the mean, this shows that the distribution for the size of the burned area is skewed to the right. Since significant fires are considered to be those equal to have area burned size of 40,000 or greater, it is just expected that the minimum value is equal or slightly above this value, which, in this case is equal to 40,000 acres. The maximum value is at 907,245 acres.*Costs:* With respect to the costs incurred for the fire, the mean is about \\$19M. The distribution of the cost is also skewed to the right due to the fact that the median (\\$5M) is less than the mean.*Days:* The length of days between start of the fire and when it was contained has a mean value of 43 days. Again, this variables distribution is also skewed to the right since median at 28 days is less than the mean. In addition, the maximum length of days that a fire was mitigated is 242 days which is about 8 months.<jupyter_code>import matplotlib.pyplot as plt 
import seaborn as sns

fire_year=fire_true.groupby(['StartYear']).size().to_frame('Number of Fires') #gets count of number of fires by year
fire_year
plt.figure(figsize=(8,4))
sns.pointplot(x=fire_year.index,y='Number of Fires',data=fire_year,color='orange'); #plots line chart of fires per year<jupyter_output><empty_output><jupyter_text>**Findings:** This confirms the statement made on the introduction above that there are no specific trends in terms of the number of fires that happen year after year. The year that has the most fire occurrences is 2007 with 53 occurrences and 2015 comes close to second having 52 occurrences.<jupyter_code>fire_season=fire_true.groupby(['Season']).size().to_frame('Number of Fires') #gets count of number of fires by season
fire_season.sort_values(by='Number of Fires',ascending=False,inplace=True)
fire_season
sns.barplot(x=fire_season.index,y='Number of Fires',data=fire_season,color='orange'); #plots bar chart of fires per season<jupyter_output><empty_output><jupyter_text>**Findings:** This bar chart shows that most of the fires occur during the summer season and goes down during the winter which is kind of what we are expecting. This shows that as it gets colder, the number of fires become less.<jupyter_code>fire_month=fire_true.groupby(['StartMonth']).size().to_frame('Number of Fires') #gets count of number of fires by month
fire_month
fire_month.index = pd.CategoricalIndex(fire_month.index, 
                               categories=['January', 'February', 'March', 'April','May','June', 'July', 'August','September', 'October', 'November', 'December'], 
                               ordered=True) #assigns months as categorical indices to be able to use it to sort the bar chart by month
fire_month = fire_month.sort_index()
fire_month
plt.figure(figsize=(15,4))
sns.barplot(x=fire_month.index,y='Number of Fires',data=fire_month,color='orange'); #plots bar chart of fires per month<jupyter_output><empty_output><jupyter_text>**Findings:** This shows that the highest occurrence of fires happen during the month of July. June comes in second and August is the third. This is somehow expected since summer starts towards the end of June so the hottest days will most likely fall during the month of July. **B. Trend Based on Average Size of Fires**<jupyter_code>fire_sy=fire_true.groupby(['StartYear']).mean()['Size'].to_frame('Average Size') #gets average size of burned area by year
fire_sy
plt.figure(figsize=(8,4))
sns.pointplot(x=fire_sy.index,y='Average Size',data=fire_sy,color='orange'); #plots line chart of average size of burned area per year<jupyter_output><empty_output><jupyter_text>**Findings:** Although the highest number of occurrences of fire happened in 2007 and 2015, 2009 has the highest average size of area burned (151,740 acres) while 2014 comes close to second (145,602 acres).<jupyter_code>fire_ss=fire_true.groupby(['Season']).mean()['Size'].to_frame('Average Size') #gets average size of area burned by season
fire_ss.sort_values(by='Average Size',ascending=False,inplace=True)
fire_ss
sns.barplot(x=fire_ss.index,y='Average Size',data=fire_ss,color='orange'); #plots bar chart of average size of area burned by season<jupyter_output><empty_output><jupyter_text>**Findings:** Again, even though summer has the highest occurrence of fire, spring has the highest average size of area burned at 139,333 acres while summer is only second at 103,874 acres. Suprisingly also, winter comes in third at an average size of 99,440 acres burned.<jupyter_code>fire_sm=fire_true.groupby(['StartMonth']).mean()['Size'].to_frame('Average Size') #gets average size of area burned by month
fire_sm
fire_sm.index = pd.CategoricalIndex(fire_sm.index, 
                               categories=['January', 'February', 'March', 'April','May','June', 'July', 'August','September', 'October', 'November', 'December'], 
                               ordered=True) #assigns months as categorical indices to be able to use it to sort the bar chart by month
fire_sm = fire_sm.sort_index()
fire_sm
plt.figure(figsize=(15,4))
sns.barplot(x=fire_sm.index,y='Average Size',data=fire_sm,color='orange'); #plots bar chart of fires per year<jupyter_output><empty_output><jupyter_text>**Findings:** Since spring has the highest average size in terms of area burned, it seems logical that a month under this season, in this case March, will have the highest average size of area burned. December comes in second as having the highest average size of are burned per month.  Part 2: California Wildfire with Weather Dataset **A. Trend Based on Number of Fires**1. Get descriptive statistics of numerical columns.2. Create charts of trend in number of fires per year, month, and season.<jupyter_code>CA_all.describe()<jupyter_output><empty_output><jupyter_text>**Findings:** *Size:* In terms of the size of the burned area, the mean is about 105,508 acres which is quite close to the overall mean value of 107,113 acres. The median is also less than the mean at 78,698 therefore distribution is also skewed to the right. Since significant fires are considered to be those equal to have area burned size of 40,000 or greater, it is just expected that the minimum value is equal or slightly above this value, which, in this case is equal to 40,000 acres. In this case, it is equal to 40,200 acres. The maximum value is at 459,123 acres.*Costs:* With respect to the costs incurred for the fire, the mean is about \\$68M. This is a lot more than the overall mean which was at \\$19M. The distribution of the cost is also skewed to the right due to the fact that the median (\\$56M) is less than the mean.*Days:* The length of days between start of the fire and when it was contained has a mean value of 45 days which is also higher than the overall mean at 43 days. Again, this variable's distribution is also skewed to the right since median at 35 days is less than the mean. In addition, the maximum length of days that a fire was mitigated is 164 days which is almost 6 months.*Temperature:* The mean temperature on the day that the fire occurred is at 80 degrees fahrenheit. The median is very close to the mean at 81.6. This can mean that the temperature is almost normally distributed. It is also interesting to note that the maximum temperature that a fire occurred is at 96 degrees fahrenheit.*Humidity:* The mean humidity on the day the fire occurred is about 27\% while median is at 24\%. The distribution is slightly skewed to the right.*Precipitation:* Looking at the observations, there was only one with value more than zero. For this reason, the researchers will not use this column for now for further analysis since it may not provide additional information.*Wind Speed:* With regards to wind speed, the mean wind speed the day the fire occurred is at 3.09mph while median is at 2.43mph. Again, this gives us information that the wind speed data is skewed to the right.<jupyter_code>CA_year=CA_all.groupby(['StartYear']).size().to_frame('Number of Fires') #gets count of number of fires by year
CA_year
plt.figure(figsize=(8,4))
sns.barplot(x=CA_year.index,y='Number of Fires',data=CA_year,color='gold'); #plots bar chart of fires per year<jupyter_output><empty_output><jupyter_text>**Findings:** Although there is really no overall trend with respect to the number of fire occurrences per year in California, it is still noticeable that since 2008, the number of fire occurrences somehow decreased. The highest occurrence happened in 2008 at 12 while 2017 comes close at 2017.<jupyter_code>CA_season= CA_all.groupby(['Season']).size().to_frame('Number of Fires') #gets count of number of fires by season
CA_season.sort_values(by='Number of Fires',ascending=False,inplace=True)
CA_season
sns.barplot(x=CA_season.index,y='Number of Fires',data=CA_season,color='gold'); #plots bar chart of fires per season<jupyter_output><empty_output><jupyter_text>**Findings:** Like the overall distribution of fire occurrences per season, it is not surprising that Summer season has the highest occurrence of fire. It is quite surprising however that there were no fire occurrence that happened during the Spring season and that there were still a moderate amount of fire that occurred during the Fall season in California.<jupyter_code>CA_month=CA_all.groupby(['StartMonth']).size().to_frame('Number of Fires') #gets count of number of fires by month
CA_month
CA_month.index = pd.CategoricalIndex(CA_month.index, 
                               categories=['January', 'February', 'March', 'April','May','June', 'July', 'August','September', 'October', 'November', 'December'], 
                               ordered=True) #assigns months as categorical indices to be able to use it to sort the bar chart by month
CA_month = CA_month.sort_index()
CA_month
plt.figure(figsize=(15,4))
sns.barplot(x=CA_month.index,y='Number of Fires',data=CA_month,color='gold');<jupyter_output><empty_output><jupyter_text>**Findings:** Same as the overall US wildfires, the highest occurrence of fires for California happened during the month of July. June comes in second and August is the third. This is somehow expected since summer starts towards the end of June so the hottest days will most likely fall during the month of July. **B. Trend Based on Average Size of Fires**<jupyter_code>CA_sy=CA_all.groupby(['StartYear']).mean()['Size'].to_frame('Average Size') #gets average size of area burned by year
CA_sy
plt.figure(figsize=(8,4))
sns.barplot(x=CA_sy.index,y='Average Size',data=CA_sy,color='gold'); #plots bar chart of average size of area burned per year<jupyter_output><empty_output><jupyter_text>**Findings:** Even though 2008 has the highest number of occurrences of fire in California, the average size of burned area for that year was not that high at 70,000+ acres. However, 2013 has the highest average size of burned area across all years at 257,314 acres.<jupyter_code>CA_ss= CA_all.groupby(['Season']).mean()['Size'].to_frame('Average Size') #gets average size of area burned by season
CA_ss.sort_values(by='Average Size',ascending=False,inplace=True)
CA_ss
sns.barplot(x=CA_ss.index,y='Average Size',data=CA_ss,color='gold'); #plots bar chart of average size of area burned per season<jupyter_output><empty_output><jupyter_text>**Findings:** The one occurrence of fire during the winter season in California burned 270,000 acres as compared with summer season that had an average of 107,579 acres.<jupyter_code>CA_sm=CA_all.groupby(['StartMonth']).mean()['Size'].to_frame('Average Size') #gets average size of area burned by month
CA_sm
CA_sm.index = pd.CategoricalIndex(CA_sm.index, 
                               categories=['January', 'February', 'March', 'April','May','June', 'July', 'August','September', 'October', 'November', 'December'], 
                               ordered=True) #assigns months as categorical indices to be able to use it to sort the bar chart by month
CA_sm = CA_sm.sort_index()
CA_sm
plt.figure(figsize=(15,4))
sns.barplot(x=CA_sm.index,y='Average Size',data=CA_sm,color='gold'); #plots bar chart of average size of area burned by month<jupyter_output><empty_output><jupyter_text>**Findings:** The one occurrence of fire during the winter season in California burned 270,000 acres happened in December. For this chart, it shows that July (which is the month that has the highest occurrence of fire) has the second highest average size of burned area which is close to what is expected. **C. Scatter Plots and Correlation Coefficient**<jupyter_code>CA_all.corr()
plt.figure(figsize=(20,8))

#plots temperature vs. area burned using seaborn
plt.subplot(131)
sns.regplot(x = 'temperature',y = 'Size', data = CA_all);
plt.title('Scatterplot temperature vs. area burned')
plt.xlabel('temperature')
plt.ylabel('area burned')

#plots humidity vs. area burned using seaborn
plt.subplot(132)
sns.regplot(x = 'humidity',y = 'Size', data = CA_all);
plt.title('Scatterplot humidity vs. area burned')
plt.xlabel('humidity')
plt.ylabel('area burned')

#plots wind speed vs. area burned using seaborn
plt.subplot(133)
sns.regplot(x='windSpeed', y='Size', data=CA_all)
plt.title('Scatterplot wind speed vs. area burned');
plt.xlabel('wind speed')
plt.ylabel('area burned')<jupyter_output><empty_output><jupyter_text>**Findings:** Since we are going to predict the size of the area burned, we will be looking at the correlation coefficient for each independent variables such as temperature, humidity, and wind speed. Costs is excluded since cost is actually a function of the size of the area burned. Again, we are not going to use precipitation since only one record showed a value other than 0.*Size vs. Temperature:* The correlation coefficient of Size vs. Temperature is at 0.003610. This means that temperature and size has no linear relationship which is confirmed by the scatter plot above.*Size vs. Humidity:* The correlation coefficient of Size vs. Temperature is at -0.055177. This means that humidity and size has a very weak negative linear relationship to no linear relationship at all which is confirmed by the scatter plot above.*Size vs. Wind Speed:* The correlation coefficient of Size vs. Temperature is at 0.032312. This means that wind speed and size has a very weak positive linear relationship to no linear relationship at all which is confirmed by the scatter plot above.  Linear Regression Modeling:<jupyter_code>CA_pm = CA_all[['Size','Cause','Days','Season','temperature','humidity','windSpeed']]
CA_pm.head()
CA_dummies = pd.get_dummies(CA_pm)
CA_dummies.head()
import statsmodels.api as sm

X=CA_dummies[['temperature','windSpeed','humidity','Cause_H','Cause_L','Cause_U','Cause_NR','Season_Winter','Season_Summer','Season_Fall']]
Y=CA_dummies['Size']

X=sm.add_constant(X)

model_1 = sm.OLS(Y,X).fit()
predictions = model_1.predict(X)

print_model = model_1.summary()
print(print_model)<jupyter_output>OLS Regression Results                            
==============================================================================
Dep. Variable:                   Size   R-squared:                       0.113
Model:                            OLS   Adj. R-squared:                 -0.024
Method:                 Least Squares   F-statistic:                    0.8253
Date:                Sun, 05 May 2019   Prob (F-statistic):              0.584
Time:                        20:05:27   Log-Likelihood:                -769.02
No. Observations:                  61   AIC:                             1556.
Df Residuals:                      52   BIC:                             1575.
Df Model:                           8                                         
Covariance Type:            nonrobust                                         
=================================================================================
                    coef    std err          t   [...]
{'fraction_non_alphanumeric': 0.0882170804481225, 'fraction_numerical': 0.014946799527106907, 'mean_word_length': 4.418179045295105, 'pattern_counts': {'":': 0, '<': 97, '<?xml version=': 0, '>': 99, 'https://': 7, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '579428', 'license': 'MIT', 'n_tokens_mistral': 10140, 'n_tokens_neox': 9313, 'n_words': 4290, 'path': 'K_Song_M_Bocalbos_R_Li_FinalProject.ipynb', 'repo_name': 'ak47m1a1/DBDA-python', 'size': '35546'}