<jupyter_start><jupyter_text>Data Gathering, Exploration and CleaningBefore building the model, I want to understand the nature of the data we have. There are several data sources we're using, all index by date.**Outcome Data**- SAR response. This data set indicates whether or not there was a SAR call on a specified day.**Features**- Date features- Weather from [NOAA](https://www.ncdc.noaa.gov/cdo-web/)  SAR Response Data First, let's dig into the SAR response data. I've deleted the mission names for privacy reasons. How often does a mission happen? - **DEM** is an id number provided by the state- **Responders** is the number of responders- **Hours** is the sum of the number of hours provided by all responders- **Miles** is the sum of the number of hours provided by all respondersThese other fields may me interesting for downstream analyses, but probably aren't important for this first classification problemn approach<jupyter_code>import pandas as pd
import numpy as np
import re
import matplotlib.pyplot as plt
import matplotlib.style as style
import seaborn as sns


sns.set()
%matplotlib inline
style.use('fivethirtyeight')
input_filepath = '../data/raw/'
sar_header = ['DEM', 'date', 'responders', 'hours', 'miles']
sar_data = pd.read_csv(f'{input_filepath}KCSARA.csv', header=None, 
                       parse_dates=['date'], names=sar_header)
sar_data = pd.DataFrame(sar_data)
sar_data.head()
sar_data.shape
sar_data.tail()<jupyter_output><empty_output><jupyter_text>There are NAs in the date column, we need to get rid of those<jupyter_code>sar_data = sar_data[pd.notnull(sar_data.date)]
def display_all(df):
    with pd.option_context("display.max_rows", 1000, "display.max_columns", 1000): 
        display(df)
display_all(sar_data.describe(include='all').T)<jupyter_output><empty_output><jupyter_text>Does the data make sense?1. How many calls occur per year?We'll take the add_datepart function from fast.ai, add some functionality and use it to add some date features<jupyter_code>#Use a function from fast.ai to extract interesting data info

def add_datepart(df, fldname, drop=True, time=False, errors="raise"):
    fld = df[fldname]
    fld_dtype = fld.dtype
    if isinstance(fld_dtype, pd.core.dtypes.dtypes.DatetimeTZDtype):
        fld_dtype = np.datetime64

    if not np.issubdtype(fld_dtype, np.datetime64):
        df[fldname] = fld = pd.to_datetime(fld, infer_datetime_format=True, errors=errors)
    targ_pre = re.sub('[Dd]ate$', '', fldname)
    attr = ['Year', 'Month', 'Week', 'Day', 'Dayofweek', 'Dayofyear',
            'Is_month_end', 'Is_month_start', 'Is_quarter_end', 'Is_quarter_start', 'Is_year_end', 'Is_year_start']
    if time: attr = attr + ['Hour', 'Minute', 'Second']
    for n in attr: df[targ_pre + n] = getattr(fld.dt, n.lower())
    df[targ_pre + 'Elapsed'] = fld.astype(np.int64) // 10 ** 9
    if drop: df.drop(fldname, axis=1, inplace=True)

add_datepart(sar_data, 'date', drop=False)
sar_data.head()
def missions_per_year(df):
    '''
    From a dataframe with the 'Year' column,
    Plot a graph showing the number of missions per year
    '''
    
    missions_per_year = df.Year.value_counts()
    plt.plot(missions_per_year.sort_index())
    plt.show()


missions_per_year(sar_data)<jupyter_output><empty_output><jupyter_text>Clearly some weirdness happening here, lets cut the data to remove anything from before 1980 and in the future.<jupyter_code>sar_data = sar_data[(sar_data.Year < 2020) & (sar_data.Year > 1979)]
missions_per_year(sar_data)<jupyter_output><empty_output><jupyter_text>It seems to me that the data prior to 2001 is incomplete in some instances and likely follows a different model than the years proir to 2001. Let's focus this analysis on the missions after the spike. For example, in 1991 there are only missions in months 1-4, even though we know the most popular months are the summer months. This is seen in several other years as well.<jupyter_code>sar_data = sar_data[(sar_data.Year < 2020) & (sar_data.Year > 2001)]
missions_per_year(sar_data)<jupyter_output><empty_output><jupyter_text>Missions per weekAs shown below, the summer weeks and weekends get the highest mission density, Saturday in particular.<jupyter_code>def missions_per_week(df):
    '''
    From a dataframe with the 'Week' column,
    Plot a graph showing the number of missions per week
    '''
    
    missions_per_week = df.Week.value_counts()
    plt.plot(missions_per_week.sort_index())
    plt.show()
    
missions_per_week(sar_data)
missions_per_weekday = sar_data.Dayofweek.value_counts()
plt.plot(missions_per_weekday.sort_index())
plt.show()<jupyter_output><empty_output><jupyter_text>Clean data to include all dates and a mission classBefore we can do any ML, we need a table with all dates that indicates whether a mission happened or not. The rest of the variables in this table aren't important for our predictions.<jupyter_code>date_range = pd.date_range(start='1/1/2002', end='4/01/2019')

clean_table = []

for d in date_range:
    if sar_data.date.isin([d]).any():
        clean_table.append([d,1])
    else:
        clean_table.append([d,0])
               
sar_clean = pd.DataFrame(clean_table)
sar_clean.columns = ['date','mission']
sar_clean.head()
# Add the date parts to this new tidy table
add_datepart(sar_clean, 'date', drop=False)
sar_clean.head()
sar_clean.mission.mean()<jupyter_output><empty_output><jupyter_text>How often is there a call? 29% of ther time.  Weather DataWe're pulling data from two weather stations in Washington State. Boeing field is a little south of Seattle and will represent the 'city' weather, and <NAME> is a weather station in the Cascade Mountain Range East of Seattle and will report on the 'mountain' weather in the model.I couldn't find freely accessible weather forecast data, which would be more relevent for predicting future missions. To try and deal with this, I'll only use weather features that we can expect a weather forecast to contain.I'm not sure what to expect, poor weather will mean less people in the mountains, but it could also mean more dangerous conditions for the people that are out.<jupyter_code>boeing = pd.read_csv(f'{input_filepath}boeing.csv', parse_dates=[2])
boeing.describe()
gardner = pd.read_csv(f'{input_filepath}gardner.csv', parse_dates=[2])
gardner.describe()
print(boeing.shape, gardner.shape)<jupyter_output>(6309, 21) (6305, 11)<jupyter_text>We want tidy data, so let's join on the date.<jupyter_code>weather = pd.merge(boeing, gardner, how='inner', on='DATE')
print(weather.shape)
weather.head()<jupyter_output>(6297, 31)<jupyter_text>The NANs for the weather codes essentially mean that that weather type didn't happen, so I'll replace them with a 0<jupyter_code>weather = weather.fillna(0)
weather.head()<jupyter_output><empty_output><jupyter_text>And finally merge the weather with the mission data to get our final dataset to use.<jupyter_code>data = pd.merge(sar_clean, weather, how='inner', left_on='date', right_on='DATE')
print(data.shape)

pd.set_option('display.max_columns', None) #show all the columns
data.head()<jupyter_output>(6281, 46)<jupyter_text>Lastly, I'll remove the unnessecary column names including station names and weather measurements that wouldn't be included in a standard forecast- more specifically, the wind gust information.<jupyter_code>data.drop(['date', 'STATION_x', 'NAME_x', 'STATION_y','NAME_y', 'WDF2', 'WDF5', 'WSF2', 'WSF5', 'PGTM', 'FMTM'], axis=1, inplace=True)
# Save the data table for now.
interim_filepath = '../data/interim/'
data.to_csv(f'{interim_filepath}sar_weather_df.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10520806071708977, 'fraction_numerical': 0.013478147081915729, 'mean_word_length': 5.280197206244864, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '292052', 'license': 'MIT', 'n_tokens_mistral': 2587, 'n_tokens_neox': 2410, 'n_words': 794, 'path': 'notebooks/.ipynb_checkpoints/01-DataExploreAndClean-checkpoint.ipynb', 'repo_name': 'afederation/RescueForcast', 'size': '7649'}