<jupyter_start><jupyter_text>Getting stared with FastAI v1: Tabular data This notebook illustrates how to get started with FastAI v1 for Tabular Data. The notebook will focus on introducing the API to newcomers, with an overview, including installation instructions for FastAI v1, given at: https://www.avanwyk.com/getting-started-with-fastai-v1-tabular-data/.The details of the deep learning techniques used (such as embedding, learning rate cycles etc.) are not covered in this notebook. For a detailed description of the techniques used, have a look at this blog post: [http://www.fast.ai/2018/04/29/categorical-embeddings/](http://www.fast.ai/2018/04/29/categorical-embeddings/).FastAI provides a convenient way of importing a lot of commonly used modules, including Numpy and Pandas:<jupyter_code>from fastai import *
from fastai.tabular import *

from sklearn.model_selection import train_test_split

np.set_printoptions(precision=4, suppress=True)
pd.set_option('display.float_format', lambda x: '%.3f' % x)<jupyter_output><empty_output><jupyter_text>Data Loading Before getting to the FastAI model, we first have to prepare our data. The dataset can be download here: https://www.kaggle.com/cdc/mortality. Download the data and place it into a directory `data`. We will be attempting to predict the age of mortaility using basic demographic data. The Kaggle data is separated into CSV files by year. We will limit the data to two year's worth, simply for performance reasons. We load the yearly data in turn and then create a single dataframe containing all the data.<jupyter_code>data_path = Path('data')
frames = []

for year in range(2005, 2007):
    frames.append(pd.read_csv(data_path/f'{year}_data.csv'))
    
df = pd.concat(frames)<jupyter_output>/home/avanwyk/anaconda/envs/fastai/lib/python3.6/site-packages/IPython/core/interactiveshell.py:2785: DtypeWarning: Columns (39,40,41,42,43,44,45,46,47,59,60,61,62,63,64,65,66) have mixed types. Specify dtype option on import or set low_memory=False.
  interactivity=interactivity, compiler=compiler, result=result)
/home/avanwyk/anaconda/envs/fastai/lib/python3.6/site-packages/IPython/core/interactiveshell.py:2785: DtypeWarning: Columns (39,40,41,42,43,44,45,46,47,59,60,61,62,63,64,65,66,67) have mixed types. Specify dtype option on import or set low_memory=False.
  interactivity=interactivity, compiler=compiler, result=result)<jupyter_text>The data contains 77 different columns (info into the meaning of the fields can be found [here](https://www.cdc.gov/nchs/nvss/mortality_public_use_data.htm)), however, we are only interested in the demographic data for this experiment, so we discard the other columns.<jupyter_code>demographic_colums = ['resident_status', 'education_1989_revision', 'education_2003_revision', 'education_reporting_flag', 'sex',
                      'detail_age_type', 'detail_age', 'marital_status', 'race']
df = df.filter(demographic_colums)
df.sample(5)<jupyter_output><empty_output><jupyter_text>Data Pre-processing The educational level is reported based on one of two systems: the 1989 revision and the 2003 revision. Here we normalize the education level to a single standard. We translate both revision's data to new categories which roughly groups similar categories from the other revision systems. category | description (highest level of education)--- | ---P | some primary school or no educationH | some high school educationT | some tertiary educationU | unknown<jupyter_code>def normalize_education(frame):
    frame['education'] = 'U'
    
    filter_1989 = frame['education_reporting_flag'] == 0
    
    frame.loc[filter_1989 & (frame.education_1989_revision <= 8.0), 'education'] = 'P'
    frame.loc[filter_1989 & (frame.education_1989_revision > 8.0) & (frame.education_1989_revision <= 12.0), 'education'] = 'H'
    frame.loc[filter_1989 & (frame.education_1989_revision > 12.0) & (frame.education_1989_revision < 99.0), 'education'] = 'T'
    frame.loc[filter_1989 & ((frame.education_1989_revision >= 99.0) | (frame.education_1989_revision.isna())), 'education'] = 'U'
    
    filter_2003 = frame['education_reporting_flag'] == 1
    frame.loc[filter_2003 & (frame.education_2003_revision <= 1), 'education'] = 'P'
    frame.loc[filter_2003 & (frame.education_2003_revision > 1) & (frame.education_2003_revision <= 3.0), 'education'] = 'H'
    frame.loc[filter_2003 & (frame.education_2003_revision > 3) & (frame.education_2003_revision < 9), 'education'] = 'T'
    frame.loc[filter_2003 & ((frame.education_2003_revision == 9) | (frame.education_2003_revision.isna())), 'education'] = 'U'
    return frame<jupyter_output><empty_output><jupyter_text>Similarly we have to normalize the age: we treat any death before 12 months as 1 year.<jupyter_code>def normalize_age(frame):
    frame['age'] = np.nan
    
    less_than_12_months = (frame['detail_age_type'] > 1) & (frame['detail_age_type'] < 9)
    age_in_years = frame['detail_age_type'] == 1
    unknown_age = frame['detail_age_type'] == 9
    
    frame.loc[less_than_12_months, 'age'] = 1
    frame.loc[age_in_years, 'age'] = frame[age_in_years].detail_age
    frame.loc[unknown_age, 'age'] = np.nan
    
    return frame
df = normalize_education(df)
df = normalize_age(df)
df.describe()<jupyter_output><empty_output><jupyter_text>We can see in the dataframe description that some of the age values are missing. For the sake of simplity we can drop those rows.<jupyter_code>df['age'] = df['age'].astype(np.float32)
df = df.dropna(subset=['age'])
df.sample(5)<jupyter_output><empty_output><jupyter_text>FastAI Our data is now ready for us to train the model. We won't be using all the columns (as some columns are simply flags indicating the type of another column). So we discard all the columns we don't need. We also split or training data into a training and validation dataset.<jupyter_code>df_fastai = df[['resident_status', 'education', 'sex', 'marital_status', 'race', 'age']]
df_fastai.sample(5)
train_df, valid_df = train_test_split(df_fastai, test_size=0.33)<jupyter_output><empty_output><jupyter_text>The FastAI API requires us to define the dependent variable, which is the variable we are trying to predict. This is `age` in our case. Further, we need to specify the categorical columns by name such that FastAI can automatically handle them for us (more on this below).<jupyter_code>dep_var = 'age'
categorical_names = ['education', 'sex', 'marital_status']

tfms = [FillMissing, Categorify]<jupyter_output><empty_output><jupyter_text>Notice how we also define to transformations: `FillMissing` and `Categorify`.The `FillMissing` transform will fill in any missing values for the _continuous_ variable columns we have using the median to do so (by default). Read more about the transformer [here](http://docs.fast.ai/tabular.transform.htmlclass-fillmissing).The `Categorify` transformation will take categorical data, automatically assign ids to each of the unqiue values and replace the column values with the ids. A special id is used for missing values. Read more about the transformer [here](http://docs.fast.ai/tabular.transform.htmlclass-fillmissing). We can now setup a [Tabular Dataset](http://docs.fast.ai/tabular.data.htmlTabularDataset) for training.  In addition to applying the transformations above the `TabularDataset` will also *normalize* the continuous variables for us.<jupyter_code>tabular_data = tabular_data_from_df('output', train_df, valid_df, dep_var, tfms=tfms, cat_names=categorical_names)<jupyter_output>/home/avanwyk/anaconda/envs/fastai/lib/python3.6/site-packages/fastai/tabular/transform.py:30: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  df[n] = df[n].astype('category').cat.as_ordered()
/home/avanwyk/anaconda/envs/fastai/lib/python3.6/site-packages/fastai/tabular/transform.py:35: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  df[n] = pd.Categorical(df[n], categories=self.categories[n], ordered=True)<jupyter_text>Model We are now ready to create a [TabularModel](http://docs.fast.ai/tabular.models.htmlclass-tabularmodel). TabularModels are relatively simple models that have Embedding layers for each of the categorical variables which are concatenated with the continuous layers before being passed to a fully connected layers. The model also includes Dropout and Batchnorm layers where expected (see below for detailed depiction of the layers).In our case we use 2 fully connected layers of dimension 100. Since we are doing regression, we then have an output layer of a single dimension (which will automatically have a Linear activation) and we use a mean squared error loss function. We also specify embedding layer sizes for the two categorical layers.<jupyter_code>learn = get_tabular_learner(tabular_data,
                            layers=[100,50,1],
                            emb_szs={'education': 6,
                                     'sex': 5,
                                     'marital_status': 8})
learn.loss_fn = F.mse_loss<jupyter_output><empty_output><jupyter_text>The code below outputs a detailed description of the model:<jupyter_code>learn.model<jupyter_output><empty_output><jupyter_text>Finally, before starting training, we need to choose an appropriate learning rate. FastAI provides an extremely neat trick for finding an appropriate learning rate. See [here](http://docs.fast.ai/callbacks.lr_finder.html) and [here](https://arxiv.org/abs/1506.01186) for details.Applying the method is very straightfoward:<jupyter_code>learn.lr_find()<jupyter_output><empty_output><jupyter_text>After running `lr_find()` we can plot the losses against the learning rates that were tried:<jupyter_code>learn.recorder.plot()<jupyter_output><empty_output><jupyter_text>We then choose a value that is smaller (FastAI recommends by an order of magnitude) than the point at which the loss explodes. In our case there is not increase in loss even for large learning rates, so we could try even bigger start and stop values for lr find.. For now, we will use `1e-1`.<jupyter_code>lr = 1e-1<jupyter_output><empty_output><jupyter_text>We can now train our model. FastAI supports training with [cyclical learning rates](http://docs.fast.ai/callbacks.one_cycle.htmlOneCycleScheduler) (the only deep learning libary to do so out of the box), which is used when we call the `fit_one_cycle` method. Here we perform a single epoch of training:<jupyter_code>learn.fit_one_cycle(1, lr)<jupyter_output><empty_output><jupyter_text>Great! Now we have a trained model, although it definitely seems to be overfitting on our data. The `recorder` records a lot of training data for us. For example, we can easily plot our loss per iterations:<jupyter_code>learn.recorder.plot_losses()<jupyter_output><empty_output><jupyter_text>Our learning rate helped a lot, as we can see, the network converged very early in the training.We can now continue training with smaller learning rates if we wanted to and attempt to find a better minima.  Random Forest Baseline As shown [here](https://gist.github.com/dienhoa/a5adf923bd8b24b3d0eadcd61aec8c2e), we can re-use the FastAI datasets (and pre-processing) to build training data for other algorithms. For example, here we build a training and validation set for a random forest regressor to use a baseline for our DNN.<jupyter_code>def get_arrays(ds):
    X_cats,X_conts = ds.cats.numpy(), ds.conts.numpy()
    y = ds.y.numpy()
    return np.concatenate((X_cats, X_conts), axis=1), y
X_train, y_train = get_arrays(tabular_data.train_ds)
X_val, y_val = get_arrays(tabular_data.valid_ds)
from sklearn.ensemble import RandomForestRegressor

rf = RandomForestRegressor()

rf.fit(X_train,y_train)

rf.score(X_val, y_val)
from sklearn.metrics import mean_squared_error

mean_squared_error(y_val, rf.predict(X_val))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09654778421611034, 'fraction_numerical': 0.026116053211458417, 'mean_word_length': 5.3360910031023785, 'pattern_counts': {'":': 0, '<': 74, '<?xml version=': 0, '>': 73, 'https://': 5, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '36921', 'license': 'MIT', 'n_tokens_mistral': 4012, 'n_tokens_neox': 3705, 'n_words': 1199, 'path': 'cdc-mortality-tabular-prediction/cdc-mortality.ipynb', 'repo_name': 'avanwyk/fastai-projects', 'size': '12253'}