<jupyter_start><jupyter_text>Titanic Part 3> "Neural Networks: do they work in Tabular? Third part!"- toc:true- branch: master- badges: true- comments: true- author: <NAME>- categories: [Deep Learning, Jupyter, Tabular Data, Neural Networks, Titanic] After spending a lot of time experimenting with different hyperparameter options, I came to the realization that I should perhaps leave aside the _Random Forest Regressor_ for the time being, as I did not want to dedicate anymore time studying related literature with seemingly marginal improvements. Instead, I decided to try out a new type of algorithm; one that is not widely used for Tabular data, and that has quite a few complications both in its training and its tuning in order to achieve a decent model.That is, of course, __Neural Networks__.  Neural NetworksI will again be using the fastai library, which adds functionality on top of PyTorch.[I have been using _fastai_ in every project displayed thus far in this blog. That is mainly because it provides a reliable and resilient API that handles low-level details, thereby allowing the engineer to focus on high-level abstraction, which can result in less implementing and more experimenting - and in case of learning and researching, the more experimenting the merrier.]  DataI generally followed the same _data preprocessing_ routine (which you can find here) in this implementation as well, with a few largely unimportant exceptions (such as removing redundant columns and swapping the classification of one or two (depending on the model attempted) column names from _Categorical_ to _Continuous)._<jupyter_code>splits = RandomSplitter(valid_pct=0.2)(range_of(df))
to = TabularPandas(df, 
                   procs, 
                   cat, 
                   cont, 
                   dep_var, 
                   splits=splits, y_block = CategoryBlock)<jupyter_output><empty_output><jupyter_text>NNs being NNs require special treatment regarding the format in which the data is inputted in the model. That's why it is necessary to create _DataLoaders batches,_ before constructing the _fastai-powered_ _Learner._<jupyter_code>dls = to.dataloaders(bs=64)<jupyter_output><empty_output><jupyter_text>Classification rather than Regression From now on I will be treating the competition as a __classification__ problem, I thereby formatted the data's labels as _categories_ _(y_block=CategoryBlock)._ This is the main difference between the data used thus far and the new _TabularPandas_ instance above.[Later on prepare to have that approach challenged, as I wanted to play around with distinct but not necessarily contrasting paths.]  Concluding the Data Preprocessing<jupyter_code>len(to.train),len(to.valid)<jupyter_output><empty_output><jupyter_text>(713, 177) A bigger training set can lead to a better data representation in the model, but a smaller validation set can exclude important features from being accounted for.  Learner & Training Wherever it is not specified otherwise, consider that the only parameters changed are the ones explicitly mentioned; in other words, all the default values were left as is unless otherwise indicated.  First model<jupyter_code>learn = tabular_learner(dls, metrics=accuracy)<jupyter_output><empty_output><jupyter_text>Training<jupyter_code>#hide_input
from IPython.display import Image

Image("lf4.PNG")<jupyter_output><empty_output><jupyter_text>A few initial notes:- The model is training unusually fast. But then again, it is training on Tabular data, and, even more so, only a handful of them.- Its accuracy is already decent.- One would expect that the model is going to start overfitting fairly soon, as I have not adjusted the training _learning rate_ (nor will I do so in this routine).<jupyter_code>#hide_input
Image("lf10.PNG")<jupyter_output><empty_output><jupyter_text>As expected, the model started overfitting early on. It hit, however, an accuracy of ~84.2% right before that. Perhaps with a few tweaks the next one can become even better.  Review<jupyter_code>#hide_input
Image("lres.PNG")<jupyter_output><empty_output><jupyter_text>Inspecting a few random results, we see that the values of the columns _Survived_ and _Survived_pred_ are almost identical. The labels and the predictions are, therefore, in this small sample fairly well matched, even after training for way too long for the model not to overfit.  Targeted Predictions I should also mention that it is possible to get the model's predictions on a specified row (or series of rows) of a _DataFrame_ object. An example case is presented below.<jupyter_code>#hide_input
Image("rcprobs.PNG")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.05256382750482729, 'fraction_numerical': 0.0036472859901308732, 'mean_word_length': 5.150395778364116, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 35, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '173932', 'license': 'Apache-2.0', 'n_tokens_mistral': 1251, 'n_tokens_neox': 1189, 'n_words': 595, 'path': '_notebooks/2020-10-15-Titanic_PART_3.ipynb', 'repo_name': 'ntoump/blog', 'size': '4675'}