<jupyter_start><jupyter_text>Kaggle Titanic Survivor Predictions<jupyter_code>from fastai.tabular import *<jupyter_output><empty_output><jupyter_text>Download data from Kaggle competition page Step 1: Fetch/download data from Kaggle website. Using instructions from Lesson 3 (copied here for reference). First, install the Kaggle API by uncommenting the following line and executing it, or by executing it in your terminal (depending on your platform you may need to modify this slightly to either add source activate fastai or similar, or prefix pip with a path. Have a look at how conda install is called for your platform in the appropriate Returning to work section of https://course.fast.ai/. (Depending on your environment, you may also need to append "--user" to the command.)<jupyter_code># install kaggle apis first
#! pip install kaggle --upgrade<jupyter_output>Collecting kaggle
[?25l  Downloading https://files.pythonhosted.org/packages/80/4d/fc83980735a7a2e3aaba1ed755462e839c68ba51ac311efdb1ab413a4eeb/kaggle-1.5.2.tar.gz (54kB)
[K    100% |████████████████████████████████| 61kB 30.1MB/s ta 0:00:01
[?25hCollecting urllib3<1.23.0,>=1.15 (from kaggle)
[?25l  Downloading https://files.pythonhosted.org/packages/63/cb/6965947c13a94236f6d4b8223e21beb4d576dc72e8130bd7880f600839b8/urllib3-1.22-py2.py3-none-any.whl (132kB)
[K    100% |████████████████████████████████| 133kB 50.8MB/s ta 0:00:01
[?25hRequirement not upgraded as not directly required: six>=1.10 in /opt/anaconda3/lib/python3.7/site-packages (from kaggle) (1.11.0)
Requirement not upgraded as not directly required: certifi in /opt/anaconda3/lib/python3.7/site-packages (from kaggle) (2018.11.29)
Requirement not upgraded as not directly required: python-dateutil in /opt/anaconda3/lib/python3.7/site-packages (from kaggle) (2.7.5)
Requirement not upgraded as not directly required: request[...]<jupyter_text>Then you need to upload your credentials from Kaggle on your instance. Login to kaggle and click on your profile picture on the top left corner, then 'My account'. Scroll down until you find a button named 'Create New API Token' and click on it. This will trigger the download of a file named 'kaggle.json'.Upload this file to the directory this notebook is running in, by clicking "Upload" on your main Jupyter page, then uncomment and execute the next two commands (or run them in a terminal).<jupyter_code># double check - have you uploaded the latest/valid API token file to working directory?
# ok then... now setting up credentials
#! mkdir -p ~/.kaggle/
#! mv kaggle.json ~/.kaggle/<jupyter_output><empty_output><jupyter_text>You're all set to download the data from [titanic competition](https://www.kaggle.com/c/titanic). You **first need to go to its main page and accept its rules**, and run the two cells below (uncomment the shell commands to download and unzip the data). If you get a `403 forbidden` error it means you haven't accepted the competition rules yet (you have to go to the competition page, click on *Rules* tab, and then scroll to the bottom to find the *accept* button).<jupyter_code>#Setup local directory to receive the dataset
path = Config.data_path()/'titanic'
path.mkdir(parents=True, exist_ok=True)
path
# initiate download of data - replace with appropriate competition string - examples below
# ! kaggle competitions download -c planet-understanding-the-amazon-from-space -f train-jpg.tar.7z -p {path}  
# ! kaggle competitions download -c planet-understanding-the-amazon-from-space -f train_v2.csv -p {path}  
# ! unzip -q -n {path}/train_v2.csv.zip -d {path}

#! kaggle competitions download -c titanic -f train.csv -p {path}
#! kaggle competitions download -c titanic -f test.csv -p {path}<jupyter_output>Warning: Your Kaggle API key is readable by other users on this system! To fix this, you can run 'chmod 600 /home/jupyter/.kaggle/kaggle.json'
Downloading train.csv to /home/jupyter/.fastai/data/titanic
  0%|                                               | 0.00/59.8k [00:00<?, ?B/s]
100%|██████████████████████████████████████| 59.8k/59.8k [00:00<00:00, 43.2MB/s]
Warning: Your Kaggle API key is readable by other users on this system! To fix this, you can run 'chmod 600 /home/jupyter/.kaggle/kaggle.json'
Downloading test.csv to /home/jupyter/.fastai/data/titanic
  0%|                                               | 0.00/28.0k [00:00<?, ?B/s]
100%|██████████████████████████████████████| 28.0k/28.0k [00:00<00:00, 21.0MB/s]<jupyter_text>If your test data has 7z zip files you will also need to install the 7zip utility. You can do this on command line (sudo apt install p7zip) or run these commands to install and unzip:<jupyter_code># Note: Run these only if you have 7zip files in the data and need to unpack them
#! conda install -y -c haasad eidl7zip
#! 7za -bd -y -so x {path}/train-jpg.tar.7z | tar xf - -C {path}<jupyter_output><empty_output><jupyter_text>Setup data for analysis and training - Using Pandas The Python Pandas library allows easy operations on tabular and time series data. It uses a DataFrame object to provide access to and execute operations on the data. First step is to move csv data into a pandas dataframe.<jupyter_code># read in the training data first
traindf = pd.read_csv(path/'train.csv')
# also read in the test data
testdf = pd.read_csv(path/'test.csv')<jupyter_output><empty_output><jupyter_text>Print out the train and test df and explore a little before converting into fastai data formats. But first let's understand the data from the Kaggle info. Here are the columns provided in the datasets:1. PassengerId - Numeric - type should be integers2. Survived - Numeric/Binary - Survived (1) or Not (0)3. Pclass - Numeric - Class of Travel (1 = 1st, 2 = 2nd, 3 = 3rd)4. Name - String - Name of Passenger5. Sex - String - Gender (M/F)6. Age - Numeric - Age of Passengers in Years (can be fractional or estimated "xx.")7. SibSp - String - Number of Sibling/Spouse aboard8. Parch - Numeric - Number of Parent/Child aboard9. Ticket - String - Actual Ticket number10. Fare - Numeric - Cost of ticket11. Cabin - String - Cabin number/name12. Embarked - String - The port in which a passenger has embarked. C - Cherbourg, S - Southampton, Q = QueenstownNotes to keep in mind:pclass: A proxy for socio-economic status (SES)1st = Upper2nd = Middle3rd = Lowersibsp: The dataset defines family relations in this way...Sibling = brother, sister, stepbrother, stepsisterSpouse = husband, wife (mistresses and fiancés were ignored)For some reason Kaggle gives this as string; though entries are numeric.parch: The dataset defines family relations in this way...Parent = mother, fatherChild = daughter, son, stepdaughter, stepsonSome children travelled only with a nanny, therefore parch=0 for them.<jupyter_code>print(traindf.shape)
print(testdf.shape)
#print(traindf.rows)
print(testdf.index)
print(traindf.columns)
print(testdf.columns)
#print(testdf.loc[345])
#print(testdf.loc[0][8])
print(traindf.loc[10])
print(testdf.loc[10])
#print(testdf)
# Plugging in code here to fix the NaN entry for Fare in test data set. See way down below for full story.
#import math
for index, row in testdf.iterrows():
#    print(index)
#    print(testdf.loc[index]['Fare'])
    if(math.isnan(testdf.loc[index]['Fare'])):
        print(index)
        print(testdf.loc[index])
        testdf.at[index, 'Fare'] = 0
        print(testdf.loc[index])<jupyter_output>152
PassengerId                  1044
Pclass                          3
Name           Storey, <NAME>
Sex                          male
Age                          60.5
SibSp                           0
Parch                           0
Ticket                       3701
Fare                          NaN
Cabin                         NaN
Embarked                        S
Name: 152, dtype: object
PassengerId                  1044
Pclass                          3
Name           Storey, <NAME>
Sex                          male
Age                          60.5
SibSp                           0
Parch                           0
Ticket                       3701
Fare                            0
Cabin                         NaN
Embarked                        S
Name: 152, dtype: object<jupyter_text>Ok, time to create tabular data for training the model from the train dataframe. Dependent variable: one that you want to predict. so in this case "survived".Categorical variables: ones that are discrete/fixed. eg name gender etcContinuous variables: ones that are continuous or have high cardinality (uncommon/unique) or are categorical but with too many options. eg age, salaryProcedures: functions that fastai can perform on a tabular dataset to augment it. done only once at start; not on the fly like in case of transforms for vision datasets.So now, we need to decide which of these to set and how.Here's the v1 thinking:DV is simple and done.Categorical - Name, Sex, Ticket, Cabin and Embarked - these are obvious categorical variablesCategorical - Pclass, SibSp, Parch - possibly also categorical? Continuous - PassengerID, Age, Fare - ID and age for sure, but fare also is most likely continuousProcedures: For now just use the ones that were used in lesson 3.<jupyter_code>dep_var = 'Survived'
cat_names = ['Name', 'Sex', 'Ticket', 'Cabin', 'Embarked', 'Pclass', 'SibSp', 'Parch']
cont_names = ['PassengerId', 'Age', 'Fare']
procs = [FillMissing, Categorify, Normalize]
#create the test TabularList data
test = TabularList.from_df(testdf.copy(), path=path, cat_names=cat_names, cont_names=cont_names)
print(len(test))
print(test[10])<jupyter_output>418
PassengerId                 902
Pclass                        3
Name           Ilieff, Mr. Ylio
Sex                        male
Age                         NaN
SibSp                         0
Parch                         0
Ticket                   349220
Fare                     7.8958
Cabin                       NaN
Embarked                      S
Name: 10, dtype: object<jupyter_text>Note on the test code below:In the first run I got an error when the preprocessor "FillMissing" was called on my test itemlist. Error was:```/opt/anaconda3/lib/python3.7/site-packages/fastai/tabular/transform.py in apply_test(self, df)     95             elif pd.isnull(df[name]).sum() != 0:     96                 raise Exception(f"""There are nan values in field {name} but there were none in the training set. ---> 97                 Please fix those manually.""")     98      99 class Normalize(TabularProc):Exception: There are nan values in field Fare but there were none in the training set.                 Please fix those manually.```So I wrote code to figure out which entries in test set had fare value of not-a-number. Turns out only 1 (with passenger ID 1044). But I can't fix it in the item list for its a slice/copy of the test dataframe. So I guess I need to fix in the dataframe itself.Leaving this code in here for kicks.<jupyter_code># TEST CODE - NO NEED TO RUN
#import math
#for r in range(len(test)):
#    if(math.isnan(test[r]['Fare'])):
#        print(r)
#        print(test[r])<jupyter_output>152
PassengerId                  1044
Pclass                          3
Name           Storey, Mr. Thomas
Sex                          male
Age                          60.5
SibSp                           0
Parch                           0
Ticket                       3701
Fare                          NaN
Cabin                         NaN
Embarked                        S
Name: 152, dtype: object<jupyter_text>Testing Databunch API Creating databunch below is causing the data to get messed up in the test itemlist and the train itemlist. So I'm going to break up the steps and instead of using databunch api do one step at a time.<jupyter_code>#the test itemlist is setup fine as above so going to create train item list now
#create the train TabularList data
#trainsrc = (TabularList.from_df(traindf, path=path, cat_names=cat_names, cont_names=cont_names)
#            .split_by_idx(valid_idx=range(800,890)))
#print(len(trainsrc))
#print(trainsrc.train)
#print(trainsrc[890])
data = (TabularList.from_df(traindf, path=path, cat_names=cat_names, cont_names=cont_names, procs=procs)
                           .split_by_idx(valid_idx=range(800,890))
                           .label_from_df(cols=dep_var)
                           .add_test(test)
                           .databunch())
data.train_ds[10]
data.test_ds[10]<jupyter_output><empty_output><jupyter_text>Ok something went wrong here while creating databunch. Why is passenger ID and age negative!! Need to check this out.<jupyter_code>data.show_batch(5)<jupyter_output><empty_output><jupyter_text>Creating the tabular learner to then train the model.<jupyter_code>learn = tabular_learner(data, layers=[200,100], metrics=accuracy)
learn
learn.fit(20, 1e-2)
learn.lr_find()
learn.recorder.plot()
learn.unfreeze()
learn.fit(20, 5e-2)
learn.lr_find()
learn.recorder.plot()
#from IPython.core.debugger import set_trace
#set_trace()
#learn.fit_one_cycle(20, max_lr=slice(1e-4, 1e-1))
#%debug
learn.unfreeze()
learn.fit(20, 1e-1)
learn.lr_find()
learn.recorder.plot()
learn.save('stage-1-lowtlhighvlacc81')<jupyter_output><empty_output><jupyter_text>Inference<jupyter_code>row = testdf.iloc[30]
print(row)
p = learn.predict(row)
print(p)
pn = p[1].numpy()
print(pn)
#learn.get_preds()
# Create results dataframe
rescolnames = ['PassengerId', 'Survived']
results = pd.DataFrame(columns=rescolnames)
print(results)
# Predict and populate results df
for index, row in testdf.iterrows():
    prediction = learn.predict(row)
    predval = prediction[1].numpy()
    passid = testdf.loc[index]['PassengerId']
    results = results.append({'PassengerId':passid, 'Survived':predval}, ignore_index=True)
#    if(index==1):
#        print(prediction, predval, passid)
#        print(row)
print(results)
# move results to csv file to submit on Kaggle
resultsfile = results.to_csv(path/'titanicresultsv1.csv', index=False)
# This is it. First iteration done!<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09929431891082757, 'fraction_numerical': 0.03143488488131727, 'mean_word_length': 3.136202830188679, 'pattern_counts': {'":': 0, '<': 55, '<?xml version=': 0, '>': 53, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 18, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '325433', 'license': 'Apache-2.0', 'n_tokens_mistral': 4700, 'n_tokens_neox': 4302, 'n_words': 1511, 'path': 'nbs/dl1/nb-Kaggle-Titanic-v1.ipynb', 'repo_name': 'nalinicommits/nbonfastai', 'size': '14037'}