<jupyter_start><jupyter_text>Titanic Tutorial <NAME>17 October 2018 1. What is the purpose of this project?2. Explain why we have Test and Train dataset. Give some quick stats on each.3. Explain your first bar chart - what is it telling us?4. Explain your first line chart - what is it telling us?5. What is overfitting? Why do we care?6. Explain the transformations. The purpose of this project is:  Version History ver / valid / kaggle / Descriptionv1 - 75%    -  75%            Basic decision tree model with label encoding  Table of Contents 0. Download Dataset1. Importing a DataFrame2. Visualize the Data3. Cleanup and Transform the Data4. Encode the Data5. Split Training and Test Sets6. Use a Decision Tree Model7. Upload to Kaggle  0. Download Dataset * Sign up for an account on Kaggle.com* Search for and open the Titanic competition https://www.kaggle.com/c/titanic* Read about the case on the Overview tab and read about the data on the Data tab* Download the dataset to the folder you will be working in* Create a notebook in the same folder that you extract the downloaded files into  1. Importing a DataFrame<jupyter_code>import numpy as np 
import pandas as pd 
data_train = pd.read_csv('train.csv') 
data_test = pd.read_csv('test.csv')
data_train.sample(3)<jupyter_output><empty_output><jupyter_text>2. Visualize the Data Visualizing data is crucial for recognizing underlying patterns to exploit in the model.<jupyter_code>import matplotlib.pyplot as plt
import seaborn as sns

%matplotlib inline
sns.barplot(x="Embarked", y="Survived", hue="Sex", data=data_train);
sns.pointplot(x="Pclass", y="Survived", hue="Sex",
    data=data_train, 
    palette={"male": "blue", "female": "pink"},
    markers=["*", "o"], linestyles=["-", "--"]);<jupyter_output>c:\users\student\appdata\local\programs\python\python37\lib\site-packages\scipy\stats\stats.py:1713: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  return np.add.reduce(sorted[indexer] * weights, axis=axis) / sumval<jupyter_text>Describe what is shown by these graphs.  3. Cleanup and Transform the Data Simplify or remove features using functions. This will avoid overfitting.Define overfitting in your own words: I'm grouping people into logical human age groups.<jupyter_code>def simplify_ages(df): 
    df.Age = df.Age.fillna(-0.5) 
    bins = (-1, 0, 5, 12, 18, 25, 35, 60, 120) 
    group_names = ['Unknown', 'Baby', 'Child', 'Teenager', 'Student',
        'Young Adult', 'Adult', 'Senior'] 
    categories = pd.cut(df.Age, bins, labels=group_names) 
    df.Age = categories 
    return df<jupyter_output><empty_output><jupyter_text>Trim the cabins to just have the first letter of the cabin type.<jupyter_code>def simplify_cabins(df):
    df.Cabin = df.Cabin.fillna('N')
    df.Cabin = df.Cabin.astype(str).str[0] 
    return df<jupyter_output><empty_output><jupyter_text>Fare is another continuous value that should be simplified. I ran `data_train.Fare.describe()` to get the distribution of the feature, then placed them into quartile bins accordingly.<jupyter_code>def simplify_fares(df): 
    df.Fare = df.Fare.fillna(-0.5) 
    bins = (-1, 0, 8, 15, 31, 1000) 
    group_names = ['Unknown', '1_quartile', '2_quartile', '3_quartile', '4_quartile'] 
    categories = pd.cut(df.Fare, bins, labels=group_names) 
    df.Fare = categories 
    return df<jupyter_output><empty_output><jupyter_text>Drop unnecessary columns. You might find a way to repurpose or use these columns in an improvement.<jupyter_code>def drop_features(df): 
    return df.drop(['Ticket', 'Name', 'Embarked'], axis=1)<jupyter_output><empty_output><jupyter_text>Apply transformations to training and testing dataset<jupyter_code>def transform_features(df): 
    df = simplify_ages(df) 
    df = simplify_cabins(df) 
    df = simplify_fares(df) 
    df = drop_features(df) 
    return df
data_train = transform_features(data_train) 
data_test = transform_features(data_test) 
data_train.head()<jupyter_output><empty_output><jupyter_text>*Bonus Material Use this function to quickly visualize any categorical attribute<jupyter_code>def visualize_cat(attr, df=data_train):
    df_i = df[['Survived', attr]].groupby([attr])['Survived'].agg(['count','sum', 'mean'])
    m = 'Pct Survived'
    df_i = df_i.rename({'count': 'frequency', 'sum': 'Survivers', 'mean': m}, axis='columns')
    print(df_i)
    fig, ax1 = plt.subplots()
    ax2 = ax1.twinx()
    df_i['frequency'].plot.bar(ax=ax1, alpha=.8)
    df_i[m].plot.line(ax=ax2, color='k')
    ax1.set_ylabel('frequency')
    ax2.set_ylabel('Pct Survived')
    ax2.set_ylim(bottom=0, top=1, auto=False)
    ax2.legend()
visualize_cat('SibSp')
visualize_cat('Sex')
visualize_cat('Age')<jupyter_output>frequency  Survivers  Pct Survived
Age                                            
Unknown            177         52      0.293785
Baby                44         31      0.704545
Child               25          9      0.360000
Teenager            70         30      0.428571
Student            162         54      0.333333
Young Adult        196         83      0.423469
Adult              195         78      0.400000
Senior              22          5      0.227273<jupyter_text>4. Encode the Data The last part of the preprocessing phase is to normalize labels. The LabelEncoder in Scikit-learn will convert each unique string value into a number, making out data more flexible for various algorithms. Consider the limitations of a label encoder and try using a different encoding method in an improvement of this project.* https://www.datacamp.com/community/tutorials/categorical-data<jupyter_code>from sklearn import preprocessing 
def encode_features(df_train, df_test): 
    features = ['Fare', 'Cabin', 'Age', 'Sex'] 
    df_combined = pd.concat([df_train[features], df_test[features]]) 
    for feature in features: 
        le = preprocessing.LabelEncoder() 
        le = le.fit(df_combined[feature]) 
        df_train[feature] = le.transform(df_train[feature]) 
        df_test[feature] = le.transform(df_test[feature]) 
    return df_train, df_test 
data_train, data_test = encode_features(data_train, data_test) 
data_train.head()<jupyter_output><empty_output><jupyter_text>5. Split Training and Test Sets * Now its time for some Machine Learning.* First, separate the features(X) from the labels(y).* X_all: All features minus the value we want to predict (Survived).* y_all: Only the value we want to predict.* Second, use Scikit-learn to randomly shuffle this data into four variables. In this case, I'm training 80% of the data, then testing against the other 20%.<jupyter_code>from sklearn.model_selection import train_test_split 

X_all = data_train.drop(['Survived', 'PassengerId'], axis=1) 
y_all = data_train['Survived'] 

num_test = 0.20 

X_train, X_test, y_train, y_test = train_test_split(X_all, y_all, 
    test_size=num_test, random_state=23)<jupyter_output><empty_output><jupyter_text>6. Use a Decision Tree Model<jupyter_code>from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score

# Choose the type of classifier. 
clf = DecisionTreeClassifier() 

# Fit the algorithm to the data. 
clf.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Test the accuracy of the model on the testing data set<jupyter_code>predictions = clf.predict(X_test)
print(accuracy_score(y_test, predictions))<jupyter_output>0.8324022346368715<jupyter_text>7. Upload to Kaggle Now make the predictions on the actual testing dataset from Kaggle<jupyter_code>ids = data_test['PassengerId'] 
predictions = clf.predict(data_test.drop('PassengerId', axis=1))
output = pd.DataFrame({ 'PassengerId' : ids, 'Survived': predictions }) 
output.to_csv('titanic-predictions.csv', index = False) 
output.head(10)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10306058713304185, 'fraction_numerical': 0.026108682073703936, 'mean_word_length': 4.229261920313521, 'pattern_counts': {'":': 2, '<': 51, '<?xml version=': 0, '>': 51, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '198854', 'license': 'MIT', 'n_tokens_mistral': 2771, 'n_tokens_neox': 2535, 'n_words': 774, 'path': 'Titanic Tutorial - Original.ipynb', 'repo_name': 'cd412/Titanic-Tutorial', 'size': '8014'}