<jupyter_start><jupyter_text>Real World Example:  AI, Machine Learning & Data Science ---  What is the Value for your Business?- By seeing acutal examples you'll be empowered to ask the right questions (and get fair help from consultants, startups, or data analytics companies)- This will help you make the correct decisions for your business DemystifyThis is a real world example of how you'd solve a Machine Learning prediciton problem.**Common Machine Learning Use Cases in Companies:**- Discover churn risk of customers- Predict optimal price levels (investments / retail)- Predict future revenues- Build recommendation systems- Customer value scoring- Fraud detection- Customer insights (characteristics)- Predict sentiment of text / client feedback- Object detecton in images- etc etc...  Why Python?Python is general purpose and can do Software development, Web development, AI. Python has experienced incredible growth over the last couple of years.Source: https://stackoverflow.blog/2017/09/06/incredible-growth-python/  Everything is free!The best software today is open source and it's also enterprise-ready. Anyone can download and use them for free (even for business purposes).**Examples of great, free AI libraries:*** Anaconda* Google's TensorFlow* Scikit-learn* Pandas* Keras* Matplotlib* SQL* Spark* Numpy State-of-the-Art algorithmsNo matter what algorithm you want to use (Linear Regression, Random Forests, Neural Networks, or Deep Learning), **all of the latest methods are implemented optimized for Python**.  Big DataPython code can run on any computer. Therefore, you can scale your computations and utilize for example cloud resources to run big data jobs.**Great tools for Big Data:**- Spark- Databricks- Hadoop / MapReduce- Kafka- Amazon EC2- Amazon S3 Note on data collection- Collect all the data you can! (storage is cheap)--- ---- Real world example of AI: Titanic AnalysisTitanic notebook is open source. All of our material is online. Anyone can developt sophisticated AI programs and solutions. ___ The difficult part is never to implement the algorithmThe hard part of a machine learning problem is to get data into the right format so you can solve the problem. We'll illustrate this below.___ ![data-x](http://oi64.tinypic.com/o858n4.jpg) __Titanic Survivor Analysis__**Sources:** * **Training + explanations**: https://www.kaggle.com/c/titanic______  Understanding the connections between passanger information and survival rateThe sinking of the RMS Titanic is one of the most infamous shipwrecks in history.  On April 15, 1912, during her maiden voyage, the Titanic sank after colliding with an iceberg, killing 1502 out of 2224 passengers and crew. This sensational tragedy shocked the international community and led to better safety regulations for ships.One of the reasons that the shipwreck led to such loss of life was that there were not enough lifeboats for the passengers and crew. Although there was some element of luck involved in surviving the sinking, some groups of people were more likely to survive than others. **Our task is to train a machine learning model on a data set consisting of 891 samples of people who were onboard of the Titanic. And then, be able to predict if the passengers survived or not.**  Import packages<jupyter_code># No warnings
import warnings
warnings.filterwarnings('ignore') # Filter out warnings

# data analysis and wrangling
import pandas as pd
import numpy as np

# visualization
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns

# machine learning
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC, LinearSVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.naive_bayes import GaussianNB # Gaussian Naive Bays
from sklearn.linear_model import Perceptron
from sklearn.tree import DecisionTreeClassifier

import xgboost as xgb

from plot_distribution import plot_distribution
plt.rcParams['figure.figsize'] = (9, 5)<jupyter_output><empty_output><jupyter_text>Load Data<jupyter_code>df = pd.read_csv('data/train.csv')<jupyter_output><empty_output><jupyter_text>___ Part 2: Exploring the Data**Data descriptions**<jupyter_code># preview the data
df.head(3)
# General data statistics
df.describe()<jupyter_output><empty_output><jupyter_text>Histograms<jupyter_code>df.hist(figsize=(13,10));
# Balanced data set?
y_numbers = df['Survived'].map({0:'Deceased',1:'Survived'}).value_counts()

y_numbers
# Imbalanced data set, our classifiers have to outperform 62 % accuracy

y_numbers[1] / y_numbers[0]<jupyter_output><empty_output><jupyter_text>>  __Interesting Fact:__ > Third Class passengers were the first to board, with First and Second Class passengers following up to an hour before departure. > Third Class passengers were inspected for ailments and physical impairments that might lead to their being refused entry to the United States, while First Class passengers were personally greeted by Captain Smith.<jupyter_code># Analysis of survival rate for the socioeconmic classes?

df[['Pclass', 'Survived']].groupby(['Pclass'], as_index=True) \
                    .mean().sort_values(by='Survived', ascending=False)<jupyter_output><empty_output><jupyter_text>___>  __Brief Remarks Regarding the Data__> * `PassengerId` is a random number (incrementing index) and thus does not contain any valuable information. > * `Survived, Passenger Class, Age, Siblings Spouses, Parents Children` and `Fare` are numerical values (no need to transform them) -- but, we might want to group them (i.e. create categorical variables). > * `Sex, Embarked` are categorical features that we need to map to integer values. `Name, Ticket` and `Cabin` might also contain valuable information.___<jupyter_code>df.head(1)<jupyter_output><empty_output><jupyter_text>Dropping Unnecessary data__Note:__ It is important to remove variables that convey information already captured by some other variable. Doing so removes the correlation, while also diminishing potential overfit.<jupyter_code># Drop columns 'Ticket', 'Cabin', 'Fare' need to do it 
# for both test and training

df = df.drop(['PassengerId','Ticket', 'Cabin','Fare'], axis=1)<jupyter_output><empty_output><jupyter_text>____ Part 3: Transforming the data 3.1 _The Title of the person can be used to predict survival_<jupyter_code># List example titles in Name column
df.Name
# Create column called Title

df['Title'] = df['Name'].str.extract(' ([A-Za-z]+)\.', expand=False)
# Double check that our titles makes sense (by comparing to sex)

pd.crosstab(df['Title'], df['Sex'])
# Map rare titles to one group

df['Title'] = df['Title'].\
              replace(['Lady', 'Countess','Capt', 'Col', 'Don', 'Dr',\
             'Major', 'Rev', 'Sir', 'Jonkheer', 'Dona'], 'Rare')

df['Title'] = df['Title'].replace('Mlle', 'Miss') #Mademoiselle
df['Title'] = df['Title'].replace('Ms', 'Miss')
df['Title'] = df['Title'].replace('Mme', 'Mrs') #Madame
# We now have more logical (contemporary) titles, and fewer groups
# See if we can get some insights

df[['Title', 'Survived']].groupby(['Title']).mean()
# We can plot the survival chance for each title

sns.countplot(x='Survived', hue="Title", data=df, order=[1,0])
plt.xticks(range(2),['Survived','Deceased']);
# Title dummy mapping: Map titles to binary dummy columns

binary_encoded = pd.get_dummies(df.Title)
df[binary_encoded.columns] = binary_encoded
# Remove unique variables for analysis (Title is generally bound to Name, so it's also dropped)
df = df.drop(['Name', 'Title'], axis=1)
df.head()<jupyter_output><empty_output><jupyter_text>Map Gender column to binary (male = 0, female = 1) categories<jupyter_code># convert categorical variable to numeric

df['Sex'] = df['Sex']. \
    map( {'female': 1, 'male': 0} ).astype(int)

df.head()<jupyter_output><empty_output><jupyter_text>Handle missing values for age<jupyter_code>df.Age = df.Age.fillna(df.Age.median())<jupyter_output><empty_output><jupyter_text>Split age into bands and look at survival rates<jupyter_code># Age bands
df['AgeBand'] = pd.cut(df['Age'], 5)
df[['AgeBand', 'Survived']].groupby(['AgeBand'], as_index=False)\
                    .mean().sort_values(by='AgeBand', ascending=True)<jupyter_output><empty_output><jupyter_text>Suvival probability against age<jupyter_code># Plot the relative survival rate distributions against Age of passangers
# subsetted by the gender

plot_distribution( df , var = 'Age' , target = 'Survived' ,\
                  row = 'Sex' )

# Recall: {'male': 0, 'female': 1}
# Change Age column to
# map Age ranges (AgeBands) to ordinal integer numbers

df.loc[ df['Age'] <= 16, 'Age'] = 0
df.loc[(df['Age'] > 16) & (df['Age'] <= 32), 'Age'] = 1
df.loc[(df['Age'] > 32) & (df['Age'] <= 48), 'Age'] = 2
df.loc[(df['Age'] > 48) & (df['Age'] <= 64), 'Age'] = 3
df.loc[ df['Age'] > 64, 'Age']=4
df = df.drop(['AgeBand'], axis=1)

df.head()

# Note we could just run 
# df['Age'] = pd.cut(df['Age'], 5,labels=[0,1,2,3,4])<jupyter_output><empty_output><jupyter_text>Travel Party SizeHow did the number of people the person traveled with impact the chance of survival?<jupyter_code># SibSp = Number of Sibling / Spouses
# Parch = Parents / Children

df['FamilySize'] = df['SibSp'] + df['Parch'] + 1

# Survival chance against FamilySize
df[['FamilySize', 'Survived']].groupby(['FamilySize'], as_index=True) \
                                .mean().sort_values(by='Survived', ascending=False)
# Plot it, 1 is survived

sns.countplot(x='Survived', hue="FamilySize", data=df, order=[1,0]);
# Create binary variable if the person was alone or not

df['IsAlone'] = 0
df.loc[df['FamilySize'] == 1, 'IsAlone'] = 1

df[['IsAlone', 'Survived']].groupby(['IsAlone'], as_index=True).mean()
# We will only use the binary IsAlone feature for further analysis

df.drop(['Parch', 'SibSp', 'FamilySize'], axis=1, inplace=True)

df.head()<jupyter_output><empty_output><jupyter_text>Feature construction<jupyter_code># We can also create new features based on intuitive combinations
# Here is an example when we say that the age times socioclass is a determinant factor

df['Age*Class'] = df.Age.values * df.Pclass.values

df.loc[:, ['Age*Class', 'Age', 'Pclass']].head()<jupyter_output><empty_output><jupyter_text>Port the person embarked fromLet's see how that influences chance of survival >___<jupyter_code># Fill NaN 'Embarked' Values in the dfs
freq_port = df['Embarked'].dropna().mode()[0]
df['Embarked'] = df['Embarked'].fillna(freq_port)
    
# Plot it, 1 is survived

sns.countplot(x='Survived', hue="Embarked", data=df, order=[1,0]);
df[['Embarked', 'Survived']].groupby(['Embarked'], as_index=True) \
                    .mean().sort_values(by='Survived', ascending=False)
# Create categorical dummy variables for Embarked values

binary_encoded = pd.get_dummies(df.Embarked)
df[binary_encoded.columns] = binary_encoded
df.drop('Embarked', axis=1, inplace=True)

df.head()<jupyter_output><empty_output><jupyter_text>Finished -- Preprocessing Complete!<jupyter_code># All features are approximately on the same scale
# no need for feature engineering / normalization

df.head(7)<jupyter_output><empty_output><jupyter_text>Sanity Check: View the correlation between features<jupyter_code># Uncorrelated features are generally more powerful predictors

colormap = plt.cm.viridis
plt.figure(figsize=(12,12))
plt.title('Pearson Correlation of Features', y=1.05, size=15)
sns.heatmap(df.corr().round(2)\
            ,linewidths=0.1,vmax=1.0, square=True, cmap=colormap, \
            linecolor='white', annot=True);<jupyter_output><empty_output><jupyter_text>___ Machine Learning, Prediction and Artifical IntelligenceNow we will use Machine Learning algorithms in order to predict if the person survived. **We will choose the best model from:**1. Logistic Regression2. K-Nearest Neighbors (KNN) 3. Support Vector Machines (SVM)4. Perceptron5. XGBoost6. Random Forest7. Neural Network (Deep Learning)  Setup Training and Validation Sets<jupyter_code>X = df.drop("Survived", axis=1) # Training & Validation data
Y = df["Survived"]              # Response / Target Variable

print(X.shape, Y.shape)
# Split training set so that we validate on 20% of the data
# Note that our algorithms will never have seen the validation 

np.random.seed(1337) # set random seed for reproducibility

from sklearn.model_selection import train_test_split

X_train, X_val, Y_train, Y_val = \
                train_test_split(X, Y, test_size=0.2)

print('Training Samples:', X_train.shape, Y_train.shape)
print('Validation Samples:', X_val.shape, Y_val.shape)<jupyter_output><empty_output><jupyter_text>___>  General ML workflow> 1. Create Model Object> 2. Train the Model> 3. Predict on _unseen_ data> 4. Evaluate accuracy.___  Compare Different Prediciton Models  1. Logistic Regression<jupyter_code>logreg = LogisticRegression()           # create
logreg.fit(X_train, Y_train)            # train
acc_log_2 = logreg.score(X_val, Y_val)  # predict & evaluate

print('Logistic Regression accuracy:',\
      str(round(acc_log_2*100,2)),'%')<jupyter_output><empty_output><jupyter_text>2. K-Nearest Neighbour<jupyter_code>knn = KNeighborsClassifier(n_neighbors = 5)                  # instantiate
knn.fit(X_train, Y_train)                                    # fit
acc_knn = knn.score(X_val, Y_val)                            # predict + evaluate

print('K-Nearest Neighbors labeling accuracy:', str(round(acc_knn*100,2)),'%')<jupyter_output><empty_output><jupyter_text>3. Support Vector Machine<jupyter_code># Support Vector Machines Classifier (non-linear kernel)
svc = SVC()                                                  # instantiate
svc.fit(X_train, Y_train)                                    # fit
acc_svc = svc.score(X_val, Y_val)                            # predict + evaluate

print('Support Vector Machines labeling accuracy:', str(round(acc_svc*100,2)),'%')<jupyter_output><empty_output><jupyter_text>4. Perceptron<jupyter_code>perceptron = Perceptron()                                    # instantiate 
perceptron.fit(X_train, Y_train)                             # fit
acc_perceptron = perceptron.score(X_val, Y_val)              # predict + evalaute

print('Perceptron labeling accuracy:', str(round(acc_perceptron*100,2)),'%')<jupyter_output><empty_output><jupyter_text>5. Gradient Boosting<jupyter_code># XGBoost, same API as scikit-learn
gradboost = xgb.XGBClassifier(n_estimators=1000)             # instantiate
gradboost.fit(X_train, Y_train)                              # fit
acc_xgboost = gradboost.score(X_val, Y_val)                  # predict + evalute

print('XGBoost labeling accuracy:', str(round(acc_xgboost*100,2)),'%')<jupyter_output><empty_output><jupyter_text>6. Random Forest<jupyter_code># Random Forest
random_forest = RandomForestClassifier(n_estimators=500)   # instantiate
random_forest.fit(X_train, Y_train)                         # fit
acc_rf = random_forest.score(X_val, Y_val)                  # predict + evaluate

print('Random Forest labeling accuracy:', str(round(acc_rf*100,2)),'%')<jupyter_output><empty_output><jupyter_text>7. Neural Networks (Deep Learning)<jupyter_code>from keras.models import Sequential
from keras.layers import Dense
model = Sequential()
model.add( Dense(units=300, activation='relu', input_shape=(13,) ))
model.add( Dense(units=100, activation='relu'))
model.add( Dense(units=50, activation='relu'))
model.add( Dense(units=1, activation='sigmoid') )
model.compile(loss = 'binary_crossentropy', optimizer = 'adam', metrics = ['accuracy'])
model.fit(X_train, Y_train, epochs = 50, batch_size= 50)
# # Evaluate the model Accuracy on test set
print('Neural Network accuracy:',str(round(model.evaluate(X_val, Y_val, batch_size=50,verbose=False)[1]*100,2)),'%')<jupyter_output><empty_output><jupyter_text>Importance scores in the random forest model<jupyter_code># Look at importnace of features for random forest

def plot_model_var_imp( model , X , y ):
    imp = pd.DataFrame( 
        model.feature_importances_  , 
        columns = [ 'Importance' ] , 
        index = X.columns 
    )
    imp = imp.sort_values( [ 'Importance' ] , ascending = True )
    imp[ : 10 ].plot( kind = 'barh' )
    print ('Training accuracy Random Forest:',model.score( X , y ))

plot_model_var_imp(random_forest, X_train, Y_train)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11429444105070251, 'fraction_numerical': 0.012461820403176543, 'mean_word_length': 4.901586157173757, 'pattern_counts': {'":': 0, '<': 109, '<?xml version=': 0, '>': 122, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '26', 'dir': 'jupyter-structured-clean-dedup', 'id': '3475', 'license': 'Apache-2.0', 'n_tokens_mistral': 5318, 'n_tokens_neox': 4959, 'n_words': 1586, 'path': '04-tools-prediction-titanic/.ipynb_checkpoints/titanic-checkpoint.ipynb', 'repo_name': 'ByronBay/dataXbwcon', 'size': '16370'}