<jupyter_start><jupyter_text>Sample Data Science WorkbookTemplate for a typical data science workflow.Based on the following [guide](https://www.kaggle.com/startupsci/titanic-data-science-solutions).  Workflow Stages1. Question or problem definition.2. Acquire training and testing data.3. Wrangle, prepare, cleanse the data.4. Analyze, identify patterns, and explore the data.5. Model, predict and solve the problem.6. Visualize, report, and present the problem solving steps and final solution.7. Supply or submit the results.  Workflow GoalsThe data science solutions workflow solves for seven major goals.**Classifying.** We may want to classify or categorize our samples. We may also want to understand the implications or correlation of different classes with our solution goal.**Correlating.** One can approach the problem based on available features within the training dataset. Which features within the dataset contribute significantly to our solution goal? Statistically speaking is there a correlation among a feature and solution goal? As the feature values change does the solution state change as well, and visa-versa? This can be tested both for numerical and categorical features in the given dataset. We may also want to determine correlation among features other than survival for subsequent goals and workflow stages. Correlating certain features may help in creating, completing, or correcting features.**Converting.** For modeling stage, one needs to prepare the data. Depending on the choice of model algorithm one may require all features to be converted to numerical equivalent values. So for instance converting text categorical values to numeric values.**Completing.** Data preparation may also require us to estimate any missing values within a feature. Model algorithms may work best when there are no missing values.**Correcting.** We may also analyze the given training dataset for errors or possibly innacurate values within features and try to corrent these values or exclude the samples containing the errors. One way to do this is to detect any outliers among our samples or features. We may also completely discard a feature if it is not contribting to the analysis or may significantly skew the results.**Creating.** Can we create new features based on an existing feature or a set of features, such that the new feature follows the correlation, conversion, completeness goals.**Charting.** How to select the right visualization plots and charts depending on nature of the data and the solution goals.  1. Question or Problem DefinitionThe question we want answered.> A bit of information provided about the competitionIn this example we'll use the Titanic dataset as done in the guide.  2. Acquire training and testing data.<jupyter_code># Imports
# Gonna try to keep this to a minimal as could get hectic
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns<jupyter_output><empty_output><jupyter_text>2.1 Data Collection<jupyter_code># Downloading dataset
# Should be the same as used in the guide but unsplit
!wget https://gist.githubusercontent.com/fyyying/4aa5b471860321d7b47fd881898162b7/raw/6907bb3a38bfbb6fccf3a8b1edfb90e39714d14f/titanic_dataset.csv<jupyter_output>--2022-02-04 17:42:57--  https://gist.githubusercontent.com/fyyying/4aa5b471860321d7b47fd881898162b7/raw/6907bb3a38bfbb6fccf3a8b1edfb90e39714d14f/titanic_dataset.csv
Resolving gist.githubusercontent.com (gist.githubusercontent.com)... 172.16.58.3, 192.168.3.11, 172.16.31.10, ...
Connecting to gist.githubusercontent.com (gist.githubusercontent.com)|172.16.58.3|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 62002 (61K) [text/plain]
Saving to: ‘titanic_dataset.csv.2’

titanic_dataset.csv 100%[===================>]  60.55K  --.-KB/s    in 0.06s   

2022-02-04 17:42:57 (969 KB/s) - ‘titanic_dataset.csv.2’ saved [62002/62002]<jupyter_text>2.2 Data Inspection<jupyter_code># Load dataset
df = pd.read_csv("titanic_dataset.csv")
df.head()<jupyter_output><empty_output><jupyter_text>2.3 Data DescriptionWhat features are available?<jupyter_code>print(df.columns.values)<jupyter_output>['PassengerId' 'Survived' 'Pclass' 'Name' 'Sex' 'Age' 'SibSp' 'Parch'
 'Ticket' 'Fare' 'Cabin' 'Embarked']<jupyter_text>What features are categorical/numeric? What features contain incomplete data? Eg Age/Cabin<jupyter_code>df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 891 entries, 0 to 890
Data columns (total 12 columns):
 #   Column       Non-Null Count  Dtype  
---  ------       --------------  -----  
 0   PassengerId  891 non-null    int64  
 1   Survived     891 non-null    int64  
 2   Pclass       891 non-null    int64  
 3   Name         891 non-null    object 
 4   Sex          891 non-null    object 
 5   Age          714 non-null    float64
 6   SibSp        891 non-null    int64  
 7   Parch        891 non-null    int64  
 8   Ticket       891 non-null    object 
 9   Fare         891 non-null    float64
 10  Cabin        204 non-null    object 
 11  Embarked     889 non-null    object 
dtypes: float64(2), int64(5), object(5)
memory usage: 83.7+ KB<jupyter_text>What features are a mixed type? Eg ticket/cabin What features may contain errors? Eg Name has a number of different formats We use describe to get an initial idea of the distribution of the features<jupyter_code>df.describe(percentiles=[0.75, 0.8])
# Review survived rate using `percentiles=[.61, .62]` knowing our problem description mentions 38% survival rate.
# Review Parch distribution using `percentiles=[.75, .8]`
# SibSp distribution `[.68, .69]`
# Age and Fare `[.1, .2, .3, .4, .5, .6, .7, .8, .9, .99]<jupyter_output><empty_output><jupyter_text>Describe categorical data<jupyter_code>df.describe(include=["O"])<jupyter_output><empty_output><jupyter_text>2.4 Assumptions based on data analysisRefer back to the workflow goals.Example:**Correlating:** Seeing how the features correlate to survival.**Completing:** Age is incomplete.**Correcting:** Ticket/Cabin could be dropped. Name is non standard.**Creating:** Extract title as a feature from name. Change age into bands.**Classifying:** Add to initial assumptions. Eg children and women more likely to survive.  3. Wrangle, prepare, cleanse the data.  3.1 Correcting by Dropping Features<jupyter_code>print(f"Size before: {df.shape}")
df = df.drop(["Ticket", "Cabin", "PassengerId"], axis=1)
print(f"Size after: {df.shape}")<jupyter_output>Size before: (891, 12)
Size after: (891, 10)<jupyter_text>3.2 Creating New Feature<jupyter_code>df["Title"] = df.Name.str.extract("([A-Za-z]+)\.")
pd.crosstab(df["Title"], df["Sex"])<jupyter_output><empty_output><jupyter_text>They then replace some of the titles with a rare tag and correct some titles. Eg Ms.<jupyter_code>df["Title"] = df["Title"].replace(
    [
        "Lady",
        "Countess",
        "Capt",
        "Col",
        "Don",
        "Dr",
        "Major",
        "Rev",
        "Sir",
        "Jonkheer",
        "Dona",
    ],
    "Rare",
)

df["Title"] = df["Title"].replace("Mlle", "Miss")
df["Title"] = df["Title"].replace("Ms", "Miss")
df["Title"] = df["Title"].replace("Mme", "Mrs")
df.Title.unique()
df[["Title", "Survived"]].groupby(["Title"]).mean()<jupyter_output><empty_output><jupyter_text>They then convert this from categorical to ordinal<jupyter_code>title_mapping = {"Mr": 1, "Miss": 2, "Mrs": 3, "Master": 4, "Rare": 5}
df["Title"] = df["Title"].map(title_mapping)
df.Title.unique()<jupyter_output><empty_output><jupyter_text>We can now drop the name column<jupyter_code>df = df.drop("Name", axis=1)
df.head()<jupyter_output><empty_output><jupyter_text>3.2 Converting Categorical Features<jupyter_code>df["Sex"] = df["Sex"].map({"female": 1, "male": 0}).astype(int)<jupyter_output><empty_output><jupyter_text>3.3 Completing Numerical FeaturesThey use a fancy correlation by guessing the age based on the other features.Not too interested in showing all that here.  4. Analysing Data  4.1 Analysing by Pivoting DataUsed to help us confirm our assumptions.<jupyter_code>df[["Pclass", "Survived"]].groupby(["Pclass"]).mean()
df[["Sex", "Survived"]].groupby(["Sex"]).mean()<jupyter_output><empty_output><jupyter_text>4.2 Analyse by Visualising  4.2.1 Correlating Numerical Features<jupyter_code>df.corr()
sns.heatmap(df.corr())
# Creates the right number of plots
g = sns.FacetGrid(df, col="Survived")
# Applies the function to each plot
g.map(plt.hist, "Age")<jupyter_output><empty_output><jupyter_text>4.2.2 Correlating Numerical and Ordinal Features<jupyter_code>grid = sns.FacetGrid(df, col="Survived", row="Pclass")
grid.map(plt.hist, "Age")<jupyter_output><empty_output><jupyter_text>4.2.3 Correlating Categorical Features<jupyter_code># Not really sure what's going on here
grid = sns.FacetGrid(df, row="Embarked")
grid.map(sns.pointplot, "Pclass", "Survived", "Sex")<jupyter_output>/opt/conda/lib/python3.9/site-packages/seaborn/axisgrid.py:670: UserWarning: Using the pointplot function without specifying `order` is likely to produce an incorrect plot.
  warnings.warn(warning)
/opt/conda/lib/python3.9/site-packages/seaborn/axisgrid.py:675: UserWarning: Using the pointplot function without specifying `hue_order` is likely to produce an incorrect plot.
  warnings.warn(warning)<jupyter_text>4.2.4 Correlating Categorical & Numerical Features<jupyter_code>grid = sns.FacetGrid(df, row="Embarked", col="Survived")
grid.map(sns.barplot, "Sex", "Fare", ci=None)<jupyter_output>/opt/conda/lib/python3.9/site-packages/seaborn/axisgrid.py:670: UserWarning: Using the barplot function without specifying `order` is likely to produce an incorrect plot.
  warnings.warn(warning)
{'fraction_non_alphanumeric': 0.11229338842975206, 'fraction_numerical': 0.038946280991735534, 'mean_word_length': 4.853083434099154, 'pattern_counts': {'":': 7, '<': 68, '<?xml version=': 0, '>': 70, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 4, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '316171', 'license': 'MIT', 'n_tokens_mistral': 3298, 'n_tokens_neox': 3011, 'n_words': 985, 'path': 'src/notebooks/eda.ipynb', 'repo_name': 'evanmahony/kaggleTemplate', 'size': '9694'}