<jupyter_start><jupyter_text>Data> You should begin your work by cleaning up your data and possibly defining tools for doing it repeateadly. ***input***: raw data

***output***: clean and tidy dataset for ML model / simulation / analytics + toy dataset for testing

***description:***

This is the first notebook of your machine learning project. In this notebook, you will load the data, inspect, clean and make it tidy. 
You will define the data points and their features and labels. The output of this notebook is a clean, tidy dataset ready for analysis and machine learning.
You can also do a basic statistical analysis of the data to better understand it.
For any functions you define for handling the data, remember to mark their cells with ` export` -comment,
so that they will be included in the data.py-module build based on this notebook.
You can also include unit tests for your own functions.  Import relevant modules

Import python modules you need for handling the data<jupyter_code>import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from sklearn import datasets<jupyter_output><empty_output><jupyter_text>Define notebook parameters

Define input, output and additional parameters of this notebook, the information needed for running the notebook.
In your own project, you can do this step in the later iterations of the work,
when you know what is required.
In this cell, only assing values to variables directly: `variable_name = value`.
**Do not derive any information in this cell as it will mess up the parameterization** - do it in the cell below.

The cell below has the tag 'parameters' - this is for the notebook parameterization tool 'papermill'
that allows you execute complete notebooks as python functions.
The values you define here will become the default values of the parameterized notebook, but you can also run the notebook with completely different setup.
More on this in the `workflow` notebook.<jupyter_code># This cell is tagged with 'parameters'
output_clean_filepath = "data/preprocessed_data/dataset_clean_switzerland_cleveland.csv"
output_toy_filepath = "data/preprocessed_data/dataset_toy_switzerland_cleveland.csv"
seed = 0<jupyter_output><empty_output><jupyter_text>Now we can define any immediate derivative operations, righ below the parameters cell:<jupyter_code># set seed
np.random.seed(seed)<jupyter_output><empty_output><jupyter_text>Load the data

Now, you should load the raw data to be processed. If the data loading process is complex,
it may be a good idea to define a function for doing it.

For the example we will be using a subset of an anonymized heart disease dataset available at http://archive.ics.uci.edu/ml/datasets/Heart+Disease.

We combine two subsets of the whole dataset, cleveland & switzeland, to get enough datapoints for analysis,
and to include missing data points (switzerland dataset is missing a lot of data).
In addition, before saving the results for later use in the end of this notebook, 
we suffle the dataset to reduce need for costly randomization operations in the future.

Here we are directly downloading an online dataset.
Typically you would probably load a dataset from some database, or disk. 

Please note, that you should not add your datasets to git,
as it is not intended for data version control and tracking large datafiles exceed the limits of it.
The 'data'-folder of this template is ignored by git.
We are still working on best practices for version control of data,
and will update the template once we find a compatible solution.<jupyter_code># Read two online csv files and concatenate them into a single pandas dataframe
df = pd.concat(
    (
        pd.read_csv(
            "http://archive.ics.uci.edu/ml/machine-learning-databases/heart-disease/processed.switzerland.data",
            sep=",",
            names=range(14)
            # na_values="?", # we could convert non-numericals to na already here
        ),
        pd.read_csv(
            "http://archive.ics.uci.edu/ml/machine-learning-databases/heart-disease/processed.cleveland.data",
            names=range(14),
        ),
    )
)<jupyter_output><empty_output><jupyter_text>Describe the data

Define data points, features and labels

Data point in the data set is a patient, the features are physiological measurements and the label is heart disease diagnosis.
The label shows if the patient is diagnosed with a heart disease (values 1-4) or not (value 0).<jupyter_code>df.shape
# view the data
df.head()<jupyter_output><empty_output><jupyter_text>Ok, so our example dataset contains 426 data points and 14 columns.
We see, that the dataset does not contain column names.
This is a bit untypical, normally the names of the variables are stored with the data.
We have to manually extract them from a text file describing the data, and add them to the dataframe.<jupyter_code># load description of the dataset and extract column names

import urllib

name_rows = []  #
url = "http://archive.ics.uci.edu/ml/machine-learning-databases/heart-disease/heart-disease.names"
file = urllib.request.urlopen(url)
line_of_interest = False  # skip unrelevant lines
# this file contains all information of the dataset, in a non-structured form
for line in file:
    decoded_line = line.decode("utf-8").strip("\n")
    if "Complete attribute documentation" in decoded_line:
        line_of_interest = False
    if line_of_interest and decoded_line != "":
        name_rows.append(decoded_line.strip(" "))
    if "Only 14 used" in decoded_line:
        line_of_interest = True

_ = [print(val) for val in name_rows]<jupyter_output>-- 1. #3  (age)
-- 2. #4  (sex)
-- 3. #9  (cp)
-- 4. #10 (trestbps)
-- 5. #12 (chol)
-- 6. #16 (fbs)
-- 7. #19 (restecg)
-- 8. #32 (thalach)
-- 9. #38 (exang)
-- 10. #40 (oldpeak)
-- 11. #41 (slope)
-- 12. #44 (ca)
-- 13. #51 (thal)
-- 14. #58 (num)       (the predicted attribute)<jupyter_text>Ok, so there are 13 features and one label (the last column in our dataset) in the data. Let's construct these into dataframe column names.<jupyter_code># set dataframe column names

# extract variable names
col_names = [
    "".join(row)
    for row in [
        [
            val.replace(" ", "").split(".")[0].split("-")[-1],
            val.replace(" ", "").split(")")[0].split("(")[-1],
        ]
        for val in name_rows
    ]
]
# add x and running number to feature names
col_names[:-1] = ["x" + val for val in col_names[:-1]]
# add y to label name
col_names[-1] = "y1" + col_names[-1][-3:]

df.columns = col_names
df.head()<jupyter_output><empty_output><jupyter_text>Ok, so now the data is in a table, and columns have meaningful names.
However, there appear to be a lot of missing values in the data, and we still don't know how clean it is.  Clean the data and make it tidy

For analytics & ML purposes, we should make the data tidy. This means that

1. Every column is a variable (feature or label)

2. Every row is an observation (data point).

3. Every cell is a single value (int, float, cathegorical, str, but no nested structures like lists or dictionaries)

To make a dataset tidy, you must really think what is your data point, and modify, explode or remove all nested structures with respect to the datapoints. 

For example with image data you want to flatten the images from matrices or tensors to vectors.

In our example, our data point is a patient. We see that the data is almost in tidy format, but still contains some missing values.

In addition, we want to make sure that each column only has data of a single type - no datatypes mixed.
Easiest way to check for data type consistency is to check what data types are present in each column:<jupyter_code># check data type concistency
df.dtypes<jupyter_output><empty_output><jupyter_text>Here, all columns that contain 'object' variables, contain elements that can not be described in sigle datatype.
These may be nested structures or mixed datatypes. We saw earlier that there were a lot of question marks in the dataset.
Let's try to replace these with np.nan, meaning a missing value, and then converting the columns with 'object' datatype to numericals:<jupyter_code># replace '?' with np nan
df.replace("?", np.nan, inplace=True)
# convert columns with 'object' type to float
df = df.astype(
    {
        "x4trestbps": float,
        "x6fbs": float,
        "x7restecg": float,
        "x8thalach": float,
        "x9exang": float,
        "x10oldpeak": float,
        "x11slope": float,
        "x12ca": float,
        "x13thal": float,
    }
)
df.dtypes<jupyter_output><empty_output><jupyter_text>Ok, now there are no 'object' datatypes - if there were, we would know that there are still some malicius values in the cells that would have to be cleaned off. For the sake of simplicity, we want to omit all categorical features in this example.<jupyter_code>df.nunique()<jupyter_output><empty_output><jupyter_text>We see that the features "x2sex", "x3cp", "x5chol", "x7restecg" and "x9exang" only contain limited number of values,
 so we assume them categorical. These, we can drop from the data.<jupyter_code># drop columns by name
df = df.drop(
    ["x2sex", "x3cp", "x5chol", "x7restecg", "x9exang"],
    axis=1,  # drop columns with categorical variables
)<jupyter_output><empty_output><jupyter_text>Let us see how much of the data is missing in each variable left<jupyter_code># count nan values per column
df.isna().sum()<jupyter_output><empty_output><jupyter_text>Ok,  there are many missing variables.

Now we have two alternatives: we can either get rid of the missing values, or use such robust methods that allow missing values in the data.

For removing the missing values we can either drop the columns with most missing data or try to impute the missing values from the data.

If there was very little data, we would most likely want to impute the missing data,
for example with use the Scikit-learn imputer: https://scikit-learn.org/stable/modules/generated/sklearn.impute.SimpleImputer.
If you do imputing of missing data, please consider the effects on data quality.

In this example, we just drop the columns with most na data<jupyter_code>df = (
    df.drop(
        ["x6fbs", "x11slope", "x12ca", "x13thal"],  # drop columns with most na values
        axis=1,
    )
    .dropna()
    .reset_index(drop=True)
)
df.head()<jupyter_output><empty_output><jupyter_text>Visualize the data

Once the dataset is looking clean, it is time to visualize it.
Visualization is probably the most powerful tool of data science.

Visualization depends highly on the data, but usually you should begin by looking at two things: distribution and correlation.

Let's make histograms of the variables, and a trellis of scatterplots for visualizing correlations.<jupyter_code># export

# ^(this cell is exported to ml_project_template/ml_project_template/data.py)
# you could also define another module to export to.
# however, all modules that you export to, must have a notebook with the same name and header!

# function for drawing histograms of a dataframe
import matplotlib.pyplot as plt
import pandas as pd


def plot_histogram(df):
    """
    Plot histograms of a dataframe
    """
    n_cols = len(df.columns)
    col_names = df.columns.values
    fig, axs = plt.subplots(
        1, n_cols, figsize=(3 * n_cols, 3), constrained_layout=True, sharey="row"
    )
    for i in range(n_cols):
        ax = axs[i]
        col_name = col_names[i]
        x = df[col_name]
        x.plot(ax=ax, kind="hist")
        xmin, xmax = min(x), max(x)
    for i in range(n_cols):  # to have equal tick lines in each plot
        ax = axs[i]
        col_name = col_names[i]
        x = df[col_name]
        xmin, xmax = min(x), max(x)
        ax.hlines(
            y=axs[df.apply(lambda x: x.max()).argmin()].get_yticks()[
                1:-1
            ],  # select ticks from the fig with smallest max value
            xmin=xmin,
            xmax=xmax,
            colors="white",
            alpha=1,
            linewidth=2,
        )
        ax.set_xlabel(col_name)
        ax.set_title(f"{i})", loc="left")
        ax.spines["left"].set_visible(False)
        ax.spines["right"].set_visible(False)
        ax.spines["top"].set_visible(False)
    return fig, ax<jupyter_output><empty_output><jupyter_text>Plot histogram of the data:<jupyter_code>_ = plot_histogram(df)<jupyter_output><empty_output><jupyter_text>From the plot 4) we see the distribution of different heart disease diagnosis.
Remember, that the diagnosis 0 is negative, meaning no heart disease, and the rest are positive, meaning different heart diagnoses.
For sake of simplicity, we will replace all different positive diagnoses with a one positive indicator. Let's replace different heart diagnoses with value 1<jupyter_code>df.y1num.replace([1, 2, 3, 4], 1, inplace=True)<jupyter_output><empty_output><jupyter_text>Let's plot the distributions again:<jupyter_code>_ = plot_histogram(df)<jupyter_output><empty_output><jupyter_text>Let's make a trellis plot of the features. Now we can visualize the diagnosis result with the glyph (color and shape of the marker).

But wait, don't we just want to plot each of the features against the label?
You could do that, too, but in the initial analysis phase it is a good practice to compare all variables against each other.
It can reveal all kinds of interesting correlations that may affect your choises later.

This will make quite a plot to digest, I agree.
For a raport or a presentation you might want to focus on just a few features.<jupyter_code># export

# Make a multiple of scatter plots
def plot_trellis(df, legend_title="y", true_label="True", false_label="False"):
    """
    Make a trellis plot of a dataframe against a binary y value in last column
    """
    n_cols = df.shape[1] - 1
    col_names = df.columns.values

    fig, axs = plt.subplots(
        n_cols, n_cols, figsize=(n_cols * 3, n_cols * 3), constrained_layout=True
    )
    for i in range(n_cols):
        for j in range(n_cols):
            ax = axs[i, j]
            if i != j:
                # first plot negative cases
                df[df.iloc[:, -1] == 0].plot(
                    ax=ax,
                    x=col_names[j],
                    y=col_names[i],
                    kind="scatter",
                    color="b",
                    marker="o",
                    alpha=0.5,
                    label=false_label,
                )
                # then positive cases
                df[df.iloc[:, -1] != 0].plot(
                    ax=ax,
                    x=col_names[j],
                    y=col_names[i],
                    kind="scatter",
                    color="r",
                    marker="x",
                    alpha=0.5,
                    label=true_label,
                )
                # Hide the right and top spines
                ax.spines["right"].set_visible(False)
                ax.spines["top"].set_visible(False)
                ax.set_title(f"{i*n_cols+j})", loc="left")
                ax.legend(title=legend_title)
            else:
                ax.annotate(xy=(0, 0.5), text=col_names[i], fontsize=20)
                ax.axis("off")  # hide the box
    return fig, ax
_ = plot_trellis(
    df, legend_title="diagnose", true_label="positive", false_label="negative"
)<jupyter_output><empty_output><jupyter_text>Some intermediate conclusions based on data visualization

Can you find anything already based on pure visualization? Already based on the visualization we can draw a couple of conclusions:

1. the four features we selected have some correlations, but are all well scattered - thus we can assume that they are somewhat independent and tell different things about the patients.
This is a good reason to include these four features in the ML model.

2. We can already draw conclusions on how individual features effect the diagnosis risk: for example age 50-70 and talach < 150 increase the risk of a positive diagnosis.

3. Visualization is a great way to detect outliers. We see that the feature 'oldpeak' has suspiciously many zero values.
These may be correct, but if this was a serious analysis I would definitely do my research on the feature to determine how the zeros should be interpret and treated.  Suffle Dataset

Before saving the dataset for further use, it is a good practice to suffle it.
This is, of course, assuming the order of the data is not meaningful (as with for example time series data).

Random samplin large datasets consumes time and resources, so if you have a pre-suffled dataset you can just read in more data that is already randomized.
Also, quite often datasets are intentionally or carelessly saved with some obvious or latent order that might include odd biases to your further analysis.
Suffling helps you to get rid of these. Suffling a large dataset may take a while.<jupyter_code># to suffle the dataset sample a fraction of 1 from it
df = df.sample(frac=1).reset_index(drop=True)  # suffle and re-index dataset
df.head()<jupyter_output><empty_output><jupyter_text>Save clean and tidy data for further use<jupyter_code># save the dataset to output path defined in the beginning of this notebook
df.to_csv(output_clean_filepath)<jupyter_output><empty_output><jupyter_text>Create small toy dataset for developing and testing the ML methods

Best way to test your ML methods in development is with data as close to the real data as possible.
However, to save you from frustrating long runtimes in development, it is better to create a small sample dataset.
You can also create unit tests with this sample dataset.

In this example we make a stratified sample that contains randomly picked elements with labels in same proportions as in the data.
This increases the chances immediately noticing how our models perform with the real data and tail values included.
This is only applicable if the label is categorical, and useful if there are many categories.

You can also just make a small sample without considering the proportions `df.sample(frac = 0.1).to_csv(output_toy_filepath)`.

If the order of the data matters, just cut out a small continuous proportion of the data for toy dataset.<jupyter_code># create simple stratified sample of the notebook
n_toy = 30  # number of samples
toy_df = (
    df.groupby("y1num", group_keys=False)
    .apply(lambda x: x.sample(int(np.rint(n_toy * x.shape[0] / df.shape[0]))))
    .sample(frac=1)
    .reset_index(drop=True)
)

# alternatively:
# toy_df = df.sample(frac = n_toy/df.shape[0]) # if simple stratification is not applicable
# toy_df = df.iloc[:n_toy,:] # with time series data. consider the size of n_toy
toy_df.head()
# save toy dataset
toy_df.to_csv(output_toy_filepath)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07881694644284572, 'fraction_numerical': 0.00857980282440714, 'mean_word_length': 3.5339453974389947, 'pattern_counts': {'":': 10, '<': 85, '<?xml version=': 0, '>': 85, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '129391', 'license': 'Apache-2.0', 'n_tokens_mistral': 5453, 'n_tokens_neox': 5302, 'n_words': 2329, 'path': '00_data.ipynb', 'repo_name': 'ssorjone/ml_project_template_new_sami', 'size': '18765'}