<jupyter_start><jupyter_text>Unit 3 Project    First section : Data cleaning  In this notebook you will be cleaning your dataset and making sure it's fully ready for modeling.The used dataset is [Hepatitis C  dataset](https://www.kaggle.com/fedesoriano/hepatitis-c-dataset)  Data preparation As a first step we need to prepare the data :1. Read the data set as a data frame ( file name is HepatitisCdata.csv) 2. The here are the column of your dataset:    1) X (Patient ID/No.)    2) Category (diagnosis) (values: '0=Blood Donor', '0s=suspect Blood Donor', '1=Hepatitis', '2=Fibrosis', '3=Cirrhosis') ( the target )     3) Age (in years)    4) Sex (f,m)    Attributes 5 to 14 refer to laboratory data:    5) ALB : Albumin Blood Test    6) ALP : Alkaline phosphatase    7) ALT : Alanine Transaminase    8) AST : Aspartate Transaminase    9) BIL : Bilirubin    10) CHE : Acetylcholinesterase    11) CHOL : Cholesterol    12) CREA : Creatinine     13) GGT : Gamma-Glutamyl Transferase    14) PROT : Proteins    3. Remember your model only accepts numbers so make sure you deal properly with the missing values and the data types and justify your solution choices  4. Make sure the dataset shape in the end is : rows 615 and  14 columns  5. Once finished save the cleaned dataset as "clean_HepatitisC.csv" file<jupyter_code>import pandas as pd 
import numpy as np
#To-Do: read the dataset 
data = pd.read_csv("HepatitisCdata.csv")
data
#To-Do  Start investigating the data types and correcting that 
# Provide explanation about each step in Markdown cells 
#Change columns type to numeric ones
#Add a fuction to convert category object to a numeric value by getting the first elemnt of the object
def convert_category(value):
    if value[0] == '0' and value[1] == 's':
        value = 1
    elif value[0] == '0':
        value = 0
    else:
        value = int(value[0]) + 1
    return float(value)
data['Category'] = data['Category'].apply(convert_category)
#Add a function to convert sex object to numeric values, 0 for male and 1 for female
def convert_sex(value):
    if value == 'm':
        value = '0'
    else:
        value = '1'
    return float(value)
data['Sex'] = data['Sex'].apply(convert_sex)
data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 615 entries, 0 to 614
Data columns (total 14 columns):
 #   Column      Non-Null Count  Dtype  
---  ------      --------------  -----  
 0   Unnamed: 0  615 non-null    int64  
 1   Category    615 non-null    float64
 2   Age         615 non-null    int64  
 3   Sex         615 non-null    float64
 4   ALB         614 non-null    float64
 5   ALP         597 non-null    float64
 6   ALT         614 non-null    float64
 7   AST         615 non-null    float64
 8   BIL         615 non-null    float64
 9   CHE         615 non-null    float64
 10  CHOL        605 non-null    float64
 11  CREA        615 non-null    float64
 12  GGT         615 non-null    float64
 13  PROT        614 non-null    float64
dtypes: float64(12), int64(2)
memory usage: 67.4 KB<jupyter_text>> Data types are all numeric Now ! Next we need to deal with missing values for the feature columns<jupyter_code>import seaborn as sns<jupyter_output><empty_output><jupyter_text>To-Do list all the columns that contain missing values along with their counts<jupyter_code># To-Do Start providing solutions for each column that has missing data 
# Treat each case seperately 
# Hint : no data row should be deleted 
# Provide evidence that you filled those missing values after each step 
#Check Nan values for each colum
print(data.isna().sum())<jupyter_output>Unnamed: 0     0
Category       0
Age            0
Sex            0
ALB            1
ALP           18
ALT            1
AST            0
BIL            0
CHE            0
CHOL          10
CREA           0
GGT            0
PROT           1
dtype: int64<jupyter_text>To replace missing values we should find out which is a better measure of the central tendency of data and use that value for replacing missing values appropriately.- For symmetric data distribution, it is good to use the mean value.- For skewed data, it is good to consider using the median value.<jupyter_code># Check distribution of each variable with missing values
# Plot - ALB
sns.boxplot(data['ALB'])
sns.distplot(data['ALB'])<jupyter_output>/home/billel/jupyter/jupyterenv/lib/python3.8/site-packages/seaborn/_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(
/home/billel/jupyter/jupyterenv/lib/python3.8/site-packages/seaborn/distributions.py:2619: FutureWarning: `distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `histplot` (an axes-level function for histograms).
  warnings.warn(msg, FutureWarning)<jupyter_text>We notice that the data is skewed. There are several or large numbers of data points that act as outliers. Outliers data points will have a significant impact on the mean and hence, we should cosinder fill in missing values with the median.<jupyter_code># Plot - ALP
sns.boxplot(data['ALP'])
sns.distplot(data['ALP'])<jupyter_output>/home/billel/jupyter/jupyterenv/lib/python3.8/site-packages/seaborn/_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(
/home/billel/jupyter/jupyterenv/lib/python3.8/site-packages/seaborn/distributions.py:2619: FutureWarning: `distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `histplot` (an axes-level function for histograms).
  warnings.warn(msg, FutureWarning)<jupyter_text>We notice that the data is kind of symetric. There are few data points that act as outliers. Thus, we should cosinder fill in missing values with the mean.<jupyter_code># Plot - ALT
sns.boxplot(data['ALT'])
sns.distplot(data['ALT'])<jupyter_output>/home/billel/jupyter/jupyterenv/lib/python3.8/site-packages/seaborn/_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(
/home/billel/jupyter/jupyterenv/lib/python3.8/site-packages/seaborn/distributions.py:2619: FutureWarning: `distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `histplot` (an axes-level function for histograms).
  warnings.warn(msg, FutureWarning)<jupyter_text>We notice that the data is skewed. There are several or large numbers of data points that act as outliers. Outliers data points will have a significant impact on the mean and hence, we should cosinder fill in missing values with the median.<jupyter_code># Plot - CHOL
sns.boxplot(data['CHOL'])
sns.distplot(data['CHOL'])<jupyter_output>/home/billel/jupyter/jupyterenv/lib/python3.8/site-packages/seaborn/_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(
/home/billel/jupyter/jupyterenv/lib/python3.8/site-packages/seaborn/distributions.py:2619: FutureWarning: `distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `histplot` (an axes-level function for histograms).
  warnings.warn(msg, FutureWarning)<jupyter_text>We notice that the data is kind of symetric. There are few data points that act as outliers. Thus, we should cosinder fill in missing values with the mean.<jupyter_code># Plot - PROT
sns.boxplot(data['PROT'])
sns.distplot(data['PROT'])<jupyter_output>/home/billel/jupyter/jupyterenv/lib/python3.8/site-packages/seaborn/_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(
/home/billel/jupyter/jupyterenv/lib/python3.8/site-packages/seaborn/distributions.py:2619: FutureWarning: `distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `histplot` (an axes-level function for histograms).
  warnings.warn(msg, FutureWarning)<jupyter_text>We notice that the data is skewed. There are several or large numbers of data points that act as outliers. Outliers data points will have a significant impact on the mean and hence, we should cosinder fill in missing values with the median.<jupyter_code>#Replace NaN values with with respective mean or median
data.fillna(data["ALB"].median(), inplace=True)
data.fillna(data["ALP"].mean(), inplace=True)
data.fillna(data["ALT"].median(), inplace=True)
data.fillna(data["CHOL"].mean(), inplace=True)
data.fillna(data["PROT"].median(), inplace=True)
print(data.isna().sum())<jupyter_output>Unnamed: 0    0
Category      0
Age           0
Sex           0
ALB           0
ALP           0
ALT           0
AST           0
BIL           0
CHE           0
CHOL          0
CREA          0
GGT           0
PROT          0
dtype: int64<jupyter_text>> We have no more missing dataAs an optional thing we can also rename the first column as 'index' or "ID" instead of Unnamed<jupyter_code>data.columns
data.rename(columns={'Unnamed: 0': "ID"}, inplace=True)
data<jupyter_output><empty_output><jupyter_text>6. Save the clean dataset :<jupyter_code># Don't forget to drop the index 
data.to_csv('clean_dataset.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08161296546401697, 'fraction_numerical': 0.025274937295002894, 'mean_word_length': 3.762057877813505, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 40, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '461139', 'license': 'MIT', 'n_tokens_mistral': 3274, 'n_tokens_neox': 3046, 'n_words': 1218, 'path': '1. Data cleaning.ipynb', 'repo_name': 'BillelBenoudjit/unit3_Project', 'size': '10366'}