<jupyter_start><jupyter_text>Feature Engineering From data analysis it is clear that some features which cannot be zero have zeros values.However they have zero values, like- BMI: 1.4%- BloodPressure: 4.6% - Glucose: 0.7% - SkinThickness: 29.6% These are clearly extreme outliers that might be caused by data entry issues, so these records need to be removed.<jupyter_code># removing records having BMI and BloodPressure as zero
clean_data = data[(data.BMI!=0) & (data.BloodPressure!=0) & (data.Glucose!=0) & (data.SkinThickness!=0)].copy()<jupyter_output><empty_output><jupyter_text>Pregnancies<jupyter_code># this feature is right skewed, so to transform it with sqrt function
trans_Pregnancies = np.sqrt(clean_data.Pregnancies)
clean_data['Pregnancies'] = trans_Pregnancies
trans_Pregnancies.plot('hist')<jupyter_output><empty_output><jupyter_text>Glucose<jupyter_code># normalising the feature
clean_data.Glucose = (clean_data.Glucose - clean_data.Glucose.mean())/clean_data.Glucose.std()
clean_data.Glucose.plot('hist')<jupyter_output><empty_output><jupyter_text>BloodPressure<jupyter_code># normalising the feature
clean_data.BloodPressure = (clean_data.BloodPressure - clean_data.BloodPressure.mean())/clean_data.BloodPressure.std()
clean_data.BloodPressure.plot('hist')<jupyter_output><empty_output><jupyter_text>SkinThickness<jupyter_code># transform right skewed distribution with sqrt transformation
clean_data.SkinThickness = np.sqrt(clean_data.SkinThickness)
clean_data.SkinThickness.plot('hist')<jupyter_output><empty_output><jupyter_text>Insulin<jupyter_code># transform right skewed data with sqrt transformation
clean_data.Insulin = np.sqrt(clean_data.Insulin)
clean_data.Insulin.plot('hist')<jupyter_output><empty_output><jupyter_text>BMI<jupyter_code># normalize data
clean_data.BMI = (clean_data.BMI - clean_data.BMI.mean())/clean_data.BMI.std()
clean_data.BMI.plot('hist')<jupyter_output><empty_output><jupyter_text>DiabetesPedigreeFunction<jupyter_code># transform right skewed data with sqrt function
clean_data.DiabetesPedigreeFunction = np.sqrt(clean_data.DiabetesPedigreeFunction)
clean_data.DiabetesPedigreeFunction.plot('hist')<jupyter_output><empty_output><jupyter_text>Age<jupyter_code># bin ages 
bins = [18, 24, 34, 50, 70, 100]
clean_data.Age = pd.cut(clean_data.Age, bins, right=True, labels=['Youth', 'Millennials', 'Adult', 'SeniorCitizens', 'Old'])
display(clean_data.Age.value_counts().plot('bar'))
encoder = LabelEncoder()
clean_data.Age = encoder.fit_transform(clean_data.Age)
clean_data.Age.value_counts().plot('bar')
# save data
clean_data.to_csv('../data/processed/clean_data.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12185976752905887, 'fraction_numerical': 0.00974878140232471, 'mean_word_length': 9.106060606060606, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '451303', 'license': 'MIT', 'n_tokens_mistral': 1001, 'n_tokens_neox': 909, 'n_words': 160, 'path': 'notebooks/FeatureEngineering.ipynb', 'repo_name': 'satishjasthi/ROC_curves_demystified', 'size': '2667'}