<jupyter_start><jupyter_text>Research Links:* https://www.researchgate.net/publication/258716133_Miniature_Near-Infrared_NIR_Spectrometer_Engine_For_Handheld_Applications* https://journals.sagepub.com/doi/abs/10.1255/jnirs.1084* https://journals.sagepub.com/doi/10.1177/0003702820985856<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from tqdm import tqdm
tqdm.pandas()
import re
import warnings
%matplotlib inline

#load Datasets
maindir = "/Users/Abimbola/Documents/MSc Computer Science/Semester 2/Computer science project/Blood-Spectorscopy/Data" # Directory with your files
train = maindir+"/Updated_Train.csv"
test =  maindir+"/Updated_Test.csv"
ss =  maindir+"/Updated_Sample_Submission.csv"
test_trimmed = maindir+"/Updated_Test_trimmed.csv"
# hdl_add_lowstd = maindir + '/Exports_hdl_rows_162_full.csv'
# hdl_add_std = maindir + '/Exports_hdl_rows_483_full.csv'
# hgb_add_lowstd = maindir + '/Export_hgb_rows_965_full.csv'
# hgb_add_std = maindir + '/Exports_hgb_rows_2457_full.csv'
# ldl_add_lowstd = maindir + '/Exports_ldl_rows_135_full.csv'
# ldl_add_std = maindir + '/Exports_ldl_rows_410_full.csv'

#Load spectra absorbance coefficients from research
spectra_hgb = maindir+"/Spectra_hgb.csv"
spectra_gluc = maindir+"/Spectra_gluc.csv"
spectra_sknfatbl = maindir+"/Spectra_skinfatblood.csv"
spectra_chl = maindir+"/Spectra_chl.csv"
# Read in the data
# hdl_add_lowstd = pd.read_csv(hdl_add_lowstd)
# hgb_add_lowstd = pd.read_csv(hgb_add_lowstd)
# ldl_add_lowstd = pd.read_csv(ldl_add_lowstd)
# hdl_add_std = pd.read_csv(hdl_add_std)
# hgb_add_std = pd.read_csv(hgb_add_std)
# ldl_add_std = pd.read_csv(ldl_add_std)<jupyter_output><empty_output><jupyter_text>The goal is to use the absorbances which is spectrum data to predict the hdl_cholesterol, hemoglobin and ldl_cholestorl level. These can either be low, ok or high. The absorbance spectrum has 170 values over the entire frequency range. In the dataset with labels trim, the edges of the absorbance spectrum have been trimmed by Bloods-ai because they believed the edges are noisy. Each measurement actually contains, 60 scans so a lot of the data will actually just be scans of the same measurement. It is important to ensure the model doesn't overfit. The plan is to collect and add more data while the competition is running.  **Exploratory Data Analysis**  Train Data<jupyter_code>train = pd.read_csv(train, index_col=0)
train.head()
train.describe()
train.info()
train.shape
train = train.drop(['id'], axis = 1)
print(f"There are",train["donation_id"].nunique(), "donation ids in the dataset")
train['donation_id'].value_counts()
# Check the unique values of the targets
print(train["hdl_cholesterol_human"].unique())
print(train["hemoglobin(hgb)_human"].unique())
print(train["cholesterol_ldl_human"].unique())<jupyter_output>['low' 'ok' 'high']
['ok' 'low' 'high']
['high' 'ok' 'low']<jupyter_text>This looks uniform, all target variables are classified into `low`, `ok` and `high`. `Reading_ID` is in test but not in train. There are three target columns in the train dataset, `hdl_cholesterol_human`, `hemoglobin(hgb)_human`, and `cholesterol_ldl_human` - Multi target data. The data from each samples are repeated about 60 times at different temperatures and humidity which were said to affect the spectrum data. This might cause overfitting if fed into the model. How do I segment this data?   Test Data<jupyter_code>test = pd.read_csv(test)
test.head()
dropped_index =  train[train["donation_id"] == 6824].index 
train_df = train.drop(dropped_index)
print(f"Train data shape is:",train_df.shape)
print(f"Test data shape is:",test.shape)
pd.value_counts(test.dtypes)
# Check for missing value
print(str(train_df.isnull().sum().sum()) + " missing value in train")
print(str(test.isnull().sum().sum()) + " missing value in test")
# Delete invalid donation sample
drop_inds = train_df[train_df['absorbance0'] < 0].index
train_df = train_df.drop(drop_inds)
train_df['std'].describe()
test['std'].describe()<jupyter_output><empty_output><jupyter_text>There's a difference in the maximum std in both train and test set. BloodsAI stated that observations with lower stds aremuch more accurate than those with high stds. Best is to drop the high stds.<jupyter_code># Remove the data with greater std from the train
train_df = train_df[train_df['std'] <= 0.0086]<jupyter_output><empty_output><jupyter_text>Combine both test and train data<jupyter_code>train_df['tag'] = 'train'
test['tag'] = 'test'
all_data = pd.concat([train_df, test])
all_data.describe(exclude='number')
# Extract absorbance columns as keys
spectra_gluc = pd.read_csv(spectra_gluc)
wavelengths = list(spectra_gluc[1:]['Glucose'])

absorbance_dict = dict()

val = 0

for wave in wavelengths:
    
    absorbance_dict[f'absorbance{val}'] = wave
    
    val += 1
all_absorbances = list(absorbance_dict.keys())
all_absorbances.extend(['temperature', 'humidity'])
donation_data = all_data[['donation_id', 'hdl_cholesterol_human', 'hemoglobin(hgb)_human',
'cholesterol_ldl_human', 'std', 'tag']]

donation_data = donation_data[~donation_data['donation_id'].duplicated()]
all_data = all_data.groupby('donation_id')[all_absorbances].mean().reset_index()
all_data = all_data.merge(donation_data, on = 'donation_id', how = 'left')
all_data.head()
all_data.columns
all_data.shape
# Perform some feature engineering to extract more ppts
all_data['low_abs'] = all_data[list(absorbance_dict.keys())].min(axis = 1)
all_data['high_abs'] = all_data[list(absorbance_dict.keys())].max(axis = 1)
all_data['med_abs'] = all_data[list(absorbance_dict.keys())].median(axis = 1)
all_data['mean_abs'] = all_data[list(absorbance_dict.keys())].mean(axis = 1)
all_data['std_abs'] = all_data[list(absorbance_dict.keys())].std(axis = 1)
all_data['sum_abs'] = all_data[list(absorbance_dict.keys())].sum(axis = 1)
all_data.head()
all_data.groupby(['hdl_cholesterol_human'])['med_abs'].mean()
all_data.groupby(['cholesterol_ldl_human'])['med_abs'].mean()
all_data.groupby(['hemoglobin(hgb)_human'])['med_abs'].mean()<jupyter_output><empty_output><jupyter_text>Target Distribution<jupyter_code>sns.countplot(x = all_data["hdl_cholesterol_human"]);
sns.countplot(x = all_data["hemoglobin(hgb)_human"]);
sns.countplot(x = all_data["cholesterol_ldl_human"]);
mapping = {'low' : '0', 'ok' : '1', 'high' : '2'}

reverse_mapping = {'0' : 'low', '1' : 'ok', '2' : 'high'}

all_data['hdl_cholesterol_human'] = all_data['hdl_cholesterol_human'].replace(mapping)

all_data['hemoglobin(hgb)_human'] = all_data['hemoglobin(hgb)_human'].replace(mapping)

all_data['cholesterol_ldl_human'] = all_data['cholesterol_ldl_human'].replace(mapping)<jupyter_output><empty_output><jupyter_text>Target data is higly imbalance<jupyter_code># Split data for modelling
train = all_data[all_data['tag'] == 'train']
test = all_data[all_data['tag'] == 'test']

train = train.drop(['tag'], axis = 1)
test = test.drop(['tag'], axis = 1)
print(train.shape)
print(test.shape)
train.to_csv("clean_train.csv")
test.to_csv("clean_test.csv")
#Reveal some pattern about the targets
cols = ['absorbance0','absorbance20', 'absorbance30','absorbance60', 'absorbance70', 'absorbance90']
for c in cols:
    sns.histplot(train[train['hdl_cholesterol_human']=='0'][str(cols[0])], label='hdl_cholesterol_human=0',color='#f67280');
    sns.histplot(train[train['hdl_cholesterol_human']=='1'][str(cols[0])], label='hdl_cholesterol_human=1',color='#c06c84');
    sns.histplot(train[train['hdl_cholesterol_human']=='2'][str(cols[0])], label='hdl_cholesterol_human=2',color='#6c5b7b');
    plt.legend(loc='upper right')
    plt.xlabel('variable {}'.format(c))
    plt.ylabel('Frequency')
    plt.title('PDF of {}'.format(c))
    plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13990236382322713, 'fraction_numerical': 0.01670092497430627, 'mean_word_length': 7.335117773019272, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 28, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '319703', 'license': 'MIT', 'n_tokens_mistral': 2954, 'n_tokens_neox': 2753, 'n_words': 578, 'path': 'data_cleaning.ipynb', 'repo_name': 'Abimbola-ai/Blood-Spectorscopy', 'size': '7784'}