<jupyter_start><jupyter_text>Fair Face Undersampling Train  Import Libraries<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns<jupyter_output><empty_output><jupyter_text>Load Data<jupyter_code>train_data1=pd.read_csv("/content/drive/MyDrive/fairfaceforprototype/train_keep_age_final.csv")
train_data=train_data1.copy()
train_data.head()
train_data.shape<jupyter_output><empty_output><jupyter_text>Undersampling data The objective is to balance the data on 'race' and 'age'. Manual undersampling was done by removing samples from the majority classes so that they are exactly equal to the minority class.<jupyter_code>plt.figure(figsize=(10,10))
chart=sns.countplot(x ='age', data = train_data)
chart.set_xticklabels(chart.get_xticklabels(), rotation=45);
train_data['age'].value_counts()
plt.figure(figsize=(10,10))
sns.countplot(x ='gender', data = train_data)
train_data['gender'].value_counts()
plt.figure(figsize=(10,10))
chart=sns.countplot(x ='race', data = train_data)
chart.set_xticklabels(chart.get_xticklabels(), rotation=45);
train_data['race'].value_counts()<jupyter_output><empty_output><jupyter_text>Taking an equal number of samples for ethnicity<jupyter_code>equal_ethnicity =train_data.groupby('race').head(7510)
plt.figure(figsize=(10,10))
chart=sns.countplot(x ='race', data = equal_ethnicity)
chart.set_xticklabels(chart.get_xticklabels(), rotation=45);
plt.figure(figsize=(10,10))
chart=sns.countplot(x ='age', data = equal_ethnicity)
chart.set_xticklabels(chart.get_xticklabels(), rotation=45);
equal_ethnicity['age'].value_counts()<jupyter_output><empty_output><jupyter_text>From equal ethnicities, take equal age groups<jupyter_code>equal_age_groups =equal_ethnicity.groupby('age').head(2212)
plt.figure(figsize=(10,10))
chart=sns.countplot(x ='age', data = equal_age_groups)
chart.set_xticklabels(chart.get_xticklabels(), rotation=45);
plt.figure(figsize=(10,10))
chart=sns.countplot(x ='race', data = equal_age_groups)
chart.set_xticklabels(chart.get_xticklabels(), rotation=45);
equal_age_groups['race'].value_counts()<jupyter_output><empty_output><jupyter_text>From equal age groups take equal ethnicities<jupyter_code>equal_ethnic_2 =equal_age_groups.groupby('race').head(1229)
plt.figure(figsize=(10,10))
chart=sns.countplot(x ='race', data = equal_ethnic_2)
chart.set_xticklabels(chart.get_xticklabels(), rotation=45);
plt.figure(figsize=(10,10))
chart=sns.countplot(x ='age', data = equal_ethnic_2)
chart.set_xticklabels(chart.get_xticklabels(), rotation=45);
equal_ethnic_2['age'].value_counts()<jupyter_output><empty_output><jupyter_text>From second round of equal ethnicities, take equal ages<jupyter_code>equal_age_2 =equal_ethnic_2.groupby('age').head(953)
plt.figure(figsize=(10,10))
chart=sns.countplot(x ='age', data = equal_age_2)
chart.set_xticklabels(chart.get_xticklabels(), rotation=45);
plt.figure(figsize=(10,10))
chart=sns.countplot(x ='race', data = equal_age_2)
chart.set_xticklabels(chart.get_xticklabels(), rotation=45);
equal_age_2['race'].value_counts()<jupyter_output><empty_output><jupyter_text>From second round of equal ages take equal ethnicities<jupyter_code>equal_ethnic_3 =equal_age_2.groupby('race').head(599)
plt.figure(figsize=(10,10))
chart=sns.countplot(x ='race', data = equal_ethnic_3)
chart.set_xticklabels(chart.get_xticklabels(), rotation=45);
plt.figure(figsize=(10,10))
chart=sns.countplot(x ='age', data = equal_ethnic_3)
chart.set_xticklabels(chart.get_xticklabels(), rotation=45);
equal_ethnic_3['age'].value_counts()
equal_ethnic_3.shape<jupyter_output><empty_output><jupyter_text>from equal ethnicites take equal ages<jupyter_code>equal_age_3 =equal_ethnic_3.groupby('age').head(664)
plt.figure(figsize=(10,10))
chart=sns.countplot(x ='age', data = equal_age_3)
chart.set_xticklabels(chart.get_xticklabels(), rotation=45);
plt.figure(figsize=(10,10))
chart=sns.countplot(x ='race', data = equal_age_3)
chart.set_xticklabels(chart.get_xticklabels(), rotation=45);
equal_age_3['race'].value_counts()
equal_age_3.shape
equal_ethnic_3.head()<jupyter_output><empty_output><jupyter_text>We take equal ethnicities as our final resampled selection. This way we ensure equal representation while minimizing the loss of data.  Generate CSV<jupyter_code>equal_ethnic_3.to_csv('/content/drive/MyDrive/fairface_resampling/resampled_train.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1390216154721274, 'fraction_numerical': 0.03048919226393629, 'mean_word_length': 11.177285318559557, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '396336', 'license': 'Unlicense', 'n_tokens_mistral': 1711, 'n_tokens_neox': 1615, 'n_words': 240, 'path': 'sourcecode/data_pre_processing/fairface_ethnicity_distribution_train.ipynb', 'repo_name': 'VinitaSilaparasetty/msc-data-science-dissertation', 'size': '4395'}