<jupyter_start><jupyter_text>Balancing datasets with conditional data generationImbalanced datasets are a common problem in machine learning. There are several different scenarios where an imbalanced dataset can lead to a less than optimal model solution. One scenario is when you're training a multi-class classifier and one or more of the classes have fewer training examples than the others. This can sometimes lead to a model that may look like it's doing well overall,when really the accuracy of the underepresented classes is inferior to that of the classes with good representation.Another scenario is when the training data has imbalanced demographic data. Part of what the Fair AI movement is about is ensuring that AI models do equally well on all demographic slices.One approach to improve representational biases in data is through by conditioning Gretel's synthetic data model to generate more examples of different classes of data.You can use the approach to replace the original data with a balanced synthetic dataset or you can use it to augment the existing dataset, producing just enough synthetic data such that when added back into the original data, the imbalance is resolved.In this notebook, we're going to step you through how to use Gretel synthetics to resolve demographic bias in a dataset. We will be creating a new synthetic dataset that can be used in place of the original one.  Begin by authenticating<jupyter_code>%%capture
!pip install -U gretel-client
# Specify your Gretel API key

from getpass import getpass
import pandas as pd
from gretel_client import configure_session, ClientConfig

pd.set_option('max_colwidth', None)

configure_session(ClientConfig(api_key=getpass(prompt="Enter Gretel API key"), 
                               endpoint="https://api.gretel.cloud"))<jupyter_output>Enter Gretel API key··········<jupyter_text>Load and view the dataset<jupyter_code>a = pd.read_csv('https://gretel-public-website.s3.amazonaws.com/datasets/experiments/healthcare_dataset_a.csv')

a<jupyter_output><empty_output><jupyter_text>Isolate the fields that require balancing* We'll balance "RACE", "ETHNICITY", and "GENDER"<jupyter_code>a["RACE"].value_counts()
a["ETHNICITY"].value_counts()
a["GENDER"].value_counts()<jupyter_output><empty_output><jupyter_text>Create a seed file* Create a csv with one column for each balance field and one record for each combination of the balance field values.  * Replicate the seeds to reach the desired synthetic data size.<jupyter_code>import itertools

# Choose your balance columns
balance_columns = ["GENDER","ETHNICITY","RACE"]

# How many total synthetic records do you want
gen_lines = len(a)

# Get the list of values for each seed field and the
# overall percent we'll need for each seed value combination
categ_val_lists = []
seed_percent = 1
for field in balance_columns:
    values = set(pd.Series(a[field].dropna()))
    category_cnt = len(values)
    categ_val_lists.append(list(values))
    seed_percent = seed_percent * 1/category_cnt  
seed_gen_cnt = seed_percent * gen_lines

# Get the combo seeds we'll need. This is all combinations of all
# seed field values
seed_fields = []
for combo in itertools.product(*categ_val_lists):
    seed_dict = {}
    i = 0
    for field in balance_columns:
        seed_dict[field] = combo[i]
        i += 1
    seed = {}
    seed["seed"] = seed_dict
    seed["cnt"] = seed_gen_cnt
    seed_fields.append(seed)

# Create a dataframe with the seed values used to condition the synthetic model
gender_all = []
ethnicity_all = []
race_all = []
for seed in seed_fields:
    gender = seed["seed"]["GENDER"]
    ethnicity = seed["seed"]["ETHNICITY"]
    race = seed["seed"]["RACE"]
    cnt = seed["cnt"]
    for i in range(int(cnt)):
        gender_all.append(gender)
        ethnicity_all.append(ethnicity)
        race_all.append(race)
    
df_seed = pd.DataFrame({"GENDER": gender_all, "ETHNICITY": ethnicity_all, "RACE": race_all})

# Save the seed dataframe to a file
seedfile = "/tmp/balance_seeds.csv"
df_seed.to_csv(seedfile, index=False, header=True)<jupyter_output><empty_output><jupyter_text>Create a synthetic config file<jupyter_code># Grab the default Synthetic Config file

from smart_open import open
import yaml

with open("https://raw.githubusercontent.com/gretelai/gretel-blueprints/main/config_templates/gretel/synthetics/default.yml", 'r') as stream:
    config = yaml.safe_load(stream)
# Adjust the desired number of synthetic records to generated

config['models'][0]['synthetics']['generate']['num_records'] = len(a)
# Adjust params for complex dataset

config['models'][0]['synthetics']['params']['data_upsample_limit'] = 10000<jupyter_output><empty_output><jupyter_text>Include a seeding task in the config<jupyter_code>task = {
    'type': 'seed',
    'attrs': {
        'fields': balance_columns
    }
}
config['models'][0]['synthetics']['task'] = task<jupyter_output><empty_output><jupyter_text>Train a synthetic model<jupyter_code>from gretel_client import projects
from gretel_client import create_project
import time
from gretel_client.config import RunnerMode
from gretel_client.helpers import poll

project = projects.create_or_get_unique_project(name='balancing-data-example')
model = project.create_model_obj(model_config=config)

training_path = 'training_data.csv'
a.to_csv(training_path)
model.data_source = training_path
model.submit(upload_data_source=True)
poll(model)<jupyter_output>[32mINFO: [0mStarting poller<jupyter_text>Generate data using the balance seeds<jupyter_code>rh = model.create_record_handler_obj(data_source=seedfile, params={"num_records": len(df_seed)})
rh.submit_cloud()
poll(rh)
synth_df = pd.read_csv(rh.get_artifact_link("data"), compression='gzip')
synth_df.head()<jupyter_output>[32mINFO: [0mStarting poller<jupyter_text>Validate the balanced demographic data<jupyter_code>synth_df["GENDER"].value_counts()
synth_df["ETHNICITY"].value_counts()
synth_df["RACE"].value_counts()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09269756585528509, 'fraction_numerical': 0.003167722574191397, 'mean_word_length': 5.710290827740492, 'pattern_counts': {'":': 4, '<': 34, '<?xml version=': 0, '>': 34, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '105433', 'license': 'Apache-2.0', 'n_tokens_mistral': 1858, 'n_tokens_neox': 1797, 'n_words': 565, 'path': 'docs/notebooks/balance_data_with_conditional_data_generation.ipynb', 'repo_name': 'robertlombardo/gretel-blueprints', 'size': '5998'}