<jupyter_start><jupyter_text>Data Preprocessing - Recipe1M<jupyter_code>from dependency import parent_dir
from common.save import make_dir, save_pickle, load_pickle, print_time
from common.basics import *

import copy
import pandas as pd
data, newdata = load_pickle(filename='/opt/eda/data/pplm_data/data_0210.pickle')<jupyter_output><empty_output><jupyter_text>Manually categorize the tags in the dataset.The showcase down below is for the cuisine type translation.(Non-North American vs North American style)If you would like to try out other tasks, you may categorize the original tags accordingly.Note: Data has been cleaned by Helena.<jupyter_code>binary_regional_tag = {'north american':['north-american', 'northeastern-united-states','southern-united-states',\
                                         'midwestern','southern','u . s .', 'ontario','canadian'],
                       'non-north american':['south-west-pacific', 'south-american', 'colombian', 'cuban', 'costa-rican', 'guatemalan', 'honduran', 'brazilian', 'ecuadorean',
                                  'peruvian', 'argentine', 'chilean', 'venezuelan','mexican','south american','caribbean','african', 'somalian', 'south-african', 'moroccan', 'ethiopian', 'egyptian',
                                  'nigerian', 'sudanese', 'congolese', 'libyan', 'angolan','moroccan','hawaiian','british-columbian', 
                                  'rosh-hashanah', 'jewish-ashkenazi', 'jewish-sephardi','cajun', 'central-american',
                                  'polish','finnish','greek','czech','italian','german','dutch',
                                  'french','belgian','portuguese','iceland','turkish',
                                  'hungarian','irish', 'welsh', 'scottish','uk and ireland',
                                  'malaysian','indonesian','vietnamese','thai','cambodian','laotian','filipino',
                                  'middle-eastern', 'iraqi', 'pakistani', 'iranian-persian','egyptian','persian'
                                  'chinese', 'chinese-new-year', 'beijing', 'korean','japanese',
                                  'georgian','russian','bangladeshi','indian','nepalese',
                                  'australian', 'polynesian', 'new-zealand', 'australian and new zealander']
               }

regionZ = [t for category, tag in binary_regional_tag.items() for t in tag]
equal = {l: i for i, lst in binary_regional_tag.items() for l in lst}

labeled, unlabeled = [], []
for i , recipe in tqdm.tqdm(enumerate(data)):
    newtags = []
    if 'tags' in recipe:
        # find the related tags
        newtags = [t for t in recipe['tags'] if t.replace(' recipe','') in regionZ]
        # renames the synonms
        newtags = [equal[t] if t in equal else t for t in newtags]
    recipe_ = copy.deepcopy(recipe)
    if newtags:
        recipe_['tags'] = list(set(newtags))
        labeled.append(recipe_)
    else:
        unlabeled.append(recipe_)
data_df = pd.DataFrame(labeled)<jupyter_output><empty_output><jupyter_text>1.Data Formatting For differnet task, we need to format the data accordingly.To train the discriminator, we need full version of recipe (title, ingredients, instructions, seperated by start and end tags)To generated the prompts for GPT2, we only need:- title, ingredients,start-directions tag- title, directions, start-ingredients tag<jupyter_code>def out_lis(ingredient_lis):
    #format the ingredients: ingre1$ingre2$ingre3
    ori = ''
    for i in ingredient_lis:
        ori = ori + '$' + i
    return ori

def give_class(tag_lis, reg_infor='north american'):
    target = 0
    
    if len(tag_lis) == 1:
        if tag_lis[0] == reg_infor:
            target = 1
    else:
        target = 2
    return target

#Here define your format
data_df['text'] = '<start-title>' + data_df['title'] +'<end-title>'+'<start-directions>' + data_df['instructions'] + '<end-directions> <start-ingredients>'
data_df['target'] = data_df['tags'].apply(lambda x: give_class(x))
#Example of the formatted text

data_df['text'].iloc[2021]<jupyter_output><empty_output><jupyter_text>2.Data for Discriminator Training To train a binary discriminator, we only take the recipes with one tag (either North American or Non-North American)<jupyter_code>data_sub = data_df[data_df['target'] != 2][['target','text']]
data_sub['text'].iloc[0]
print("Negative: ", data_sub[data_sub['target']==0].target.count(), "Positive: ", data_sub[data_sub['target']==1].target.count())
#data_sub.to_csv('/opt/eda/data/pplm_data/discriminator_training/north_american_data.tsv', sep='\t', index=False, header=False)<jupyter_output><empty_output><jupyter_text>3.Data for Other Experiments - Back Translation (North American Recipes --> East Asian Recipes --> North American Recipes)<jupyter_code>data_na = data_df[data_df['target'] == 1]
data_na = data_na[['text']]

#Sample 50 prompts for back translation
#data_na.iloc[:50].to_csv('/opt/eda/data/pplm_data/back_translation/exp_controllable_ingre/na_selected50.txt', sep='\t', index=False, header=False)
#Back translation groundtruth

data_na.iloc[:50].to_csv('/opt/eda/data/pplm_data/back_translation/na_selected50_groundtruth.txt', sep='\t', index=False)<jupyter_output><empty_output><jupyter_text>- Source Data for North American Prevalence Ingredients<jupyter_code>#Need full version of north-american recipes

na_data = data_df[data_df['target']== 1]
#na_data.to_csv('/opt/eda/data/pplm_data/ingredients_eval/source_data_na.txt', sep='\t', index=False)
#Distributed sampling 150 recipes for experiments
sample_df = data_df.groupby('target').apply(lambda x: x.sample(n=50))

#sample_df[['ingredients','instructions','title','text','target']].to_csv('/opt/eda/data/pplm_data/back_translation/exp_controllable_ingre/random150_groundtruth.txt', sep='\t', index=False)
#sample_df['text'].to_csv('/opt/eda/data/pplm_data/back_translation/exp_controllable_ingre/random150.txt', sep='\t', index=False, header=False)

#Randomly sampling 150 North-american for experiments
sample_na = data_df[data_df['target']==1].sample(150)

#sample_na[['ingredients','instructions','title','text','target']].to_csv('/opt/eda/data/pplm_data/back_translation/exp_controllable_ingre/na150_groundtruth.txt', sep='\t', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14478918058870327, 'fraction_numerical': 0.009228321400159108, 'mean_word_length': 4.123064384678076, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 32, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 13, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '273870', 'license': 'Apache-2.0', 'n_tokens_mistral': 2135, 'n_tokens_neox': 2014, 'n_words': 413, 'path': 'notebooks/01-1-Data Pre-processing.ipynb', 'repo_name': 'yijunzhouzoey/PPLM', 'size': '6285'}