<jupyter_start><jupyter_text>This notebook helps prepare train/dev set for fine-tuning BERT-like models Obtaining input dataThe main steps for obtaining labelled content grains reside on EMR jupyter notebook `/yotta/v1b/export-content-level-decision-and-text-for-selected-customer.ipynb`, which will export the labelled and downsampled grains into AWS S3.You can transport the data using `aws s3 sync` to your local laptop and then upload to SageMaker instance (known painpoint: we should have a direct method that can read data from S3 on a SageMaker instance). Encode structured content grain data into a flat text stringOnce labelled grains are obtained, there are multiple ways to encode the grain content into a single text for text classification models.For example, a content grain may contain a listing (with title and content). One could simply concatenate both title and content into a single string, but use marker tokens to mark the begining of title and body "title: Gucci handbag; body: good condition handbag sold cheap". Downsample and make positive & negative samples balancedIn addition, we further downsample the normal grains so we have the same number of positive samples and negative samples in the training and evalutation set. Technicall, we just need a fairly balanced dataset (e.g. negative = 5 * positive). However, for simplicity, we just downsample so negative = postive). When this happens, the classification accuracy equals AUC. Output data layoutOutput train/dev datasets are stored under `$REPO/transformers/glue_data/SIFT/${CUSTOMER}` following glue data SST-2 task convention.<jupyter_code># base path for the input data
data_base = '/home/ec2-user/SageMaker/data/finetuning/'
import pandas as pd
import numpy as np
import os
df = pd.read_parquet(data_base)
samplecnt = df.groupby('customer_id').count()
samplecnt<jupyter_output><empty_output><jupyter_text>Only take customers with enough training data<jupyter_code>qualified = samplecnt[samplecnt['content_id'] > 20000].reset_index()
qualified
cids = qualified.reset_index()['customer_id'].values
id2name = {'5149e065d3ed172298000273': 'vimeo.com',
 '58754bade4b0ce8dc071573e': 'new.scribd.com',
 '589cf354e4b054e606d8845b': 'poshmark.com',
 '58a34952e4b00180500916f4': 'grailed.com',
 '58efbfa6e4b059942106c512': 'firstmet.com',
 '5add315a4f0cafb2e90f04cd': 'sittercity.com',
 '5b217c8b4f0ca0319d9d7edb': 'trover.com',
 '5cbe28fe4f0c60f5b8a10824': 'segundamano.mx'}
def customer_name(cid):
    return id2name[cid].split('.')[-2]
qualified['customer_name'] = qualified['customer_id'].apply(customer_name)
qualified
import json

SNIPPET_TYPES = ['post', 'listing', 'comment', 'message', 'profile', 'review']

def is_empty_snippet(snippet):
    return len(snippet['subject']) == 0 and len(snippet['content']) == 0

def encode_optional_attr(obj, attr):
    if obj[attr]:
        return attr + ": " + obj[attr]
    return ""

def encode_snippet(snippet):
    return ";".join(["type:" + snippet['snippet_type'],
                     encode_optional_attr(snippet, 'subject'),
                     encode_optional_attr(snippet, 'content')])

def encode_snippets(ev):
    snippets = [ev[s] for s in SNIPPET_TYPES]
    return ';'.join([encode_snippet(s) for s in snippets\
                     if (not is_empty_snippet(s))])

MAX_SENT_LENTH = 510
def encode_text_event(ev):
    sections = [encode_optional_attr(ev, 'subject'),
                encode_optional_attr(ev, 'content'),
                encode_snippets(ev)]
    text = ';'.join([s for s in sections if s])
    return text[:MAX_SENT_LENTH]
qualifieddf = df[df['customer_id'].isin(id2name)].copy()
qualifieddf['text'] = qualifieddf['slim_content_event'].apply(encode_text_event)
qualifieddf['customer_name'] = qualifieddf['customer_id'].apply(customer_name)
qualifieddf['text']
from datetime import datetime, timedelta
d = (qualifieddf['grain_time'].max().date() - qualifieddf['grain_time'].min().date() )
d.days
def sample(df):
    positive = df[df['label'] == True]
    negative = df[df['label'] == False]
    npos = len(positive)
    nneg = len(negative)
    print(f'before sampling, total pos: {npos}, total neg: {nneg}')
    ratio = float(npos)/nneg
    print(f'pos/neg ratio before sampling: {ratio}')
    negative = negative.sample(frac=ratio, replace=False)
    nneg = len(negative)
    ratio = float(npos)/nneg
    print(f'pos/neg ratio aftert sampling: {ratio}')
    print(f'total positives: {npos}, negatives: {nneg}')
    return pd.concat([positive, negative])

def slim(df):
    return df[['customer_name', 'customer_id', 'user_id', 'content_id', \
               'text', 'label', 'is_sampled', 'decision_time', 'grain_time']]
    
def train_test_split(df, cname, train_ratio = 0.8, test_delay_days=3):
    """
        train_ratio: portion of the time period reserved for training
    """
    cdf = df[df['customer_name'] == cname].copy()
    print(f'=======>customer name: {cname}')
    earliest_dt = cdf['grain_time'].min().to_pydatetime()
    latest_dt = cdf['grain_time'].max().to_pydatetime()
    delta = (latest_dt - earliest_dt).days
    train_days = round(delta * train_ratio)
    test_start_offset_days = train_days + test_delay_days
    train_end_dt = earliest_dt + timedelta(days=train_days)
    test_start_dt = earliest_dt + timedelta(days=test_start_offset_days)
    print(f'train start: {earliest_dt}, train end: {train_end_dt}.')
    print(f'test start: {test_start_dt}, test end: {latest_dt}')
    traindf = cdf[cdf['grain_time'] <= pd.Timestamp(train_end_dt)]
    testdf = cdf[cdf['grain_time'] >= pd.Timestamp(test_start_dt)]
    train_content_ids = set(traindf['content_id'].values)
    testdf = testdf[~(testdf['content_id'].isin(train_content_ids))]
    test_content_ids = set(testdf['content_id'].values)
    intersect = train_content_ids.intersection(test_content_ids)
    assert len(intersect) == 0
    print('-->sampling train set')
    traindf = sample(traindf)
    print('-->sampling test set')
    testdf = sample(testdf)
    msk = np.random.rand(len(testdf)) < 0.5
    # first split the test set into dev and test
    return slim(traindf), slim(testdf[msk]), slim(testdf[~msk])


ratio_per_customer = {
    'scribd': 0.6
}

def create_sift_dataset(df, tgt_base, ratio_per_customer):
    names = set(df['customer_name'].values)
    for name in names:
        train_ratio = ratio_per_customer.get(name, 0.8)
        if name == 'scribd':
            continue
        traindf, devdf, testdf = train_test_split(df, name, train_ratio=train_ratio)
        tgt_folder = f"{tgt_base}/{name}/"
        os.makedirs(tgt_folder, exist_ok=True)
        traindf.to_parquet(f'{tgt_folder}/train.parquet')
        devdf.to_parquet(f'{tgt_folder}/dev.parquet')
        testdf.to_parquet(f'{tgt_folder}/test.parquet')
        
    
create_sift_dataset(qualifieddf, '/home/ec2-user/SageMaker/sift/transformers/glue_data/SIFT', ratio_per_customer)<jupyter_output>=======>customer name: sittercity
train start: 2019-06-01 00:07:50.164000, train end: 2019-08-20 00:07:50.164000.
test start: 2019-08-23 00:07:50.164000, test end: 2019-09-09 23:59:36.145000
-->sampling train set
before sampling, total pos: 7829, total neg: 29669
pos/neg ratio before sampling: 0.26387812194546495
pos/neg ratio aftert sampling: 1.0
total positives: 7829, negatives: 7829
-->sampling test set
before sampling, total pos: 2636, total neg: 7806
pos/neg ratio before sampling: 0.3376889572124007
pos/neg ratio aftert sampling: 1.0
total positives: 2636, negatives: 2636
=======>customer name: trover
train start: 2019-06-01 00:02:17.048000, train end: 2019-08-20 00:02:17.048000.
test start: 2019-08-23 00:02:17.048000, test end: 2019-09-09 23:57:27.522000
-->sampling train set
before sampling, total pos: 8920, total neg: 17218
pos/neg ratio before sampling: 0.5180624927401557
pos/neg ratio aftert sampling: 1.0
total positives: 8920, negatives: 8920
-->sampling test set
before samp[...]
{'fraction_non_alphanumeric': 0.1240535083291267, 'fraction_numerical': 0.05363452801615346, 'mean_word_length': 4.697340043134436, 'pattern_counts': {'":': 1, '<': 10, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '535597', 'license': 'Apache-2.0', 'n_tokens_mistral': 2926, 'n_tokens_neox': 2618, 'n_words': 681, 'path': 'notebooks/Prepare-SIFT-Data-For-Fine-Tuning.ipynb', 'repo_name': 'SiftScience/transformers', 'size': '7924'}