<jupyter_start><jupyter_text>Cross_check_datapoints.ipynb: Train an ensemble of lower accuracy models, then use them to find inaccurate labels in the ewt conll dataset.   IntroductionThis document is currently a Work in Progress. The goal is to do something similar to conll_3.ipynb, where an ensemble of models are used together to find possible incorrect labels in the orignal corpus. There are three main phases:  1. PreprocessingThe preprocessing phase uses many of the integrations for Text-extensions for Pandas to prep the dataset for training and inferencing on ahead of susspicious enry flagging Broadly, what we do is: 1. Import all the folds of the dataset we're using (Universal dependencies EWT) 1. Create a Pandas Categorical datatype on over which to classify1. Retokenize that dataset using Huggingface Transformers to Bert-compatible tokens1. Correlate the new tokens with their original counterpart's parts of speech1. Create the Bert embeddings for each sub-token1. Convert the parts of speech tags to our categoical datatype 2. Model Training    During the model training phase, we train a number of models ontop of the bert embeddings ontop of the Bert embeddings we calculated in phase one These fall into two categories: - a large model accross the whole Bert embedding space with relatively high accuracy- a large number of lower accuracy models trained on gaussian random projections of the Bert embedding space 3. Cross-correlationIn this phase, we run inference on the corpus and, using the models in conjunction look for places where there is high agreement between models that disagrees with the corpus ground truth label, flag suspicious entries and go through some descriptive stats.<jupyter_code>import os
import sys
import numpy as np
import pandas as pd
import transformers
import sklearn

import sklearn.pipeline
import multiprocessing
import pickle

import matplotlib.pyplot as plt
import ipywidgets

# And of course we need the text_extensions_for_pandas library itself.
try:
    import text_extensions_for_pandas as tp
except ModuleNotFoundError as e:
    # If we're running from within the project source tree and the parent Python
    # environment doesn't have the text_extensions_for_pandas package, use the
    # version in the local source tree.
    if "../.." not in sys.path:
        sys.path.insert(0, "../..")
    import text_extensions_for_pandas as tp
import text_extensions_for_pandas.cleaning as cleaning
# init some globals we need later on 
BASE_DIR = '../../notebooks/CoNLL_u_test_inputs/'
FEATHER_FILE = "conllu_database.feather"
ewt_base_url = "https://raw.githubusercontent.com/UniversalDependencies/UD_English-EWT/master/en_ewt-ud-"
corpus_df = None
load_cached_models = False
# We're going to need the whole ewt dataset for this: download them, and parse them in 
fold_paths = {"test":  tp.io.conll.maybe_download_dataset_data(BASE_DIR, ewt_base_url + "test.conllu"),
              "dev":   tp.io.conll.maybe_download_dataset_data(BASE_DIR, ewt_base_url + "dev.conllu"),
              "train": tp.io.conll.maybe_download_dataset_data(BASE_DIR, ewt_base_url + "train.conllu")}
fold_docs = {}
for fold,fold_path in fold_paths.items(): 
    fold_docs[fold] = tp.io.conll.conll_u_to_dataframes(fold_path)
    print(f"converted fold: '{fold}' to list of {len(fold_docs[fold])} dataframes")
    #     uncomment to display segments of the extracted folds 
fold_docs['dev'][0].head()<jupyter_output>converted fold: 'test' to list of 316 dataframes
converted fold: 'dev' to list of 318 dataframes
converted fold: 'train' to list of 540 dataframes<jupyter_text>Initialize elements for preprocessing stepsInstantiate pretrained tokenizer and BERT models from transformers library, and create a pandas categorical datatype for parts of speech<jupyter_code>bert_model_name = "dslim/bert-base-NER"
tokenizer = transformers.BertTokenizerFast.from_pretrained(bert_model_name)
bert = transformers.BertModel.from_pretrained(bert_model_name)<jupyter_output><empty_output><jupyter_text>Preprocess the documentFirst we walk though steps 3-6 one by one. Then, because steps 3-6 can only be done on a document-by-document basis, we use a method to do them in a batch adn run them  on the whole corpus. Note this process is computationally intensive so it may take a few minutes to run.<jupyter_code>demo_doc= fold_docs['dev'][0]
# create BERT compatible tokens using our tokenizer
bert_toks = tp.io.bert.make_bert_tokens(demo_doc.loc[0,'span'].target_text, tokenizer)

bert_toks
# re-correlate our original spans with their bert-compatible equivalents
spans = tp.TokenSpanArray.align_to_tokens(bert_toks["span"],demo_doc["span"])

pd.DataFrame({'span' :spans, 'ent_type': demo_doc['upostag']})
    
# carry over some features from the old spans to the new tokenization
spans_df = spans.as_frame()[['begin_token', 'end_token']]
spans_df['upostag'] = demo_doc['upostag']
for i,b_tok,e_tok,pos in spans_df.itertuples():
    bert_toks.loc[b_tok:e_tok-1 , ["postag","raw_span",'raw_span_id']] = pos,spans[i],i
bert_toks
# now translate into a categorical datatype in pandas 
# first, create the dtype as well as translators to and from it 
upostags_list = list(fold_docs['train'][0].append(fold_docs['train'][1:])['upostag'].unique())
upostag_dtype = pd.CategoricalDtype(categories = upostags_list)
upostag_dict = {upostags_list[i]:i for i in range(len(upostags_list)) }
# then convert our document
bert_toks['upostag'].fillna('X',inplace=True) # in our Labels, 'X' is a standin for "N/A" so convert N/A's to 'X'
bert_toks["postag_id"] = bert_toks['upostag'].apply(lambda t: int(upostag_dict[str(t)]))
bert_toks = bert_toks.astype({'upostag_id':'int','upostag':upostag_dtype})
bert_toks = tp.io.bert.add_embeddings(bert_toks,bert)
bert_toks<jupyter_output><empty_output><jupyter_text>Preprocess the whole corpus as abovethe `cleaning` module has methods to do the above process to a whole corpusat a time. Use that for readability<jupyter_code>corpus_df, upostag_dtype, upostags_list, upostag_dict = tp.cleaning.preprocess_documents(fold_docs,'upostag',False,)
corpus_df.head()
# re-read feather document if need be: 
if corpus_df is None or corpus_df.size == 0:
    corpus_df = pd.read_feather("outputs/conll_u_corpus.feather")
    corpus_df
# now get ready to train our model: 
train_df = corpus_df[corpus_df["fold"] == "train"]
train_df<jupyter_output><empty_output><jupyter_text>Train an ensemble of modelsuse embeddings to quickly train multiple models at different sophistication levels<jupyter_code>import sklearn.linear_model
import sklearn.metrics
import sklearn.pipeline

LBFGS_ITERATIONS = 10000

load_from_file = True # we've trained a model already so just load it from there 
pickle_model_file = "conllu_pos_classifier.pickle"


if load_from_file: 
    with open(BASE_DIR+ pickle_model_file, 'rb') as file: 
        base_model = pickle.load(file)
    print("loaded")
else: 
    # train a simple multinomial logisticmodel on the training set: 
    _MULTI_CLASS = "multinomial"
    base_pipeline = sklearn.pipeline.Pipeline([
        # Standard scaler. This only makes a difference for certain classes
        # of embeddings.
        #("scaler", sklearn.preprocessing.StandardScaler()),
        ("mlogreg", sklearn.linear_model.LogisticRegression(
            multi_class=_MULTI_CLASS,
            verbose=10,
            max_iter=10000
        ))
    ])

    base_model = base_pipeline.fit(X_train, Y_train)

X_train = train_df["embedding"].values
Y_train = train_df["upostag_id"]    
base_model
inferred = cleaning.infer_and_extract_raw_entites(corpus_df[corpus_df["fold"] == "test"],upostags_list,base_model,keep_cols=['upostag','upostag_id'])
inferred.head()
# show descriptive metrics 
cleaning.analysis.create_f1_score_report(inferred,'upostag','predicted_class')<jupyter_output><empty_output><jupyter_text>Make a large number of less accurate models, to cross-correlateThe `cleaning.ensemble` module contains a function `train_reduced_model()` that produces detuned versions of our multilogreg model. This function works by setting upa two-stage pipeline. The first stage is a `sklearn.random_projection.GaussianRandomProjection`model that reduces the number of dimensions of the input embeddings, and the secondstage is a multinomial logistic regression model trained with `sklearn.linear_model.LogisticRegression`.<jupyter_code>reduced_model = cleaning.ensemble.train_reduced_model(X_train, Y_train, 16, None, max_iter=LBFGS_ITERATIONS)
reduced_model
def calc_descriptive_stats(df:pd.DataFrame,Y_actual_col,Y_pred_col):
    ret = {
        'precision': sklearn.metrics.precision_score(df[Y_actual_col],df[Y_pred_col],average='weighted'),
        'recall'   : sklearn.metrics.recall_score(df[Y_actual_col],df[Y_pred_col],average='weighted'),
        'f1-score' : sklearn.metrics.f1_score(df[Y_actual_col],df[Y_pred_col],average='weighted')
    }
    return ret
print('Reduced: ')
reduced_out = cleaning.infer_and_extract_raw_entites(corpus_df[corpus_df["fold"] == "test"],upostags_list,reduced_model,keep_cols=['upostag','upostag_id'])
display(calc_descriptive_stats(reduced_out,'upostag','predicted_class'))
print('Original: ')
display(calc_descriptive_stats(inferred,'upostag','predicted_class'))<jupyter_output>Reduced:<jupyter_text>Train the ensemble for realthe `cleaning` module contains a method to train multiple reduced accuracy models concurrently using `ray`. We use this to quickly train models of varying reduced accuracy.<jupyter_code># now train a bunch of models using this
N_COMPONENTS = [32,64,128,192,256,384,512]  # Values for the n_components parameter TODO: this might need to be higher? 
SEEDS = [1,2,3]              # values for the random seed (these are arbitrary)

params = [{"n_components": c, "seed": s} 
          for c in N_COMPONENTS
          for s in SEEDS]
def params_to_name(p):
    return f"{p['n_components']}_{p['seed']}"


models = cleaning.train_model_ensemble(train_df,'upostag_id',model_sizes= N_COMPONENTS,model_seeds=SEEDS )
# Also include the model with no Gaussian random projections.
models["768_1"] = base_model
params.append({"n_components": 768, "seed": 1})
# save the models for future reference. (training takes a long time.)
cached_models_dir = 'cached_models/'
fname = cached_models_dir + 'cross_check_models_reduced.pickle'
# if needed load from file 
if False:
    with open(fname, 'rb') as file:
        models = pickle.load(file)
    load_cached_models = False
# else: 
# #     with open(fname,'wb') as file:
# #         pickle.dump(models, file)
# # Calculate run inference with all of our models
test_df = corpus_df[corpus_df["fold"] == "test"]
test_res_by_model = {name: cleaning.infer_and_extract_raw_entites(
                        test_df,
                        upostags_list,
                        models[name],
                        keep_cols=['upostag','upostag_id'])
                     for name in models.keys() }

test_res_by_model['256_1'].head()
# now get precision and recall for each
precisions = []
for p in params:
    name = params_to_name(p)
    row = {'name': name}
    row.update(p)
    row.update(calc_descriptive_stats(test_res_by_model[name],
                                      'upostag','predicted_class'))
    precisions.append(row)
f1_scores = pd.DataFrame(precisions).sort_values('n_components')
f1_scores 
# plot the tradeoff between dimensionality and F1 score -(x scale is log base 2)
x = f1_scores['n_components']
y = f1_scores['f1-score']

plt.figure(figsize=(4,6))
plt.scatter(x,y)
plt.xscale('log',base=2)
# plt.yscale('log')
plt.ylabel('F1 score')
plt.xlabel('kernel size')
plt.show()
# this seems like it needs a high number of features to accurately predict.
# it might be interesting to see how the accuracy of models we use can impact 
# the quality of the 'flags' that we raise on the corpus<jupyter_output><empty_output><jupyter_text>Now group and find agreements and disagreements between groups and 'gold' This process is largely analagous to what is done in conll_3.ipynb. Even though this problem is a token classification problem and the other one is an entity recognition one, by treating each token / part_of_speech pair as a seperate entity we are able to do much of the same grouping and comparisonsThis process works as follows: 1. Start with our set of predictions made by each model as a seperate dataframe1. Grab a copy of the 'gold standard' tags from the corpus, and format it similarly to the predictions1. Label each of the 'labels' dataframes with its original source1. Combine the dataframes and use a single Pandas Groupby operation to find common token/Part-of-speech pairings, count the number of models in agreement, and determine if the pair is in the gold standard dataset 1. Split into our 'in_gold' and 'not_in_gold' flagged subsets for human inspection<jupyter_code>res_dict = {name: test_res_by_model[name] for name in test_res_by_model.keys()}
# show that we have our outputs setup nicely 
res_dict[names[1]].head()
# start with results dictionary dataframes... 
a_model = '256_1' # just took one at random
# copy res_dict because we will be modifying it
res_feats = {name: test_res_by_model[name].copy() for name in test_res_by_model.keys()}
res_feats[a_model].head()
# now extract the 'in_gold' features and make them into a similar format as the results
gold_feats = res_feats[list(res_feats.keys())[0]][['fold','doc_num','token_id','upostag','upostag_id']].copy()
gold_feats['model'] = 'GOLD'
gold_feats['in_gold'] =True
gold_feats['gold_pos'] = gold_feats['upostag']
gold_feats.head()
# we will quickly want to modify them to label who created them, and remove the predicted cols
for model_name in res_feats.keys():
    res_feats[model_name]['model'] = model_name
    res_feats[model_name]['in_gold'] = False
    res_feats[model_name]['gold_pos'] = res_feats[model_name]['upostag']
    res_feats[model_name].drop(columns = ['upostag_id','upostag'],inplace=True)
    res_feats[model_name].rename(columns= {'predicted_id':'upostag_id','predicted_class':'upostag'},inplace=True)
res_feats[a_model].head()
# Now combine and collate 
res_feats['gold'] = gold_feats
result_feats_df= pd.concat(list(res_feats.values()))
result_feats_df.head()
# add a count column so that only model-predicted results are tallied
result_feats_df['counts'] = 1
result_feats_df.loc[result_feats_df['in_gold'],'counts']=0

# create groupby aggregation dict 
aggby_firsts = ['fold','doc_num','token_id','gold_pos']
aggby= {'in_gold':'any','counts':'sum','model': lambda x: list(x)}
aggby.update({field: 'first' for field in aggby_firsts})
             
#now finally our groupby
grouped_feats = result_feats_df.groupby(['raw_span','upostag'])\
                    .agg(aggby).sort_values(['counts','fold','doc_num','token_id'],ascending=False)\
                    .reset_index()
grouped_feats.head(10)
# create the not_in_gold data subset, showing features that
# several models agree upon but disagree with the corpus label
not_in_gold = grouped_feats[~ grouped_feats['in_gold']]
not_in_gold.head(10)
# Display an example of a not-in-gold flagged token
# here, all 22 models agree that 'wikipedia.org' is a proper noun
# in this context, wheras the corpus lists it as 'X' type 
def display_span(span:tp.array.span):
    # use the inbuilt display features in SpanArray to display the token
    display(tp.SpanArray(span.target_text, [span.begin],[span.end]))
display_span(not_in_gold.reset_index().iloc[5]['raw_span'])

# now generate in_gold, the set of tokens that are in the corpus
# but have low agreement between the models and the corpus
in_gold = grouped_feats[grouped_feats['in_gold']].sort_values('counts')# ascending
in_gold.head(10)
# display an example of an in-gold flagged token
# here the corpus states that the word 'must' is of the AUX
# part of speech, with 0 models in agreement with it. 
# here we also display the other predictions made by models to provide 
# context
targ_sp = in_gold.reset_index().iloc[8]['raw_span']
display_span(targ_sp)
grouped_feats 
grouped_feats[grouped_feats['raw_span']==targ_sp]
counts_ng = (
    not_in_gold.rename(columns={"counts": "numModels"})
    .groupby("numModels")
    .agg({"fold": "count"})
    .rename(columns={"fold": "count"})
    .sort_values("numModels", ascending=False)
)
counts_ng.plot()
counts_ng.head(10)
counts_ig = (
    in_gold.rename(columns={"counts": "num_models"})
    .groupby("num_models")
    .agg({"fold": "count"})
    .rename(columns={"fold": "no. elts"})
)
counts_ig.plot()
counts_ig.head(10)
# save a copy of our data to a file 
grouped_feats.to_csv(cached_models_dir +'v3_outputs.csv')<jupyter_output><empty_output><jupyter_text>Now repeat this flagging process on the `dev` set but using text extensions' `cleaning` module the `cleaning` module of Text extensions for pandas provides functionality to allow this whole inference-flagging process a lot easier. We demonstrate this below<jupyter_code># select the dev df and run inference 
def_df = corpus_df[corpus_df.fold =='dev']

dev_res_by_model = {name: cleaning.infer_and_extract_raw_entites(
                        def_df,
                        upostags_list,
                        models[name],
                        keep_cols=['upostag','upostag_id'])
                     for name in models.keys() }
dev_res_by_model['256_1'].head()
#calculate `in_gold` and `not_in_gold` subsets
dev_in_gold, dev_not_in_gold = cleaning.flag_suspicious_labels(dev_res_by_model,'upostag','predicted_class',keep_cols = [])
dev_in_gold.head(10)
dev_not_in_gold.head(10)
dev_counts_ng = (
    dev_not_in_gold.rename(columns={"count": "numModels"})
    .groupby("numModels")
    .agg({"fold": "count"})
    .rename(columns={"fold": "count"})
    .sort_values("numModels", ascending=False)
)
dev_counts_ng.plot()
dev_counts_ng.head(10)
dev_counts_ig = (
    dev_in_gold.rename(columns={"count": "numModels"})
    .groupby("numModels")
    .agg({"fold": "count"})
    .rename(columns={"fold": "count"})
    .sort_values("numModels", ascending=False)
)
dev_counts_ig.plot()
dev_counts_ig.head(10)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10056266548984996, 'fraction_numerical': 0.007447043248014122, 'mean_word_length': 5.099932705248991, 'pattern_counts': {'":': 19, '<': 35, '<?xml version=': 0, '>': 35, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '560283', 'license': 'Apache-2.0', 'n_tokens_mistral': 5978, 'n_tokens_neox': 5743, 'n_words': 1681, 'path': 'tutorials/cleaning/Cross_check_datapoints.ipynb', 'repo_name': 'ZachEichen/text-extensions-for-pandas', 'size': '18128'}