<jupyter_start><jupyter_text>Relation Exraction Dataset based on CONLL data In this small tutorial we are going to look at how we could make the weak annotation of data for relation extraction task using simple patterns. For that, we are going to use dataset in CONLL format that already has gold labels, which we are going to use for comparing with the labels we receive. The examples of such files could be downloaded by the following link: https://ucloud.univie.ac.at/index.php/s/cGQNRgbPHZU89mYThe steps are the following:- process the data in CONLL format (an example could be downloaded by the following link:)- extract all the relevant for us information (the sample and the relation label)- create a sample set of hand-crafted patterns in the format: _"\\$ARG1 \ \\$ARG2"_ where _\\$ARG1_ and _\\$ARG2_ are the entities between that the relation is hold- annotated sample with Spacy package in order to get the entities- take the entities pairwise and extract the part of sentences between them (e.g. from a sentence _"<NAME> born on 13 October 1925"_ a subsentence _"\\$ARG1 was born on \\$ARG2"_ will be extracted given _"<NAME>"_ and _"13 October 1925"_ were the only entities defined by Spacy)- check if there is a pattern in this subsentences. If so, it receives a label corresponding to the relation that pattern support and becomes a part of our new weakly annotated training set.   Imports<jupyter_code>import spacy
import pandas as pd
# import en_core_web_sm
import re
import itertools
from tqdm import tqdm
tqdm.pandas()

pd.set_option('display.max_colwidth', -1)
# in oder to avoid issues with en_core_web_lg, please execute the following command
!python -m spacy download en_core_web_sm
ARG1 = "$ARG1"
ARG2 = "$ARG2"
FINAL_DF_COLUMNS = ['sample', 'extr_sample', 'pattern', 'weak_label', 'gold_label']<jupyter_output><empty_output><jupyter_text>We are going to use this function each time we want to print out a Dataframe in order to escape dollar symbol in patterns<jupyter_code>def escape_dollar(strings):
    return [re.sub("\\$", "\\\\$", str(string)) for string in strings]<jupyter_output><empty_output><jupyter_text>Read and preprocess CONLL data Firstly, we are going to read the input .conll file. Please make sure that it is saved in the same directory as this tutorial notebook and path to it is solved in conll_file variable.<jupyter_code>conll_file = "dev.conll"
def process_data(path_to_data):
    samples, relations = [], []
    with open(path_to_data, encoding='utf-8') as f:
        for line in f:
            line = line.strip()
            if line.startswith("# id="):    # Instance starts
                sample = ""
                label = line.split(" ")[3][5:]
            elif line == "":  # Instance ends
                samples.append(sample)
                relations.append(label)
            elif line.startswith("#"):  # comment
                continue
            else:
                parts = line.split("\t")
                token = parts[1]
                if token == "-LRB-":
                    token = "("
                elif token == "-RRB-":
                    token = ")"
                sample += " " + token
    return pd.DataFrame.from_dict({"sample": samples, "label": relations})

samples = process_data(conll_file)<jupyter_output><empty_output><jupyter_text>Let's inspect all relation labels that we have in our dataset<jupyter_code>print(set(samples["label"]))<jupyter_output>{'org:founded', 'org:country_of_headquarters', 'per:city_of_birth', 'per:origin', 'per:other_family', 'per:country_of_birth', 'org:dissolved', 'org:shareholders', 'per:parents', 'per:cause_of_death', 'per:spouse', 'per:stateorprovince_of_death', 'no_relation', 'org:members', 'org:stateorprovince_of_headquarters', 'org:number_of_employees/members', 'per:cities_of_residence', 'org:political/religious_affiliation', 'per:children', 'per:date_of_death', 'org:top_members/employees', 'org:city_of_headquarters', 'org:subsidiaries', 'per:city_of_death', 'per:countries_of_residence', 'per:schools_attended', 'per:stateorprovinces_of_residence', 'org:alternate_names', 'org:parents', 'per:religion', 'org:founded_by', 'per:age', 'per:employee_of', 'per:title', 'per:date_of_birth', 'org:member_of', 'per:alternate_names', 'per:country_of_death', 'per:stateorprovince_of_birth', 'per:charges', 'org:website', 'per:siblings'}<jupyter_text>To make the calculation quicklier, let's choose only the samples that contain a relation (that is, labelled not with "no_relation" label).<jupyter_code>selected_samples = samples[samples["label"]!='no_relation'].sample(n=1000, random_state=100)
selected_samples.head()<jupyter_output><empty_output><jupyter_text>Defining patterns In order to turn the data into distantly supervised one, we write down a couple of simple patterns for each relation that could help us to find these relations in training samples. For complexity reasons we reduced the number of relation we want to write patterns for and choose 3 relations from the TACRED relation labels: "org:alternate_names", "per:date_of_birth" and "org:top_members/employees".<jupyter_code>relation_patterns = {"org:alternate_names":
                     ["$ARG1 ( $ARG2 ", 
                       "$ARG1 formerly known as $ARG2",
                       "$ARG1 aka $ARG2", 
                       "$ARG1 ( also known as $ARG2 )"],
                     "per:date_of_birth": 
                     ["$ARG1 ( born $ARG2 )", 
                       "$ARG1 ( born $ARG2 in",
                       "$ARG1 ( $ARG2 -",
                       "$ARG1 was born in $ARG2"],
                     "org:top_members/employees":
                     ["$ARG1 , executive director of $ARG2",
                       "$ARG1 , head of $ARG2",
                       "$ARG1 , who heads $ARG2",
                       "$ARG1 , chief executive of $ARG2"]}

relation_patterns_df = pd.DataFrame.from_dict(relation_patterns, orient='index')
relation_patterns_df['raw pattern']=relation_patterns_df.apply(lambda row: row.dropna().tolist(), axis=1)
relation_patterns_df = relation_patterns_df[['raw pattern']]
relation_patterns_df.apply(lambda x: escape_dollar(x)).head()<jupyter_output><empty_output><jupyter_text>Since we want to make a simple regex search, we convert the handßwritten patterns into regexes.<jupyter_code>def preprocess_patterns(patterns):
    regex_patterns = [re.sub("\\\\\\$ARG", "(A )?(a )?(The )?(the )?\\$ARG", re.escape(pattern)) for pattern in patterns]
    return regex_patterns

relation_patterns_df["regex pattern"] = relation_patterns_df["raw pattern"].apply(preprocess_patterns)
relation_patterns_df.apply(lambda x: escape_dollar(x)).head()<jupyter_output><empty_output><jupyter_text>Entity pairing We get the sentence annotations with the Spacy package<jupyter_code>analyzer = spacy.load("en_core_web_sm")
selected_samples["spacy info"] = selected_samples["sample"].apply(lambda x: analyzer(x).to_json())<jupyter_output><empty_output><jupyter_text>After that, we take the arguments pairwise in each sentence and create subsentences that include:- the first entity substituted with "ARG1"- the second argument substituted with "$ARG2"- all the words inbetween<jupyter_code>def get_extracted_sample(sample):
    return [(ARG1 + sample["text"][ent1["end"]:ent2["start"]] + ARG2) if ent1["end"] < ent2["end"] 
            else (ARG2 + sample["text"][ent2["end"]:ent1["start"]] + ARG1)
            for ent1, ent2 in itertools.permutations(sample["ents"],2)]

selected_samples["extr"] = selected_samples["spacy info"].apply(lambda x: get_extracted_sample(x))
selected_samples = selected_samples.loc[selected_samples["extr"].apply(lambda x: len(x) != 0)]
selected_samples[["sample", "label", "extr"]].apply(lambda x: escape_dollar(x)).head()<jupyter_output><empty_output><jupyter_text>Pattern search Now we look for patterns in each of these subsentences. If there is a match, we add a corresponding sentence to the final DataFrame with the information about pattern that matched and weak label it received. The fold labels are still there for comparing.<jupyter_code>def pattern_search(extr_sample, patterns, row):
    for relation, rel_patterns in patterns.iterrows():
        matches = [[row["sample"], extr_sample, pattern, relation, row["label"]] 
                   for pattern in rel_patterns["regex pattern"]
                   if re.match(pattern, extr_sample) is not None]   # todo
        if len(matches) > 0:
            return pd.DataFrame(matches, columns = FINAL_DF_COLUMNS)
all_matches = pd.DataFrame(columns = FINAL_DF_COLUMNS)
for _, row in selected_samples.iterrows():
    for cand_sample in row["extr"]:
        df_found = pattern_search(cand_sample, relation_patterns_df, row)
        if isinstance(df_found, pd.DataFrame) and not df_found.empty:
            all_matches = pd.concat([all_matches, df_found])
all_matches.apply(lambda x: escape_dollar(x)).head()
all_matches.size<jupyter_output><empty_output><jupyter_text>But here we can observe some misclassified sentences: for example, a sentence "A professor emeritus at Yale University , Mandelbrot was born in Poland but as a child moved with his family to France where he was educated" was assigned with a label "per:date_of_birth" (presumably by a pattern "ARG1 was born in ARG2"), what is definitely wrong. In order to avoid such mistake, let's add additional constraints on the argument types.<jupyter_code>relation_to_types = {"org:alternate_names": ['PERSON', 'PERSON'], 
                     "per:date_of_birth": ['PERSON', 'DATE'],
                     "org:top_members/employees": ['PERSON', 'ORG']}<jupyter_output><empty_output><jupyter_text>So, when we look for these patterns in samples, we should take into account the entity types referred to corresponding relation.<jupyter_code>relation_patterns_df["type"] = pd.Series(relation_to_types)
relation_patterns_df.apply(lambda x: escape_dollar(x)).head()<jupyter_output><empty_output><jupyter_text>The information about type of the entities we get from the Spacy annotations as well and build a new format of extracted subsentences: now they are not lists but dictionaries in format {subsentence: entity types}<jupyter_code>def get_arg_types(sample):
    return [{ARG1 + sample["text"][ent1["end"]:ent2["start"]] + ARG2 : [ent1["label"], ent2["label"]]}
            if ent1["end"] < ent2["end"]
            else {(ARG2 + sample["text"][ent2["end"]:ent1["start"]] + ARG1) : [ent2["label"], ent1["label"]]}
            for ent1, ent2 in itertools.permutations(sample["ents"],2)]
selected_samples["extr_with_types"] = selected_samples["spacy info"].apply(lambda x: get_arg_types(x))<jupyter_output><empty_output><jupyter_text>We do a similar pattern search, but now taking into account entity types.<jupyter_code>def pattern_search_filter_types(cand_sample_with_types, patterns, row):
    for relation, rel_patterns in patterns.iterrows():
        cand_sample_types = [item for type_pair in cand_sample_with_types.values() for item in type_pair]
        if rel_patterns["type"] == cand_sample_types:
            sample = list(cand_sample_with_types.keys())[0]
            matches = [[row["sample"], sample, pattern, relation, row["label"]] 
                       for pattern in rel_patterns["regex pattern"]
                       if re.match(pattern, sample) is not None]   # todo
            if len(matches) > 0:
                return pd.DataFrame(matches, columns = FINAL_DF_COLUMNS)
all_matches_filter_types = pd.DataFrame(columns = FINAL_DF_COLUMNS)
for _, row in selected_samples.iterrows():
    for cand_sample_with_types in row["extr_with_types"]:
        df_found = pattern_search_filter_types(cand_sample_with_types, relation_patterns_df, row)
        if isinstance(df_found, pd.DataFrame) and not df_found.empty:
            all_matches_filter_types = pd.concat([all_matches_filter_types, df_found])
all_matches_filter_types.apply(lambda x: escape_dollar(x))
all_matches_filter_types.size<jupyter_output><empty_output><jupyter_text>Input matrices creation So, now we have a smaller dataset (only 60 found samples compared to 120 without argument types filtering), but much more clean. By the way: we have already done our first weakly data preprocessing :)Some weak labels are still not the same as the gold labels, but mostly this happens because they are hold in different parts of a sentence. For example, "Gwathmey was born in 1938 , the only child of painter <NAME> and his wife , Rosalie , a photographer." sentence definitely contains both relations: "per:date_of_birth" and "per:children" - so, our distant label is still true.  The final step would be a creation of the input matrices needed for knodle framework, namely:- X: input_features (sentence x features): containing information about the data samples- Z: rule_matches (sentence x labelling): in our case labelling functions are patterns and this binary matrix contains information about which pattern match for each sentence- T: mapping_rules_labels (labelling functions x classes): which pattern corresponds to which classWe are not going to discuss here encoding of the data samples and creation of X (input_features) matrix, but let's construct Z (rule_matches) and T (mapping_rules_labels) matrices.<jupyter_code>def z_matrix_creation(all_matches_filter_types, relation_patterns_df):
    
    # create empty matrix with columns as patterns + column "sample"
    sentences_lf_df_columns = [item for relation in relation_patterns_df["raw pattern"] for item in relation] 
    rule_matches = pd.DataFrame(columns=escape_dollar(sentences_lf_df_columns))
    rule_matches.insert(0, "sample", all_matches_filter_types["sample"])

    for index, row in all_matches_filter_types.iterrows():   # iterate over rows of found samples
        curr_sample, curr_pattern, curr_relation = row[0], row[2], row[3]
        pattern_index = list(relation_patterns_df.loc[curr_relation]["regex pattern"]).index(curr_pattern)    # find idx of current regex pattern in patterns lists in relation_patterns_df
        raw_pattern = relation_patterns_df.loc[curr_relation, "raw pattern"][pattern_index]   # take the corresponding raw pattern
        rule_matches.loc[(rule_matches["sample"] == curr_sample), escape_dollar([raw_pattern])] = 1   # add 1 on the intersection of sample and pattern in Z matrix

    return rule_matches.fillna(0)

rule_matches = z_matrix_creation(all_matches_filter_types, relation_patterns_df)
rule_matches.head()
mapping_rules_labels = pd.concat([pd.Series(escape_dollar(v), name=k).astype(str) for k, v in relation_patterns.items()], axis=1)
mapping_rules_labels = pd.get_dummies(mapping_rules_labels.stack()).sum(level=1).clip(upper=1)
mapping_rules_labels<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10355591748099892, 'fraction_numerical': 0.008075461454940282, 'mean_word_length': 4.456127360236949, 'pattern_counts': {'":': 11, '<': 65, '<?xml version=': 0, '>': 65, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '263484', 'license': 'Apache-2.0', 'n_tokens_mistral': 4421, 'n_tokens_neox': 4280, 'n_words': 1350, 'path': 'examples/data_preprocessing/tac_based_dataset/[WIP]_patterns_aka_lfs/Prepare_TAC_based_RE_dataset_with_patters_aka_lfs.ipynb', 'repo_name': 'michael-aloys/knodle', 'size': '14771'}