<jupyter_start><jupyter_text>1. Load the data<jupyter_code>#'https://raw.githubusercontent.com/covid19kg/covid19kg/master/supplement/terminology.csv'
url = 'https://raw.githubusercontent.com/CoronaWhy/task-vt/kaleidoescape_kg/immunology_kg/relations/covid19_frauenhofer_annotations.csv'
pybel_pd = pd.read_csv(url)
pybel_pd.head(7)
#list all types of relations
relation_categories = pybel_pd['relation'].unique()
relation_categories<jupyter_output><empty_output><jupyter_text>2. Snorkel ExampleFor the purpose of this example we'll only focus on rows with 'negativeCorrelation' and 'positiveCorrelation' as their relations.<jupyter_code>example_data = pybel_pd[(pybel_pd['relation']=='negativeCorrelation') | (pybel_pd['relation']=='positiveCorrelation') ]
example_data['relation'] = example_data['relation']=='negativeCorrelation'
example_data.reset_index(inplace=True,drop=True)
example_data.drop('Unnamed: 0',inplace=True,axis=1)
example_data.head()<jupyter_output>C:\Users\Cafral\Anaconda3\lib\site-packages\ipykernel_launcher.py:2: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  
C:\Users\Cafral\Anaconda3\lib\site-packages\pandas\core\frame.py:4102: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  errors=errors,<jupyter_text>2.1 Split the data into training and testing Ideally should have training , validation and testing set. Also, here , I'm using a fixed testing period but k-fold cross validation techniques are a more robust way of determining the accuracy of the generated labels.<jupyter_code>df_train,df_test,y_train,y_test = train_test_split(example_data[['text']],
                                                   example_data[['relation']],
                                                   test_size=0.20,
                                                   shuffle=False,random_state=1)<jupyter_output><empty_output><jupyter_text>2.2 Reading sentences to understand syntactic differences betweem negative and positive correlation sentencesThe utility of snorkel is that it allows you to create multiple labelling functions which (try to) mimic the rules that a human annotator of data would apply while deciding how to label unlabelled data. For instance a human annotator looking to identify negative correlation is sentences will follow the following rules: A. Positive rules: rules which tell you what a sentences with negative correlation looks like1) does the sentence contain the reduction related words like words 'decreased','reduced','lowered'2) does the sentence contain the expression 'negatively correlated' or 'negative correlation' or 'negatively related' or 'inversely related' or 'inverse relation'3) does the syntax follow the structure 'increase in x is associated with a decrease in y' (and vice versa)4) does the sentence contain the expression 'negative effect'5) does the sentence contain the expression 'move in opposite directions' B. Negative rules: rules which tell you what a sentences without negative correlation looks like1) does the sentence contain the increase related words like words 'increased','improved'2) does the sentence contain the expression 'positively correlated' or 'positive correlation' or 'positively related'3) does the syntax follow the structure 'increase in x is associated with a increase in y' (and vice versa)4) does the sentence contain the expression 'positive effect'5) does the sentence contain the expression 'move in the same direction'These rules can be coded using snorkel. Importantly it requires both positive and negative rules.   2.2.1 Examining Negative correlation sentences to understand their syntactic structure and then define labelling functions accordingly<jupyter_code>neg_correl_df = example_data[example_data['relation']==1]
neg_correl_df.reset_index(inplace=True,drop=True)
neg_correl_df.head(8)
neg_correl_df['text'][0]<jupyter_output><empty_output><jupyter_text>2.2.2 Examining positive correlation sentences to understand their syntactic structure and then define labelling funtions accordingly<jupyter_code>positive_relation_df = example_data[example_data['relation']==0]
positive_relation_df.reset_index(inplace=True,drop=True)
positive_relation_df.head(8)
positive_relation_df['text'][0]<jupyter_output><empty_output><jupyter_text>2.3 Source-Target dictionary A simple but clean rule for identifying negative correlation sentences would be if negative tokens occured in the words between the source and the target. So, a source-target dictonary is created for some of the examples (in the final pipeline the source target dictonary will be obtained from the spacy pipeline).<jupyter_code>example_sources = ['tetrandrine','triiodothyronine','methylprednisolone','IFN-α','angiotensin','ACE2','fibrin','hemorrhage','chloroquine']

example_targets = ['TPC2','recovered','death','fever','vasodilator','Ang','COVID-19','fibrinolysis','dendritic'] #low-grade fever

example_source_target_dict = list(OrderedDict.fromkeys(zip(example_sources,example_targets)))
example_source_target_dict<jupyter_output><empty_output><jupyter_text>2.4 Labeling functions for RE<jupyter_code>spacy = SpacyPreprocessor(text_field="text", doc_field="doc", memoize=True)

ABSTAIN = -1
NOT_FOUND = 0
FOUND = 1<jupyter_output><empty_output><jupyter_text>2.4.1 Preprocessing<jupyter_code>get_source_target = make_source_target_preprocessor(spacy, example_sources, example_targets)

candidate = example_data.loc[0]
candidate_with_function_applied = get_source_target(candidate) 

print("Sentence: ", candidate["text"],'\n',)
print("Source-target pair: ", candidate_with_function_applied.source_target)
get_text_between = make_text_between_preprocessor(spacy, example_sources, example_targets)

############ function example ###############################
candidate = example_data.loc[2]

candidate_with_function_applied = get_text_between(candidate)

print("Sentence: ", candidate["text"],'\n')
print("Text Between: ", candidate_with_function_applied.text_between,'\n')
print("Text to the left: ", candidate_with_function_applied.text_to_source_left,'\n')<jupyter_output>Sentence:  Thyroid stimulating hormone and free triiodothyronine concentrations were significantly lower in deceased patients (0.7 mIU/mL and 2.8 pmol/L) than in recovered patients (1.4 mIU/mL and 4.3 pmol/L). 

Text Between:  triiodothyronine concentrations were significantly lower in deceased patients (0.7 mIU/mL and 2.8 pmol/L) than in 

Text to the left:  Thyroid stimulating hormone and free<jupyter_text>2.4.2 Defining the labelling functions (lf)For the final labelling model to work, at least 3 rules are needed. A. Positive rules: rules which tell you what a sentences with negative correlation looks like    1) does the sentence contain the reduction related words like words 'decreased','reduced','lowered'<jupyter_code>reduction_tokens = {'decreased',
                            'lower',
                            'reduced',
}
@labeling_function(pre=[spacy],resources=dict(reduction_tokens=reduction_tokens))
def contains_reduction_tokens(x,reduction_tokens):
    
    tokens = [str(token) for token in x.doc]
    return FOUND if len(reduction_tokens.intersection(set(tokens))) > 0 else ABSTAIN

#positive rule - version 2
@labeling_function(pre=[spacy,get_text_between],resources=dict(reduction_tokens=reduction_tokens))
def contains_reduction_tokens_text_between(x,reduction_tokens):
    relation_text = x.text_between
    relation_text_tokens = [str(token) for token in relation_text]
    return FOUND if len(reduction_tokens.intersection(set(relation_text_tokens))) > 0 else ABSTAIN<jupyter_output><empty_output><jupyter_text>2) does the sentence contain the expression 'negatively correlated' or 'negative correlation' or 'negatively related' or 'inversely related' or 'inverse relation'<jupyter_code>negative_correlation_regex_1 = 'negative correlation'
negative_correlation_regex_2 = 'negatively correlated'
negative_correlation_regex_3 = 'negatively related'
negative_correlation_regex_4 = 'inversely related'
negative_correlation_regex_5 = 'inverse relation'
negative_correlation_regex_6 = 'negative effect'
negative_correlation_regex_7 = 'move in opposite directions'

@labeling_function()
def contains_negative_corrrelation_regex(x):
    if re.search(negative_correlation_regex_1, x.text, flags=re.I):
        return FOUND
    elif re.search(negative_correlation_regex_2, x.text, flags=re.I):
        return FOUND
    elif re.search(negative_correlation_regex_3, x.text, flags=re.I):
        return FOUND
    elif re.search(negative_correlation_regex_4, x.text, flags=re.I):
        return FOUND
    elif re.search(negative_correlation_regex_5, x.text, flags=re.I):
        return FOUND
    elif re.search(negative_correlation_regex_6, x.text, flags=re.I):
        return FOUND
    elif re.search(negative_correlation_regex_7, x.text, flags=re.I):
        return FOUND

    else: 
        return ABSTAIN<jupyter_output><empty_output><jupyter_text>3) does the syntax follow the structure 'increase in x is associated with a decrease in y' (and vice versa)<jupyter_code>@labeling_function(pre=[spacy,get_text_between])
def contains_increase_decrease_pattern(x):
    if ('increase' in [str(token) for token in x.text_to_source_left]) & ('decrease' in [str(token) for token in x.text_between]):
        return FOUND
    elif ('decrease' in [str(token) for token in x.text_to_source_left]) & ('increase' in [str(token) for token in x.text_between]):
        return FOUND
    else:
        return ABSTAIN<jupyter_output><empty_output><jupyter_text>B. Negative rules: rules which tell you what a sentences without negative correlation looks like1) does the sentence contain the increase related words like words 'increased','higher'<jupyter_code>increase_tokens = {'increased',
                            'higher',
}
@labeling_function(pre=[spacy],resources=dict(increase_tokens=increase_tokens))
def contains_increase_tokens(x,increase_tokens):
    tokens = [str(token) for token in x.doc]
    return NOT_FOUND if len(increase_tokens.intersection(set(tokens))) > 0 else ABSTAIN


@labeling_function(pre=[spacy,get_text_between],resources=dict(increase_tokens=increase_tokens))
def contains_increase_tokens_text_between(x, increase_tokens):
    relation_text = x.text_between
    relation_text_tokens = [str(token) for token in relation_text]
    return NOT_FOUND if len(increase_tokens.intersection(set(relation_text_tokens))) > 0 else ABSTAIN<jupyter_output><empty_output><jupyter_text>2) does the sentence contain the expression 'positively correlated' or 'positive correlation' or 'positively related' or 'positive effect' or 'move in the same direction'<jupyter_code>#regex
positive_correlation_regex_1 = 'positive correlation'
positive_correlation_regex_2 = 'positively correlated'
positive_correlation_regex_3 = 'positively related'
positive_correlation_regex_4 = 'positive effect'
positive_correlation_regex_5 = 'move in the same direction'

@labeling_function()
def contains_positive_corrrelation_regex(x):
    if re.search(positive_correlation_regex_1, x.text, flags=re.I):
        return FOUND
    elif re.search(positive_correlation_regex_2, x.text, flags=re.I):
        return FOUND
    elif re.search(positive_correlation_regex_3, x.text, flags=re.I):
        return FOUND
    elif re.search(positive_correlation_regex_4, x.text, flags=re.I):
        return FOUND
    elif re.search(positive_correlation_regex_5, x.text, flags=re.I):
        return FOUND    
    else:
        return ABSTAIN<jupyter_output><empty_output><jupyter_text>3) does the syntax follow the structure 'increase in x is associated with a increase in y' (and vice versa)<jupyter_code>@labeling_function(pre=[spacy,get_text_between])
def contains_increase_increase_pattern(x):
    if ('increase' in [str(token) for token in x.text_to_source_left]) & ('increase' in [str(token) for token in x.text_between]):
        return FOUND
    elif ('decrease' in [str(token) for token in x.text_to_source_left]) & ('decrease' in [str(token) for token in x.text_between]):
        return FOUND
    else:
        return ABSTAIN<jupyter_output><empty_output><jupyter_text>2.5 Creating all the labels for the different rules<jupyter_code>label_functions_list = [contains_reduction_tokens,
                        contains_reduction_tokens_text_between,
                        contains_negative_corrrelation_regex,
                        contains_increase_decrease_pattern,
                        contains_increase_tokens,
                        contains_increase_tokens_text_between,
                        contains_positive_corrrelation_regex,
                        contains_increase_increase_pattern
                       ]

applier = PandasLFApplier(label_functions_list)

label_matrix_train = applier.apply(df_train)<jupyter_output>C:\Users\Cafral\Anaconda3\lib\site-packages\tqdm\std.py:658: FutureWarning: The Panel class is removed from pandas. Accessing it from the top-level namespace will also be removed in the next version
  from pandas import Panel
100%|█████████████████████████████████████████████████████████████████████████████| 2227/2227 [00:10<00:00, 213.19it/s]<jupyter_text>2.6 Examining the quality of the labels<jupyter_code>#in the absence of a benchmark to compare against
LFAnalysis(L=label_matrix_train, lfs=label_functions_list).lf_summary()
#examining the quality of the labels in the presence of a benchmark to compare against
LFAnalysis(L=label_matrix_train, lfs=label_functions_list).lf_summary(y_train.values)
#examine which sentences were picked up as showing negative correlation by each label function
df_train.iloc[label_matrix_train[:, 0] == FOUND]<jupyter_output><empty_output><jupyter_text>2.7 Predict the final labelDifferent models can be used to create the final model that aggrateges the different label functions to perdict the final lebel.<jupyter_code>#testing data
label_matrix_test = applier.apply(df_test)
#Model 1 : majority model (mm)
majority_model = MajorityLabelVoter()

#training data
mm_preds_class_train = majority_model.predict(L=label_matrix_train)
mm_preds_proba_train = majority_model.predict_proba(L=label_matrix_train)

#testing data
mm_preds_class_test = majority_model.predict(L=label_matrix_test)
mm_preds_proba_test = majority_model.predict_proba(L=label_matrix_test)
mm_preds_class_train # only the 1s and 0s are labels. T-1s are abstains i.e. unlabeled data points
mm_preds_proba_train
# Model 2:label model (lm)

#call the model
label_model = LabelModel(cardinality=2, verbose=True)

#fit the model
num_epochs = 1000
log_frequency = 100
random_seed = 1
label_model.fit(L_train=label_matrix_train, n_epochs=num_epochs, log_freq=log_frequency, seed=random_seed)

#generate lables for training data
lm_preds_proba_train = label_model.predict_proba(label_matrix_train)
lm_preds_class_train = probs_to_preds(lm_preds_proba_train)

#generate labels for testing data
lm_preds_proba_test = label_model.predict_proba(label_matrix_test)
lm_preds_class_test = probs_to_preds(lm_preds_proba_test)
# Model 3 : Random Voter (rv)

random_voter = RandomVoter()

#training data
rv_preds_class_train = random_voter.predict(L=label_matrix_train)
rv_preds_proba_train = random_voter.predict_proba(L=label_matrix_train)

#testing data
rv_preds_class_test = random_voter.predict(L=label_matrix_test)
rv_preds_proba_test = random_voter.predict_proba(L=label_matrix_test)<jupyter_output><empty_output><jupyter_text>2.8 Comparing different models<jupyter_code>all_models = {'Majority Model':majority_model,
              'Label Model':label_model,
              'Random Voter Model':random_voter}

for model_name,model in all_models.items():
    
    #accuracy
    train_acc = model.score(L=label_matrix_train, Y=y_train, tie_break_policy="random")["accuracy"]
    test_acc = model.score(L=label_matrix_test, Y=y_test, tie_break_policy="random")["accuracy"]
    
    #auc
    train_auc = metric_score(y_train, probs=model.predict_proba(L=label_matrix_train), metric='roc_auc')
    test_auc = metric_score(y_test, probs=model.predict_proba(L=label_matrix_test), metric='roc_auc')
    
    print(f'{model_name}','\n',
          'Accuracy:','\n','train->',train_acc,'\n','test->',test_acc,'\n',
          'AUC:','\n','train->',train_auc,'\n','test->',test_auc,'\n')<jupyter_output>Majority Model 
 Accuracy: 
 train-> 0.5635383924562192 
 test-> 0.4703770197486535 
 AUC: 
 train-> 0.6010212548732079 
 test-> 0.5227059436913452 

Label Model 
 Accuracy: 
 train-> 0.5527615626403233 
 test-> 0.49012567324955114 
 AUC: 
 train-> 0.524345344386498 
 test-> 0.4437434827945777 

Random Voter Model 
 Accuracy: 
 train-> 0.5024696901661428 
 test-> 0.5008976660682226 
 AUC: 
 train-> 0.5126309212678816 
 test-> 0.5023114355231144<jupyter_text>The label model has the highest test AUC so that's the  best model.  3. Filter out unlabeled points<jupyter_code>#training labels
df_train_filtered, probs_train_filtered = filter_unlabeled_dataframe(
    X=df_train['text'], 
    y=mm_preds_proba_train, 
    L=label_matrix_train
)

#testing labels
df_test_filtered, probs_test_filtered = filter_unlabeled_dataframe(
    X=df_test['text'], 
    y=mm_preds_proba_test, 
    L=label_matrix_test
)

print('Total points labelled in training data:',len(df_train_filtered))
print('Total points labelled in testing data:',len(df_test_filtered))<jupyter_output>Total points labelled in training data: 405
Total points labelled in testing data: 91
{'fraction_non_alphanumeric': 0.09417894559919324, 'fraction_numerical': 0.02112163146394756, 'mean_word_length': 5.685393258426966, 'pattern_counts': {'":': 0, '<': 73, '<?xml version=': 0, '>': 92, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '461857', 'license': 'MIT', 'n_tokens_mistral': 5943, 'n_tokens_neox': 5440, 'n_words': 1361, 'path': 'immunology_kg/notebooks/snorkel_re_example/Snorkel RE example.ipynb', 'repo_name': 'kritim13/task-vt', 'size': '17849'}