<jupyter_start><jupyter_text>Relation Extraction Ensure that FastAPI is up and running. [instructions here](../fast_api/readme.md)  Table of contents1. Data prep2. Standard        2.1. Train Model        2.2. Predict Labels        2.3. Evaluate Model3. NeXT        3.1. Soft Match            3.1.1. Train Model            3.1.2. Predict Labels            3.1.3. Evaluate Model        3.2. Strict Match            3.2.1. Train Model            3.2.2. Predict Labels            3.2.3. Evaluate Model<jupyter_code># imports
import requests
import json
FAST_API_URL = "http://localhost:9000"<jupyter_output><empty_output><jupyter_text>---  1. Data prep We use a subset of `SemEval 2010 Task 8` dataset in this demo. --- Prepare data for Standard training<jupyter_code># load labelled data without explanations for Standard training
re_standard_labeled_data = list(map(
    lambda x: {'text': x['text'], 'label': x['label']}, 
    json.load(open('data/semeval_labeled.json'))
))
re_standard_labeled_data[0]
re_standard_dev_data = json.load(open('data/semeval_dev.json'))
re_standard_dev_data[0]<jupyter_output><empty_output><jupyter_text>--- Prepare data for NeXT training<jupyter_code># load unlabelled data for NeXT training
re_next_unlabeled_data = json.load(open('data/semeval_unlabeled.json'))
re_next_unlabeled_data[0]
# load labelled data with explanations for NeXT training
re_next_explanation_triples = json.load(open('data/semeval_labeled.json'))
re_next_explanation_triples[0]<jupyter_output><empty_output><jupyter_text>--- Prepare data for evaluation<jupyter_code># load evaluation samples
re_eval_data = json.load(open('data/semeval_test.json'))
# [{text: str, label: str}]
re_eval_data = list(map(lambda doc: [doc['text'], doc['label']], re_eval_data))
# [Sentence, Label] pairs
re_eval_data[0]<jupyter_output><empty_output><jupyter_text>--- Prepare data for prediction<jupyter_code>re_predict_data = list(map(lambda x: x[0], re_eval_data))[:10]
re_predict_data[0]<jupyter_output><empty_output><jupyter_text>--- Define label space<jupyter_code># dictionary of (key: String, value: Int), key being the label and value being unique integer ID
re_label_space = {
  "Product-Producer(e1,e2)": 0,
  "Component-Whole(e2,e1)": 1,
  "Product-Producer(e2,e1)": 2,
  "Instrument-Agency(e2,e1)": 3,
  "Message-Topic(e1,e2)": 4,
  "Entity-Origin(e2,e1)": 5,
  "Entity-Origin(e1,e2)": 6,
  "Entity-Destination(e1,e2)": 7,
  "Cause-Effect(e1,e2)": 8,
  "Content-Container(e1,e2)": 9,
  "Member-Collection(e2,e1)": 10,
  "Instrument-Agency(e1,e2)": 11,
  "Message-Topic(e2,e1)": 12,
  "Content-Container(e2,e1)": 13,
  "no_relation": 14,
  "Member-Collection(e1,e2)": 15,
  "Entity-Destination(e2,e1)": 16,
  "Component-Whole(e1,e2)": 17,
  "Cause-Effect(e2,e1)": 18
}<jupyter_output><empty_output><jupyter_text>--- Define NER label space<jupyter_code>re_ner_label_space = json.load(open('data/semeval_ner_space.json'))
re_ner_label_space<jupyter_output><empty_output><jupyter_text>------ 2. Standard Check [this](../fast_api/json_schema.pyL516) json schema for a list of all parameters.Check [this](../model_training/internal_api/defaults.py) for default values.  2.1. Train model  Define training parameters<jupyter_code>re_standard_train_params = {
    # a string name representing the model name
    "experiment_name": "semeval_re_standard",
    # a string name representing the dataset name
    "dataset_name": "semeval",
    # task type - "re" for Relation Extraction    
    "task": "re",
    # training batch size
    "match_batch_size": 50,
    # learning rate
    "learning_rate": 0.1,
    # number of epochs
    "epochs": 5,
    # embedding to be used for training. usual default: "glove.840B.300d"
    "embeddings": "charngram.100d",
    # embedding dimension of the "embeddings" provided
    "emb_dim": 100,
    # number of hidden dimensions
    "hidden_dim": 100,
    # random seed
    "random_state": 7698,
    # when "True" data has to be passed, 
    # "False" when re-training or the data was processed earlier and can be retrieved
    "build_data": True,
    # label representing the relation "no_relation"
    "none_label_key": "no_relation",
}<jupyter_output><empty_output><jupyter_text>Run model training<jupyter_code># depending on input size, and computing environment this might take time.
# please check FAST API logs for updates
response = requests.post(
    FAST_API_URL + '/training/standard/api/',
    json={
        'params': re_standard_train_params,
        'label_space': re_label_space,
        'labeled_data': re_standard_labeled_data,
        'dev_data': re_standard_dev_data,
        'ner_label_space': re_ner_label_space
    }
)
# JSON with "save_path" key is returned when successful
response.text<jupyter_output><empty_output><jupyter_text>--- 2.2. Predict labels  Define prediction parameters<jupyter_code>re_standard_predict_params = {
    # a string name representing the model name
    "experiment_name": "semeval_re_standard",
    # a string name representing the dataset name
    "dataset_name": "semeval",
    # task type - "re" for Relation Extraction
    "task": "re",
    # evaluation batch size
    "eval_batch_size": 50,
    # embedding to be used for training. usual default: "glove.840B.300d"
    "embeddings": "charngram.100d",
    # embedding dimension of the "embeddings" provided
    "emb_dim": 100,
    # number of hidden dimensions
    "hidden_dim": 100,
    # label representing the relation "no_relation"
    "none_label_key": "no_relation",
}<jupyter_output><empty_output><jupyter_text>Fetch predictions<jupyter_code>response = requests.post(
    FAST_API_URL + '/training/standard/predict/',
    json={
        'params': re_standard_predict_params,
        'label_space': re_label_space,
        'prediction_data': re_predict_data,
        'ner_label_space': re_ner_label_space
    }
)
# returns JSON object with keys
# "class_probs" - list of list representing predictions for each label
# "class_preds" - list representing the label that was predicted
response.text<jupyter_output><empty_output><jupyter_text>--- 2.3. Evaluate model  Define evaluation parameters<jupyter_code>re_standard_eval_params = {
    # a string name representing the model name
    "experiment_name": "semeval_re_standard",
    # a string name representing the dataset name
    "dataset_name": "semeval",
    # task type - "re" for Relation Extraction
    "task": "re",
    # evaluation batch size
    "eval_batch_size": 50,
    # embedding to be used for training. usual default: "glove.840B.300d"
    "embeddings": "charngram.100d",
    # embedding dimension of the "embeddings" provided
    "emb_dim": 100,
    # number of hidden dimensions
    "hidden_dim": 100,
    # label representing the relation "no_relation"
    "none_label_key": "no_relation",
}<jupyter_output><empty_output><jupyter_text>Run model evaluation<jupyter_code>response = requests.post(
    FAST_API_URL + '/training/standard/eval/',
    json={
        'params': re_standard_eval_params,
        'label_space': re_label_space,
        'eval_data': re_eval_data,
        'ner_label_space': re_ner_label_space
    }
)
# returns JSON object with keys
# "avg_loss" and "avg_eval_f1_score"
response.text<jupyter_output><empty_output><jupyter_text>------ 3. NeXT Check [this](../fast_api/json_schema.pyL293) json schema for a list of all parameters.Check [this](../model_training/internal_api/defaults.py) for default values.  3.1. Soft Match  3.1.1. Train model  Define training parameters<jupyter_code>re_next_soft_match_params = {
    # a string name representing the model name
    "experiment_name": "semeval_re_nle_soft_match",
    # a string name representing the dataset name
    "dataset_name": "semeval",
    # integer representing the unlabelled dataset size
    "dataset_size": len(re_next_unlabeled_data),
    # task type - "re" for Relation Extraction
    "task": "re",
    # the total batch_size is "match_batch_size" + "unlabeled_batch_size"
    "match_batch_size": 50,
    "unlabeled_batch_size": 100,
    # learning rate
    "learning_rate": 0.1,
    # number of epochs
    "epochs": 5,
    # embedding to be used for training. usual default: "glove.840B.300d"
    "embeddings": "charngram.100d", # usual default: "glove.840B.300d"
    # embedding dimension of the "embeddings" provided
    "emb_dim": 100,
    # soft_match loss weightage
    "gamma": 0.5,
    # number of hidden dimensions
    "hidden_dim": 100,
    # random seed
    "random_state": 7698,
    # FIND module number of hidden dimensions
    "pre_train_hidden_dim": 300,
    # FIND module training_size
    "pre_train_training_size": 50000,
    
    # enable soft match
    "soft_match": True,
    
    # "find" to train only FIND module, 
    # "clf" to train only the classifier,
    # "both" to train FIND module + classifier
    "stage": "both",
    
    # for the FIND module,
    # when "True" data has to be passed, 
    # "False" when re-training or the data was processed earlier and can be retrieved
    "pre_train_build_data": True,
    
    # when "True" data has to be passed, 
    # "False" when re-training or the data was processed earlier and can be retrieved
    "build_data": True,
    
    "pre_train_epochs": 5,
}<jupyter_output><empty_output><jupyter_text>Run model training<jupyter_code># depending on input size, and computing environment this might take time.
# please check FAST API logs for updates
response = requests.post(
    FAST_API_URL + '/training/next/api/',
    json={
        'params': re_next_soft_match_params,
        'label_space': re_label_space,
        'explanation_triples': re_next_explanation_triples,
        'unlabeled_text': re_next_unlabeled_data,
        'ner_label_space': re_ner_label_space
    }
)
# JSON with "save_path" key is returned when successful
response.text<jupyter_output><empty_output><jupyter_text>--- 3.1.2. Predict labels  Define prediction parameters<jupyter_code>re_next_soft_predict_params = {
    # a string name representing the model name
    "experiment_name": "semeval_re_nle_soft_match",
    # a string name representing the dataset name
    "dataset_name": "semeval",
    # integer representing the unlabelled dataset size
    "train_dataset_size": len(re_next_unlabeled_data),
    # task type - "re" for Relation Extraction
    "task": "re",
    # evaluation batch size
    "eval_batch_size": 50,
    # embedding to be used for training. usual default: "glove.840B.300d"
    "embeddings": "charngram.100d", # usual default: "glove.840B.300d"
    # embedding dimension of the "embeddings" provided
    "emb_dim": 100,
    # number of hidden dimensions
    "hidden_dim": 100,
    # FIND module training_size
    "pre_train_training_size": 50000,
    # label representing the relation "no_relation"
    "none_label_key": "no_relation",
}<jupyter_output><empty_output><jupyter_text>Fetch predictions<jupyter_code>response = requests.post(
    FAST_API_URL + '/training/next/predict/',
    json={
        'params': re_next_soft_predict_params,
        'label_space': re_label_space,
        'prediction_data': re_predict_data,
        'ner_label_space': re_ner_label_space
    }
)
# returns JSON object with keys
# "class_probs" - list of list representing predictions for each label
# "class_preds" - list representing the label that was predicted
response.text<jupyter_output><empty_output><jupyter_text>--- 3.1.3. Evaluate model  Define evaluation parameters<jupyter_code>re_next_soft_eval_params = {
    # a string name representing the model name
    "experiment_name": "semeval_re_nle_soft_match",
    # a string name representing the dataset name
    "dataset_name": "semeval",
    # integer representing the unlabelled dataset size
    "train_dataset_size": len(re_next_unlabeled_data),
    # task type - "re" for Relation Extraction
    "task": "re",
    # evaluation batch size
    "eval_batch_size": 50,
    # embedding to be used for training. usual default: "glove.840B.300d"
    "embeddings": "charngram.100d", # usual default: "glove.840B.300d"
    # embedding dimension of the "embeddings" provided
    "emb_dim": 100,
    # number of hidden dimensions
    "hidden_dim": 100,
    # FIND module training_size
    "pre_train_training_size": 50000,
    # label representing the relation "no_relation"
    "none_label_key": "no_relation",
}<jupyter_output><empty_output><jupyter_text>Run model evaluation<jupyter_code>response = requests.post(
    FAST_API_URL + '/training/next/eval/',
    json={
        'params': re_next_soft_eval_params,
        'label_space': re_label_space,
        'eval_data': re_eval_data,
        'ner_label_space': re_ner_label_space
    }
)
# returns JSON object with keys
# "avg_loss", "avg_eval_ent_f1_score", "avg_eval_ent_f1_score" and "no_relation_thresholds"
response.text<jupyter_output><empty_output><jupyter_text>------ 3.2. Strict Match  3.2.1. Train model  Define training parameters<jupyter_code># parameters are same as in "soft_match" except "soft_match" is set to "False" here
re_next_strict_match_params = {
    # a string name representing the model name
    "experiment_name": "semeval_re_nle_strict_match",
    # a string name representing the dataset name
    "dataset_name": "semeval",
    # integer representing the unlabelled dataset size
    "dataset_size": len(re_next_unlabeled_data),
    # task type - "re" for Relation Extraction
    "task": "re",
    # the total batch_size is "match_batch_size" + "unlabeled_batch_size"
    "match_batch_size": 50,
    "unlabeled_batch_size": 100,
    # learning rate
    "learning_rate": 0.1,
    # number of epochs
    "epochs": 5,
    # embedding to be used for training. usual default: "glove.840B.300d"
    "embeddings": "charngram.100d", # usual default: "glove.840B.300d"
    # embedding dimension of the "embeddings" provided
    "emb_dim": 100,
    # soft_match loss weightage
    "gamma": 0.5,
    # number of hidden dimensions
    "hidden_dim": 100,
    # random seed
    "random_state": 7698,
    # FIND module number of hidden dimensions
    "pre_train_hidden_dim": 300,
    # FIND module training_size
    "pre_train_training_size": 50000,
    
    # disable soft match
    "soft_match": False,
    
    # "find" to train only FIND module, 
    # "clf" to train only the classifier,
    # "both" to train FIND module + classifier
    "stage": "both",
    
    # for the FIND module,
    # when "True" data has to be passed, 
    # "False" when re-training or the data was processed earlier and can be retrieved
    "pre_train_build_data": True,
    
    # when "True" data has to be passed, 
    # "False" when re-training or the data was processed earlier and can be retrieved
    "build_data": True,
    
    "pre_train_epochs": 5,
}<jupyter_output><empty_output><jupyter_text>Run model training<jupyter_code># depending on input size, and computing environment this might take time.
# please check FAST API logs for updates
response = requests.post(
    FAST_API_URL + '/training/next/api/',
    json={
        'params': re_next_strict_match_params,
        'label_space': re_label_space,
        'explanation_triples': re_next_explanation_triples,
        'unlabeled_text': re_next_unlabeled_data,
        'ner_label_space': re_ner_label_space
    }
)
# JSON with "save_path" key is returned when successful
response.text<jupyter_output><empty_output><jupyter_text>--- 3.2.2. Predict labels  Define prediction parameters<jupyter_code>re_next_strict_predict_params = {
    # a string name representing the model name
    "experiment_name": "semeval_re_nle_strict_match",
    # a string name representing the dataset name
    "dataset_name": "semeval",
    # integer representing the unlabelled dataset size
    "train_dataset_size": len(re_next_unlabeled_data),
    # task type - "re" for Relation Extraction
    "task": "re",
    # evaluation batch size
    "eval_batch_size": 50,
    # embedding to be used for training. usual default: "glove.840B.300d"
    "embeddings": "charngram.100d", # usual default: "glove.840B.300d"
    # embedding dimension of the "embeddings" provided
    "emb_dim": 100,
    # number of hidden dimensions
    "hidden_dim": 100,
    # FIND module training_size
    "pre_train_training_size": 50000,
    # label representing the relation "no_relation"
    "none_label_key": "no_relation",
}<jupyter_output><empty_output><jupyter_text>Fetch predictions<jupyter_code>response = requests.post(
    FAST_API_URL + '/training/next/predict/',
    json={
        'params': re_next_strict_predict_params,
        'label_space': re_label_space,
        'prediction_data': re_predict_data,
        'ner_label_space': re_ner_label_space
    }
)
# returns JSON object with keys
# "class_probs" - list of list representing predictions for each label
# "class_preds" - list representing the label that was predicted
response.text<jupyter_output><empty_output><jupyter_text>--- 3.2.3. Evaluate model  Define evaluation parameters<jupyter_code>re_next_strict_eval_params = {
    # a string name representing the model name
    "experiment_name": "semeval_re_nle_strict_match",
    # a string name representing the dataset name
    "dataset_name": "semeval",
    # integer representing the unlabelled dataset size
    "train_dataset_size": len(re_next_unlabeled_data),
    # task type - "re" for Relation Extraction
    "task": "re",
    # evaluation batch size
    "eval_batch_size": 50,
    # embedding to be used for training. usual default: "glove.840B.300d"
    "embeddings": "charngram.100d", # usual default: "glove.840B.300d"
    # embedding dimension of the "embeddings" provided
    "emb_dim": 100,
    # number of hidden dimensions
    "hidden_dim": 100,
    # FIND module training_size
    "pre_train_training_size": 50000,
    # label representing the relation "no_relation"
    "none_label_key": "no_relation",
}<jupyter_output><empty_output><jupyter_text>Run model evaluation<jupyter_code>response = requests.post(
    FAST_API_URL + '/training/next/eval/',
    json={
        'params': re_next_strict_eval_params,
        'label_space': re_label_space,
        'eval_data': re_eval_data,
        'ner_label_space': re_ner_label_space
    }
)
# returns JSON object with keys
# "avg_loss", "avg_eval_ent_f1_score", "avg_eval_ent_f1_score" and "no_relation_thresholds"
response.text<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1068173134654222, 'fraction_numerical': 0.02254243763986682, 'mean_word_length': 4.043214973850812, 'pattern_counts': {'":': 127, '<': 101, '<?xml version=': 0, '>': 101, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '494290', 'license': 'MIT', 'n_tokens_mistral': 6309, 'n_tokens_neox': 5970, 'n_words': 1546, 'path': 'example_notebooks/relation_extraction.ipynb', 'repo_name': 'INK-USC/LEAN-LIFE', 'size': '18321'}