<jupyter_start><jupyter_text>EDA - Grail QAThe data is in `JSON` format, and the schema is provided below. For our purposes, we only care about "domains" and "question".```{  "answer": {    "[]": {      "answer_argument": "string",      "answer_type": "string",      "entity_name": "string"    }  },  "domains": {    "[]": "string"  },  "function": "string",  "graph_query": {    "edges": {      "[]": {        "end": "int32",        "friendly_name": "string",        "relation": "string",        "start": "int32"      }    },    "nodes": {      "[]": {        "class": "string",        "friendly_name": "string",        "function": "string",        "id": "string",        "nid": "int32",        "node_type": "string",        "question_node": "int32"      }    }  },  "level": "string",  "num_edge": "int32",  "num_node": "int32",  "qid": "string",  "question": "string",  "s_expression": "string",  "sparql_query": "string"}```<jupyter_code>import pandas as pd

pd.options.display.max_rows = 150
pd.options.display.max_colwidth = 0
from src.data.utils import *

train = pd.DataFrame(get_domains_and_questions('train', 'grail_qa'))
train.head()
train.domains.value_counts()<jupyter_output><empty_output><jupyter_text>From this list, there are a handful of candidate subdomains that may fall under the umbrella domains of `healthcare` and `technology`.   Candidates for `healthcare`<jupyter_code>from functools import partial

def get_candidate_sample(df, subdomain, n=5):
    return df.loc[df.domains == subdomain].sample(n)

get_sample = partial(get_candidate_sample, train)
get_sample('medicine')
get_sample('biology')
get_sample('biology;people')
get_sample('business;medicine')<jupyter_output><empty_output><jupyter_text>Candidates for `technology`<jupyter_code>get_sample('computer')
get_sample('spaceflight')
get_sample('aviation')
get_sample('automotive')
get_sample('broadcast')
get_sample('internet')
get_sample('engineering')
get_sample('military')<jupyter_output><empty_output><jupyter_text>OntologyIn a real-time meeting, we took samples as below of candidate subdomains and agreed on which of them we thought should map to `healthcare` or `technology`.<jupyter_code>get_candidate_sample(train, 'engineering', n=10)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1595037660611431, 'fraction_numerical': 0.008418254319893664, 'mean_word_length': 5.102702702702703, 'pattern_counts': {'":': 31, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '448918', 'license': 'MIT', 'n_tokens_mistral': 777, 'n_tokens_neox': 736, 'n_words': 165, 'path': 'notebooks/amqdn-0.1-eda-grail-qa.ipynb', 'repo_name': 'kinga-k-farkas/glg-ack', 'size': '2257'}