<jupyter_start><jupyter_text>Information Retrieval Exercise 1 NotebookThis is the template notebook for Exercise 1. The specification for the exercise and the corresponding Exercise 1 Quiz submission instance are available on the Moodle page of the course. Assumed KnowledgeThis Exercise assumes knowedge of Pandas and PyTerrier from the introductory lab. The relevant parts of the PyTerrier documentation are: - [Using Terrier indices in PyTerrier](https://pyterrier.readthedocs.io/en/latest/terrier-indexing.html) - [Terrier Retrieval using PyTerrier](https://pyterrier.readthedocs.io/en/latest/terrier-retrieval.html), e.g. BatchRetrieve - [Operators on PyTerrier transformers](https://pyterrier.readthedocs.io/en/latest/operators.html) Setup<jupyter_code>!pip install python-terrier
import pyterrier as pt
if not pt.started():
  pt.init()

import pandas as pd
pd.set_option('display.max_colwidth', 150)<jupyter_output><empty_output><jupyter_text>Datasets for Ex1For Exercise 1, we'll be using the Datasets API to obtain the files we need for this exercise. PyTerrier actually provides many datasets. You can list all of them using `pt.list_datasets()`.<jupyter_code>pt.list_datasets()<jupyter_output><empty_output><jupyter_text>There are several sets of files we need for Exercise 1: - the index for 50% of the TREC GOV corpus. We provide this through the "50pct" dataset, but you will need the username and password that we assigned to you after you signed the user license agreement. - the topics (queries) and qrels (relevance assessments) for evaluating the performance of our search engine. These come from the "trec-wt-2004" dataset.Update your username and password. DO NOT SHARE your login details with other students - all they need to do is submit a signed user agreement to us.<jupyter_code>USERNAME = "TODO"
PASSWORD = "<PASSWORD>"

dotgov_50pct = pt.get_dataset("50pct", user='2210049p', password='<PASSWORD>')
dotgov_topicsqrels =  pt.get_dataset("trec-wt-2004")<jupyter_output><empty_output><jupyter_text>The size of the "50pct" index is 800MB - this will take a minute or so for Colab to download before we load it for the first time.<jupyter_code>indexref = dotgov_50pct.get_index()
index = pt.IndexFactory.of(indexref)<jupyter_output><empty_output><jupyter_text>Q1Using this setup, you now have sufficient knowledge from the introductory lab to complete Q1 of this assessed exercise, namely to get the indexing statistics of the "50pct" collection.<jupyter_code>#Get index statistics of collection
print(index.getCollectionStatistics().toString())<jupyter_output>Number of documents: 807775
Number of terms: 2043788
Number of postings: 177737957
Number of fields: 0
Number of tokens: 572916194
Field names: []
Positions:   false<jupyter_text>Retrieval & EvaluationIn our experiments, we are using three sets of topics: "hp", "np" and "td". We can obtain them from the `dotgov_topicsqrels` dataset as follows:<jupyter_code>topics = dotgov_topicsqrels.get_topics(variant="hp")
topics.head(5)<jupyter_output><empty_output><jupyter_text>Let's create a simple TF_IDF retriever - we will use this for demonstrating IR evaluation using PyTerrier.<jupyter_code>retr = pt.BatchRetrieve(index, wmodel="TF_IDF")<jupyter_output><empty_output><jupyter_text>Let's see how we can actually evaluate our TF_IDF retrieval system. Firstly, we'll need the qrels.<jupyter_code>qrels = dotgov_topicsqrels.get_qrels(variant='hp')<jupyter_output><empty_output><jupyter_text>We can use `pt.Utils.evaluate(results, qrels)` to evaluate the results    Args:     - metrics, `default = ["map", ndcg"]`, select the evaluation metrics     - perquery, `default = False`, select whether to show the mean of the metrics over all queries or the metrics for each query<jupyter_code>res = retr.transform(topics)
eval = pt.Utils.evaluate(res, qrels, metrics=["map"])
eval<jupyter_output><empty_output><jupyter_text>However, creating the res dataframe for each system in turn, and then evaluating it is laborious. We strongly recommend using [`pt.Experiment()`](https://pyterrier.readthedocs.io/en/latest/experiments.html) to evaluate one or more retrieval systems at once, in a declarative manner. Take the time to read the [documentation for `pt.Experiment()`](https://pyterrier.readthedocs.io/en/latest/experiments.html) to understand its available functionality. Q2-Q4 will all require that you adapt the arguments to `pt.Experiment()` and use its output in different ways.<jupyter_code>pt.Experiment(
    [retr],
     dotgov_topicsqrels.get_topics(variant='hp'),
     dotgov_topicsqrels.get_qrels(variant='hp'), 
     eval_metrics=['map']
)<jupyter_output><empty_output><jupyter_text>Q2(a)You now have enough information to complete Q2(a) of the assessed exercise. *Hint*: We encourage you to write your own functions that perform reusable operations across different topic sets.<jupyter_code>#Get the indexes for different models
pl2 = pt.BatchRetrieve(index, wmodel="PL2")
tfidf = pt.BatchRetrieve(index, wmodel="TF_IDF")
bm25 = pt.BatchRetrieve(index, wmodel="BM25")
#define two functions for comparing the baseline model with bm25 and pl2

#for baseline we use TF_IDF
#topic can se be set as desired
def baseline_bm25(baseline_model, model2, topic):
  result = pt.Experiment(
    [baseline_model, model2],
    dotgov_topicsqrels.get_topics(variant=topic),
    dotgov_topicsqrels.get_qrels(variant=topic),
    eval_metrics=["map"],
    names=["TF_IDF", "BM25"],
    baseline = 0)
  return result


def baseline_pl2(baseline_model, model2, topic):
  result = pt.Experiment(
    [baseline_model, model2],
    dotgov_topicsqrels.get_topics(variant=topic),
    dotgov_topicsqrels.get_qrels(variant=topic),
    eval_metrics=["map"],
    names=["TF_IDF", "PL2"],
    baseline = 0)
  return result
baseline_bm25(tfidf, bm25, "hp")
baseline_bm25(tfidf, bm25, "np")
baseline_bm25(tfidf, bm25, "td")
baseline_pl2(tfidf, pl2, "hp")
baseline_pl2(tfidf, pl2, "np")
baseline_pl2(tfidf, pl2, "td")
#define a function to return the results for the comparison of the three models
# on the different topic sets hp,np,td
def map(model1, model2, model3, topic):
  result = pt.Experiment(
    [model1, model2, model3],
    dotgov_topicsqrels.get_topics(variant=topic),
    dotgov_topicsqrels.get_qrels(variant=topic),
    eval_metrics=["map"],
    names=["TF_IDF", "PL2", "BM25"])
  return result
hp_map = map(tfidf, pl2, bm25, "hp")
hp_map
np_map = map(tfidf, pl2, bm25, "np")
np_map
td_map = map(tfidf, pl2, bm25, "td")
td_map<jupyter_output><empty_output><jupyter_text>Print results of MAPs of each model on each topic set<jupyter_code>#print the average map of the three topic sets for TF_IDF
print((hp_map[hp_map['name'] == 'TF_IDF']['map'] + 
np_map[hp_map['name'] == 'TF_IDF']['map'] + 
td_map[hp_map['name'] == 'TF_IDF']['map'])/3)
#print the average map of the three topic sets for PL2
print((hp_map[hp_map['name'] == 'PL2']['map'] + 
np_map[hp_map['name'] == 'PL2']['map'] + 
td_map[hp_map['name'] == 'PL2']['map'])/3)
#print the average map of the three topic sets for BM25
print((hp_map[hp_map['name'] == 'BM25']['map'] + 
np_map[hp_map['name'] == 'BM25']['map'] + 
td_map[hp_map['name'] == 'BM25']['map'])/3)<jupyter_output>2    0.243463
Name: map, dtype: float64<jupyter_text>Q2(b) Question 2(b) is concerned with drawing interpolated recall precision curves. You will need to use the `"iprec_at_recall"` measure.*Hint*: Matplotlib has a [`savefig()`](https://chartio.com/resources/tutorials/how-to-save-a-plot-to-a-file-using-matplotlib/the-savefig-method) function for saving a PNG of a figure.<jupyter_code>#define a function to calculate the iprec_at_recall values for the three models
#on the different topic sets individually
def iprec_at_recall(model1, model2, model3, topic, metric):
  result = pt.Experiment(
    [model1, model2, model3],
    dotgov_topicsqrels.get_topics(variant=topic),
    dotgov_topicsqrels.get_qrels(variant=topic),
    eval_metrics=[metric],
    names=["TF_IDF", "PL2", "BM25"])
  return result
#hp precision-recall values for the three models
hp_recall = iprec_at_recall(tfidf, pl2, bm25, "hp", "iprec_at_recall")
hp_recall
#np precision-recall values for the three models
np_recall = iprec_at_recall(tfidf, pl2, bm25, "np", "iprec_at_recall")
np_recall
#td precision-recall values for the three models
td_recall = iprec_at_recall(tfidf, pl2, bm25, "td", "iprec_at_recall")
td_recall
import matplotlib.pyplot as plot
from google.colab import files
#set indexes for all dataframes produced to display the graphs for each topic set
hp_recall.set_index('name', inplace=True)
np_recall.set_index('name', inplace=True)
td_recall.set_index('name', inplace=True)
hp_recall.T.plot(rot=90)
np_recall.T.plot(rot=90)
td_recall.T.plot(rot=90)<jupyter_output><empty_output><jupyter_text>Q3 Query Expansion Query expansion is one of the most well-known and effective techniques for improving the effectiveness of a search engine. We'll be using Terrier's Bo1 query expansion model.See the [relevant documentation](https://pyterrier.readthedocs.io/en/latest/rewrite.htmlbo1queryexpansion) about creating a QE transformer pipeline in PyTerrier using the Bo1 model.You now have enough information to conduct Q3(a) and (b).<jupyter_code>#Rewrite the bo1 query expansion on the index
bo1 = pt.rewrite.Bo1QueryExpansion(index)

#Pass in the pipelines of the query expansion on all models
pipelineQE_PL2 = pl2 >> bo1 >> pl2
pipelineQE_TFIDF = tfidf >> bo1 >> tfidf
pipelineQE_BM25 = bm25 >> bo1 >> bm25
#Print the MAP of the PL2 query expanded with hp variant
pt.Experiment(
    [pl2, pipelineQE_PL2],
    dotgov_topicsqrels.get_topics(variant="hp"),
    dotgov_topicsqrels.get_qrels(variant="hp"),
    eval_metrics=["map"],
    names=["PL2", "PL2_QE"],
    baseline = 0)
#Print the MAP of the pl2 query expanded with np variant
pt.Experiment(
    [pl2, pipelineQE_PL2],
    dotgov_topicsqrels.get_topics(variant="np"),
    dotgov_topicsqrels.get_qrels(variant="np"),
    eval_metrics=["map"],
    names=["PL2", "PL2_QE"],
    baseline = 0)
#Print the MAP of the pl2 query expanded with td variant
pt.Experiment(
    [pl2, pipelineQE_PL2],
    dotgov_topicsqrels.get_topics(variant="td"),
    dotgov_topicsqrels.get_qrels(variant="td"),
    eval_metrics=["map"],
    names=["PL2", "PL2_QE"],
    baseline = 0)<jupyter_output><empty_output><jupyter_text>Q3(b)This question calls for a query-delta bar chart (See example in Lecture 5). You will need to use the ` perquery=True` option for `pt.Experiment()`. *Hints*:  - You may need a [self-join](https://www.w3schools.com/sql/sql_join_self.asp) on a dataframe. - You can iterate through a dataframe using [`dataframe.iterrows()`](https://cmdlinetips.com/2018/12/how-to-loop-through-pandas-rows-or-how-to-iterate-over-pandas-rows/)<jupyter_code>#create an experiment dataframe to produce the map values of
#pl2 on td as per instruction per query on the query set
pl2_df = pt.Experiment(
    [pl2],
    dotgov_topicsqrels.get_topics(variant="td"),
    dotgov_topicsqrels.get_qrels(variant="td"),
    eval_metrics=["map"],
    names=["PL2"],
    perquery=True)
#create the same experiment dataframe as above to produce the map
#values of the pl2 on the td as per instruction per query after the query expansion is applied
pl2_qe_df = pt.Experiment(
    [pipelineQE_PL2],
    dotgov_topicsqrels.get_topics(variant="td"),
    dotgov_topicsqrels.get_qrels(variant="td"),
    eval_metrics=["map"],
    names=["BR(PL2))"],
    perquery=True)
#create a new column from the QE dataframe called 
#values_QE on the dataframe and join it on the initial dataframe
values = pl2_qe_df['value']
pl2_df = pl2_df.join(values, rsuffix='_QE')
#display the new resulting dataframe
pl2_df
#subtract the old map values per query from the new ones to see +ve or -ve imporvement
pl2_df['delta_value'] = pl2_df['value_QE'] - pl2_df['value']
#display the final df
pl2_df
#take the absolut of all the delta AP values on the dataframe which are >0.02 (2%)
pl2_df[pl2_df['delta_value'].abs() > 0.02]
#get the topics with queries
topics_td = dotgov_topicsqrels.get_topics(variant="td")
import numpy as np

#create a list with the query qid matched to their texts to use for labelling on
#on the chart
query_texts = np.array(pt.get_dataset("trec-wt-2004").get_topics("td")["query"])
query_qid = np.array(pt.get_dataset("trec-wt-2004").get_topics("td")["qid"])

qid_texts = []
for each in range(0,len(query_texts)):
  qid_texts.append(query_qid[each] + ":" + query_texts[each])

print(qid_texts)
#visualise the dataframe
pl2_df['qid_texts'] = qid_texts
pl2_df
#create the chart of the affected queries with all delta AP values >0.02
pl2_df[pl2_df['delta_value'].abs() > 0.02].sort_values('delta_value', 
                                                           ascending=False).plot.bar(x="qid_texts", 
                                                                                     y="delta_value", 
                                                                                     rot=90, 
                                                                                     title="delta-bar chart")
#most negatively affected query
topics_td[topics_td['qid'] == '74']
#most positively affected query
topics_td[topics_td['qid'] == '97']

pipelineQE_PL2.search("ufos")['query'][0]
pipelineQE_PL2.search("skin cancer")['query'][0]<jupyter_output><empty_output><jupyter_text>Q4 Word EmbeddingsQ4 asks for a word2vec-based query expansion model. Word2vec (also called word embeddings) is a shallow neural network where semantically similar words end up with similar embedding vectors.If you haven't taken Text-as-Data, you can do some background reading about word embeddings at: - https://towardsdatascience.com/introduction-to-word-embedding-and-word2vec-652d0c2060fa - https://en.wikipedia.org/wiki/Word2vec - https://en.wikipedia.org/wiki/Word_embeddingIn general, while word2vec is still a very widely used model, note that it has been surpassed by more complex models such as BERT. But word2vec is still useful to consider in the context of query expansion. In this part of the exercise, we will use [Gensim](https://radimrehurek.com/gensim/auto_examples/tutorials/run_word2vec.html), a Python toolkit for working with a word2vec model. We are providing a pre-trained word2vec model that Gensim will download and open - the file is very large, so this might take a few minutes to download and a couple of minutes to load. You can read on while it opens.<jupyter_code>import gensim.downloader as api
%time model = api.load("glove-wiki-gigaword-300")<jupyter_output>CPU times: user 1min 55s, sys: 1.15 s, total: 1min 56s
Wall time: 1min 56s<jupyter_text>`model` is of type [gensim.models.keyedvectors.KeyedVectors](https://radimrehurek.com/gensim/models/keyedvectors.htmlgensim.models.keyedvectors.KeyedVectors).You can think of this as a dictionary mapping string words to the vector embeddings for each word.  For example, we can get the vector for the word `'government'` as follows:<jupyter_code>emb = model.get_vector("government")
print(emb.shape)
print(emb)<jupyter_output>(300,)
[ 2.5470e-02 -1.0216e-01 -2.8455e-01  2.8603e-01  1.9018e-01 -4.7529e-01
  1.4468e-01  1.9805e-01 -4.2931e-02 -2.9325e+00 -6.3137e-01  9.7630e-02
  1.0803e-01  5.4072e-01 -1.1834e-02  6.8399e-01  3.7241e-02 -3.8651e-01
  5.1331e-01 -4.0682e-01  3.3802e-01  3.9780e-02  1.0934e+00 -2.6993e-02
 -3.2652e-01  1.8426e-01  2.5084e-02 -1.9201e-02 -1.9761e-01  4.9951e-01
 -2.3426e-01 -1.1926e-01  4.3991e-02  3.2443e-01 -2.6306e-01 -1.7991e-01
  2.3484e-01  3.4124e-01 -3.1304e-01 -1.9777e-01 -1.9651e-01  2.8835e-01
 -1.6815e-01  3.8187e-01 -1.6562e-01 -4.0849e-01 -3.9998e-01  2.6185e-01
 -5.3972e-01  2.4353e-01  1.3366e-01 -3.0213e-01  3.7712e-01  1.8414e-01
  7.4082e-03  1.4511e-01  3.2574e-01  1.6853e-01 -1.0872e-01  1.9417e-02
 -3.2784e-01  3.9864e-01 -7.8588e-01  2.3249e-01  2.8092e-02 -4.3737e-01
  1.3954e-01 -4.0254e-01 -3.2712e-01  4.1180e-01  1.3976e-01  6.8819e-01
  3.9624e-02  2.7002e-02  2.3729e-01 -6.6964e-02  5.5686e-02  2.4415e-01
  1.2575e-03  5.1176e-01  1.3287e-01 -3.3563[...]<jupyter_text>As you can see, each word is represented by a 300-dimension vector.We can also ask `model` for the most similar words to `'government'` using [`model.most_similar()`](https://radimrehurek.com/gensim/models/keyedvectors.htmlgensim.models.keyedvectors.KeyedVectors.most_similar). It returns the 10 most similar words, based on the cosine similarity of their emebddings to that of `'government'`. See also: [Example in Gensim documentation](https://radimrehurek.com/gensim/models/keyedvectors.htmlwhat-can-i-do-with-word-vectors).<jupyter_code>model.most_similar("government")<jupyter_output><empty_output><jupyter_text>As you can see, some words are clearly related to the original word `'government'`, including some lexical variations (`'governments'`), as well as semantically similar (`"authorities"`) words. You can also see some words that perhaps seem unrelated - probably they are highly weighted because they appeared in similar contexts to `"government"` (e.g. `"saying"`). Q4 Word2Vec-based Query ExpansionThe next task is to use `model` to develop your custom transformer for a word2vec-based query expansion.*Hints*: - Inspired by Pandas, PyTerrier has the notion of [apply functions](https://pyterrier.readthedocs.io/en/latest/apply.html) for applying transformations. - What to do with out-of-vocabulary (OOV) words? - How many similar terms to identify for each existing query term?   - What is the relative importance of these new terms compared to the existing query terms? e.g. you might want to give more emphasis to the original query terms.  - How/if to integrate the Word2Vec cosine distance into your weighting formula?  - How to deal with special characters not recognised by the default Terrier query parser, causing a QueryParserException (e.g `/`)?You now have sufficient information to make a start on Q4.<jupyter_code>tf = pt.BatchRetrieve(index, wmodel="Tf")

#create a list to put all the special characters in 
special_characters = []

#for each token in the vocabulary take only the tokens which are greater that 1 character in length
for token in model.vocab:
  if len(token) > 1: continue

  #when finding a special character raise an error and put the character in the list
  try:
    query = pd.DataFrame([
    ["1", token]
    ], columns = ["qid", "query"])
    tf(query)
  except:
    special_characters.append(token)


#add the . and , in the list
special_characters.extend(['.', ','])

print(special_characters)
#create a function to check if the term of the query has a special character in it
#return True if yes Flase if not
def containsSpecialCharacter(term, special_characters):
   for char in special_characters:
        if char in term:
             return True
   return False

#create a query expanded word to vec model
def w2vQE(query):

  #split the query on spaces
  splitted_query_terms = query["query"].split(" ")
  similar_words_dict = dict()
  final_query = list()

  #check the similarity of each query term from the splitted list
  #get the 5 most similar terms
  for qt in splitted_query_terms:
    try:
      similar_qt = model.most_similar(qt, topn=5)
    except KeyError:
      continue

    #for each term in the most similar terms check if the it contains a special
    #character and create an entry in the most similar words dictionary
    for st in similar_qt:
      term, similarity = st
      if not containsSpecialCharacter(term, special_characters) and term not in splitted_query_terms:
        similar_words_dict[term] = max(similar_words_dict.get(term, 0), similarity)


  #sort the dictionary for the most similar terms
  most_similart_qt = sorted(similar_words_dict, key=similar_words_dict.get, reverse=True)[:4]

  #extend the final query by giving a weight of 8 in each term
  final_query.extend([ (term, 8) for term in splitted_query_terms ])
  
  #extend the query with the most similar terms for each query term 
  final_query.extend([ (term, similar_words_dict.get(term)) for term in most_similart_qt ])
  final_query = " ".join(f"{term}^{weight}" for term, weight in final_query)
  print(final_query)

  return final_query
#create the pipeline for the baseline model and w2v expanded model applied on the baseline
pipeline_baseline_model = pt.BatchRetrieve(index, wmodel="PL2")
pipeline_w2v_QE = pt.apply.query(w2vQE) >> pt.BatchRetrieve(index, wmodel="PL2")

#produce the evaluation
evaluation = pt.Experiment(
    [pipeline_baseline_model, pipeline_w2v_QE],
    dotgov_topicsqrels.get_topics(variant="td"),
    dotgov_topicsqrels.get_qrels(variant="td"),
    eval_metrics = ["map"],
    names = ["baseline PL2", "w2v_QE"],
    baseline = 0)

display(evaluation)
#produce the evaluation
evaluation_1 = pt.Experiment(
    [pipeline_baseline_model],
    dotgov_topicsqrels.get_topics(variant="td"),
    dotgov_topicsqrels.get_qrels(variant="td"),
    eval_metrics = ["map"],
    names = ["baseline PL2"],
    perquery = True)

#produce the evaluation
evaluation_2 = pt.Experiment(
    [pipeline_w2v_QE],
    dotgov_topicsqrels.get_topics(variant="td"),
    dotgov_topicsqrels.get_qrels(variant="td"),
    eval_metrics = ["map"],
    names = ["w2v_QE"],
    perquery = True)
display(evaluation_1)
display(evaluation_2)
new = evaluation_1['value'] - evaluation_2['value']
new.tolist()
print(sum(i < 0 for i in new))
print(sum(i > 0 for i in new))

pipeline_w2v_QE.search("ufos")['query'][0]<jupyter_output>ufos^8 extraterrestrials^0.6519498825073242 ufo^0.6218308210372925 saucers^0.5136354565620422 extraterrestrial^0.5028389692306519<jupyter_text>That's all Folks**Submission Instructions:** Complete this notebook, and answer the related questions in the Exercise 1 Quiz Instance on Moodle. As part of the Quiz, you will be asked to upload your .ipynb notebook solution and answer various corresponding questions as per the exercise specification (use File... Download .ipynb).Your responses to the Quiz along with your ipynb notebook solution must be submitted by **Wednesday 24th February 2021, 4:30pm**.<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10639628372342326, 'fraction_numerical': 0.045515454707879224, 'mean_word_length': 4.454080389768575, 'pattern_counts': {'":': 1, '<': 79, '<?xml version=': 0, '>': 98, 'https://': 17, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 16, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '226530', 'license': 'MIT', 'n_tokens_mistral': 8248, 'n_tokens_neox': 7517, 'n_words': 2193, 'path': 'IR_Exercise_1_2210049p.ipynb', 'repo_name': 'podyssea/InformationRetrieval', 'size': '22380'}