<jupyter_start><jupyter_text>How to download and parse TREC-COVID data> Your first step to contribute to the improvement of the [cord19 search application](https://cord19.vespa.ai/).[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/vespa-engine/pyvespa/blob/master/docs/sphinx/source/use_cases/cord19/cord19_download_parse_trec_covid.ipynb)  Download the data The files used in this section were originally found at `https://ir.nist.gov/covidSubmit/data.html`. We will download both the topics and the relevance judgements data. Do not worry about what they are just yet, we will explore them soon.<jupyter_code>!curl -fsSLO https://data.vespa.oath.cloud/blog/cord19/topics-rnd5.xml
!curl -fsSLO https://data.vespa.oath.cloud/blog/cord19/qrels-covid_d5_j0.5-5.txt<jupyter_output><empty_output><jupyter_text>Parse the data  Topics The topics file is in XML format. We can parse it and store in a dictionary called `topics`. We want to extract a `query`, a `question` and a `narrative` from each topic.<jupyter_code>import xml.etree.ElementTree as ET

topics = {}
root = ET.parse("topics-rnd5.xml").getroot()
for topic in root.findall("topic"):
    topic_number = topic.attrib["number"]
    topics[topic_number] = {}
    for query in topic.findall("query"):
        topics[topic_number]["query"] = query.text
    for question in topic.findall("question"):
        topics[topic_number]["question"] = question.text        
    for narrative in topic.findall("narrative"):
        topics[topic_number]["narrative"] = narrative.text<jupyter_output><empty_output><jupyter_text>There are a total of 50 topics. For example, we can see the first topic below:<jupyter_code>topics["1"]<jupyter_output><empty_output><jupyter_text>Each topic has many relevance judgements associated with them.  Relevance judgements We can load the relevance judgement data directly into a pandas `DataFrame`.<jupyter_code>import pandas as pd

relevance_data = pd.read_csv("qrels-covid_d5_j0.5-5.txt", sep=" ", header=None)
relevance_data.columns = ["topic_id", "round_id", "cord_uid", "relevancy"]<jupyter_output><empty_output><jupyter_text>The relevance data contains all the relevance judgements made throughout the 5 rounds of the competition. `relevancy` equals to 0 is irrelevant, 1 is relevant and 2 is highly relevant.<jupyter_code>relevance_data.head()<jupyter_output><empty_output><jupyter_text>We are going to remove two rows that have relevancy equal to -1, which I am assuming is an error.<jupyter_code>relevance_data[relevance_data.relevancy == -1]
relevance_data = relevance_data[relevance_data.relevancy >= 0]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10958395245170877, 'fraction_numerical': 0.01188707280832095, 'mean_word_length': 5.616707616707616, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 27, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 3}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '170354', 'license': 'Apache-2.0', 'n_tokens_mistral': 899, 'n_tokens_neox': 869, 'n_words': 233, 'path': 'docs/sphinx/source/use_cases/cord19/cord19_download_parse_trec_covid.ipynb', 'repo_name': 'vespa-engine/pyvespa-documentation', 'size': '2692'}