<jupyter_start><jupyter_text>Setup an Elasticsearch Instance in Google Colab Everthing to connect to Elasticsearch, for detailed explaination see [this Notebook.](https://)Download:<jupyter_code>import os
from subprocess import Popen, PIPE, STDOUT
# download elasticsearch
!wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.9.1-linux-x86_64.tar.gz -q
!tar -xzf elasticsearch-7.9.1-linux-x86_64.tar.gz
!chown -R daemon:daemon elasticsearch-7.9.1<jupyter_output><empty_output><jupyter_text>Start a local server:<jupyter_code># start server
es_server = Popen(['elasticsearch-7.9.1/bin/elasticsearch'], 
                  stdout=PIPE, stderr=STDOUT,
                  preexec_fn=lambda: os.setuid(1)  # as daemon
                 )
# client-side
!pip install elasticsearch -q
from elasticsearch import Elasticsearch
from datetime import datetime
es = Elasticsearch(["localhost:9200/"])
#wait a bit
import time
time.sleep(30)
es.ping()  # got True<jupyter_output>[?25l
[K     |█▌                              | 10kB 3.6MB/s eta 0:00:01
[K     |███                             | 20kB 1.8MB/s eta 0:00:01
[K     |████▌                           | 30kB 2.0MB/s eta 0:00:01
[K     |██████                          | 40kB 2.0MB/s eta 0:00:01
[K     |███████▌                        | 51kB 1.6MB/s eta 0:00:01
[K     |█████████                       | 61kB 1.9MB/s eta 0:00:01
[K     |██████████▌                     | 71kB 2.2MB/s eta 0:00:01
[K     |████████████                    | 81kB 2.2MB/s eta 0:00:01
[K     |█████████████▌                  | 92kB 2.3MB/s eta 0:00:01
[K     |███████████████                 | 102kB 2.4MB/s eta 0:00:01
[K     |████████████████▌               | 112kB 2.4MB/s eta 0:00:01
[K     |██████████████████              | 122kB 2.4MB/s eta 0:00:01
[K     |███████████████████▍            | 133kB 2.4MB/s eta 0:00:01
[K     |█████████████████████           | 143kB 2.4MB/s eta 0:00:01
[K     |██████████████████████▍    [...]<jupyter_text>Process DataGet different corpora, format them and feed them to elasticsearch  ADI CorpusYou can get the corpus from [this link](http://ir.dcs.gla.ac.uk/resources/test_collections/adi/).    ParsingFor detailed information about the parsing of this corpus look at [ this Notebook](https://colab.research.google.com/github/pragmalingu/experiments/blob/master/00_Data/ADICorpus.ipynb) or for parsing in generel read [this guide](https://pragmalingu.de/docs/guides/how-to-parse). An overview of the format of the files can be found here: [Data Sets](https://pragmalingu.de/docs/guides/data-comparison)<jupyter_code># download and unzip data
!wget http://ir.dcs.gla.ac.uk/resources/test_collections/adi/adi.tar.gz
!tar -xf adi.tar.gz

# set paths to the dowloaded data as variables
PATH_TO_ADI_TXT = '/content/ADI.ALL'

from collections import defaultdict
import re
import json
from io import StringIO
import numpy as np

# get the text and query files

ID_marker = re.compile('\.I')

def get_data(PATH_TO_FILE, marker):
  """
  Reads file and spilts text into entries at the ID marker '.I'.
  First entry is empty, so it's removed.
  'marker' contains the regex at which we want to split
  """
  with open (PATH_TO_FILE,'r') as f:
    text = f.read().replace('\n'," ")
    lines = re.split(marker,text)
    lines.pop(0)
  return lines

adi_txt_list = get_data(PATH_TO_ADI_TXT, ID_marker)

# process text file

adi_title_start = re.compile('\.T')
adi_author_start = re.compile('\.A')
adi_text_start = re.compile('\.W')

adi_txt_data = defaultdict(dict)

for line in adi_txt_list:
  entries = re.split(adi_title_start,line,1)
  id = entries[0].strip()
  no_id = entries[1]
  if len(re.split(adi_author_start, no_id,1)) > 1:
    no_id_entries = re.split(adi_author_start, no_id,1)
    adi_txt_data[id]['title'] = no_id_entries[0]
    no_title = no_id_entries[1]
    no_title_entries = re.split(adi_text_start, no_title)
    adi_txt_data[id]['author'] = no_title_entries[0]
    adi_txt_data[id]['text'] = no_title_entries[1]
  else:
    no_id_entries = re.split(adi_text_start, no_id)
    adi_txt_data[id]['title'] = no_id_entries[0]
    adi_txt_data[id]['text'] = no_id_entries[1]<jupyter_output><empty_output><jupyter_text>IndexingCreate an index for the adi corpus and index all the documents. This is only possible if it isn't created yet.(For more information see the [Elasticsearch documentation](https://elasticsearch-py.readthedocs.io/en/master/api.htmlelasticsearch.client.IndicesClient.create))<jupyter_code>#create index, see https://elasticsearch-py.readthedocs.io/en/master/api.html#elasticsearch.client.IndicesClient.create
adi_index = "adi-corpus"
es.indices.create(adi_index)
#index documents, see https://elasticsearch-py.readthedocs.io/en/master/#example-usage
for ID, doc_data in adi_txt_data.items():
  es.index(index=adi_index, id=ID, body=doc_data)
#print new index list
create_response = es.cat.indices()
print(create_response)<jupyter_output><empty_output><jupyter_text>CACM CorpusYou can get the corpus from [this link](http://ir.dcs.gla.ac.uk/resources/test_collections/cacm/).    ParsingFor detailed information about the parsing of this corpus look at [ this Notebook](https://colab.research.google.com/github/pragmalingu/experiments/blob/master/00_Data/CACMCorpus.ipynb) or for parsing in generel read [this guide](https://pragmalingu.de/docs/guides/how-to-parse). An overview of the format of the files can be found here: [Data Sets](https://pragmalingu.de/docs/guides/data-comparison)<jupyter_code># download and unzip data
!wget http://ir.dcs.gla.ac.uk/resources/test_collections/cacm/cacm.tar.gz
!tar -xf cacm.tar.gz

# set paths to the dowloaded data as variablesDownload and unzip data.

PATH_TO_CACM_TXT = '/content/cacm.all'

from collections import defaultdict
import re
import json
from io import StringIO
import numpy as np

# get the text and query files

ID_marker = re.compile('^\.I',re.MULTILINE)

def get_data(PATH_TO_FILE, marker):
  """
  Reads file and spilts text into entries at the ID marker '.I'.
  First entry is empty, so it's removed.
  'marker' contains the regex at which we want to split
  """
  with open (PATH_TO_FILE,'r') as f:
    text = f.read()
    lines = re.split(marker,text)
    lines.pop(0)
  return lines

cacm_txt_list = get_data(PATH_TO_CACM_TXT, ID_marker)

# process text file

cacm_chunk_title = re.compile('\.[T]\n')
cacm_chunk_markers = re.compile('^\.', re.MULTILINE)
cacm_chunk_txt = re.compile('W\n')
cacm_chunk_pub = re.compile('B\n')
cacm_chunk_author =re.compile('A\n')
cacm_chunk_add = re.compile('N\n')
cacm_chunk_cross = re.compile('X\n')

cacm_txt_data = defaultdict(dict)

for line in cacm_txt_list:
    entries = re.split(cacm_chunk_title, line)
    id = entries[0].strip()  # save id
    no_id = entries[1]
    list_of_entries = re.split(cacm_chunk_markers, no_id)
    cacm_txt_data[id]['title'] = list_of_entries[0].strip()
    for entry in list_of_entries:
        if len(re.split(cacm_chunk_txt, entry)) == 2: # check for text
            cacm_txt_data[id]['text'] = re.split(cacm_chunk_txt, entry)[1]
        if len(re.split(cacm_chunk_pub, entry)) == 2: # check for publication date
          cacm_txt_data[id]['publication_date'] = re.split(cacm_chunk_pub, entry)[1]
        if len(re.split(cacm_chunk_author, entry)) == 2: # check for author
            cacm_txt_data[id]['author'] = re.split(cacm_chunk_author, entry)[1]
        if len(re.split(cacm_chunk_add, entry)) == 2: # check for additional date
            cacm_txt_data[id]['add_date'] = re.split(cacm_chunk_add, entry)[1]
        if len(re.split(cacm_chunk_cross, entry)) == 2:  # check for cross-references
            cacm_txt_data[id]['cross-references'] = re.split(cacm_chunk_cross, entry)[1]<jupyter_output><empty_output><jupyter_text>IndexingCreate an index for the CACM corpus and index all the documents. This is only possible if it isn't created yet.(For more information see the [Elasticsearch documentation](https://elasticsearch-py.readthedocs.io/en/master/api.htmlelasticsearch.client.IndicesClient.create))<jupyter_code>#create index, see https://elasticsearch-py.readthedocs.io/en/master/api.html#elasticsearch.client.IndicesClient.create
cacm_index = "cacm-corpus"
es.indices.create(cacm_index)
#index documents, see https://elasticsearch-py.readthedocs.io/en/master/#example-usage
for ID, doc_data in cacm_txt_data.items():
  es.index(index=cacm_index, id=ID, body=doc_data)
#print new index list
create_response = es.cat.indices()
print(create_response)<jupyter_output><empty_output><jupyter_text>CISI CorpusYou can get the corpus from [this link](http://ir.dcs.gla.ac.uk/resources/test_collections/cisi/).    ParsingFor detailed information about the parsing of this corpus look at [ this Notebook](https://colab.research.google.com/github/pragmalingu/experiments/blob/master/00_Data/CISICorpus.ipynb) or for parsing in generel read [this guide](https://pragmalingu.de/docs/guides/how-to-parse). An overview of the format of the files can be found here: [Data Sets](https://pragmalingu.de/docs/guides/data-comparison)<jupyter_code># download and unzip data
!wget http://ir.dcs.gla.ac.uk/resources/test_collections/cisi/cisi.tar.gz
!tar -xf cisi.tar.gz

# set paths to the dowloaded data as variablesDownload and unzip data.
PATH_TO_CISI_TXT = '/content/CISI.ALL'

from collections import defaultdict
import re
import json
from io import StringIO
import numpy as np

# get the text file

ID_marker = re.compile('^\.I',re.MULTILINE)

def get_data(PATH_TO_FILE, marker):
  """
  Reads file and spilts text into entries at the ID marker '.I'.
  First entry is empty, so it's removed.
  'marker' contains the regex at which we want to split
  """
  with open (PATH_TO_FILE,'r') as f:
    text = f.read()
    lines = re.split(marker,text)
    lines.pop(0)
  return lines

cisi_txt_list = get_data(PATH_TO_CISI_TXT, ID_marker)

# process text file

cisi_title_start = re.compile('[\n]\.T')
cisi_author_start = re.compile('[\n]\.A')
cisi_date_start = re.compile('[\n]\.B')
cisi_text_start = re.compile('[\n]\.W')
cisi_cross_start = re.compile('[\n]\.X')

cisi_txt_data = defaultdict(dict)

for line in cisi_txt_list:
  entries = re.split(cisi_title_start,line,1)
  id = entries[0].strip()#save the id
  no_id = entries[1] 
  
  if len(re.split(cisi_author_start, no_id)) >= 2: # is there just one author?
    no_id_entries = re.split(cisi_author_start, no_id,1)
    cisi_txt_data[id]['title'] = no_id_entries[0].strip() # save title
    no_title = no_id_entries[1]

    if len(re.split(cisi_date_start, no_title)) > 1: # is there a publication date?
      no_title_entries = re.split(cisi_date_start, no_title)
      cisi_txt_data[id]['author'] = no_title_entries[0].strip() # save athour
      no_author = no_title_entries[1]
      no_author_entries = re.split(cisi_text_start, no_author)
      cisi_txt_data[id]['publication_date'] = no_author_entries[0].strip() # save publication date
      no_author_date = no_author_entries[1]
    else:
      no_title_entries = re.split(cisi_text_start, no_title)
      cisi_txt_data[id]['author'] = no_title_entries[0].strip() # save athour
      no_author_date = no_title_entries[1]

  else:
    no_id_entries = re.split(cisi_author_start, no_id)
    cisi_txt_data[id]['title'] = no_id_entries[0].strip() # save title
    cisi_txt_data[id]['author'] = no_id_entries[1].strip() # save first author
    no_title_entries = re.split(cisi_text_start, no_title)
    cisi_txt_data[id]['author'] += ','+no_title_entries[0].strip() # save second athour
    no_author_date = no_title_entries[1]

  last_entries = re.split(cisi_cross_start, no_author_date)
  cisi_txt_data[id]['text'] = last_entries[0].strip() # save text
  cisi_txt_data[id]['cross-refrences'] = last_entries[1].strip() # save cross refrences<jupyter_output><empty_output><jupyter_text>IndexingCreate an index for the CISI corpus and index all the documents. This is only possible if it isn't created yet.(For more information see the [Elasticsearch documentation](https://elasticsearch-py.readthedocs.io/en/master/api.htmlelasticsearch.client.IndicesClient.create))<jupyter_code>#create index, see https://elasticsearch-py.readthedocs.io/en/master/api.html#elasticsearch.client.IndicesClient.create
cisi_index = "cisi-corpus"
es.indices.create(cisi_index)
#index documents, see https://elasticsearch-py.readthedocs.io/en/master/#example-usage
for ID, doc_data in cisi_txt_data.items():
  es.index(index=cisi_index, id=ID, body=doc_data)
#print new index list
create_response = es.cat.indices()
print(create_response)<jupyter_output><empty_output><jupyter_text>Cranfield CorpusYou can get the corpus from [this link](http://ir.dcs.gla.ac.uk/resources/test_collections/cran/).    ParsingFor detailed information about the parsing of this corpus look at [ this Notebook](https://colab.research.google.com/github/pragmalingu/experiments/blob/master/00_Data/CranfieldCorpus.ipynb) or for parsing in generel read [this guide](https://pragmalingu.de/docs/guides/how-to-parse). An overview of the format of the files can be found here: [Data Sets](https://pragmalingu.de/docs/guides/data-comparison)<jupyter_code># download and unzip data
!wget http://ir.dcs.gla.ac.uk/resources/test_collections/cran/cran.tar.gz
!tar -xf cran.tar.gz

# set paths to the dowloaded data as variables
PATH_TO_CRAN_TXT = '/content/cran.all.1400'

from collections import defaultdict
import re
import json
from io import StringIO
import numpy as np

# get the text entries from the text and query files

ID_marker = re.compile('\.I')

def get_data(PATH_TO_FILE, marker):
  """
  Reads file and spilts text into entries at the ID marker '.I'.
  First entry is empty, so it's removed.
  'marker' contains the regex at which we want to split
  """
  with open (PATH_TO_FILE,'r') as f:
    text = f.read().replace('\n'," ")
    lines = re.split(marker,text)
    lines.pop(0)
  return lines

cran_txt_list = get_data(PATH_TO_CRAN_TXT, ID_marker)

# process text file

cran_chunk_start = re.compile('\.[A,B,T,W]')
cran_txt_data = defaultdict(dict)

for line in cran_txt_list:
  entries= re.split(cran_chunk_start,line)
  id = entries[0].strip()
  title = entries[1]
  author = entries[2]
  publication_date = entries[3]
  text = entries[4:]
  cran_txt_data[id]['title'] = ''.join(title)
  cran_txt_data[id]['author'] = author
  cran_txt_data[id]['publication_date'] = publication_date
  cran_txt_data[id]['text'] = ''.join(text)<jupyter_output>--2020-09-16 06:31:35--  http://ir.dcs.gla.ac.uk/resources/test_collections/cran/cran.tar.gz
Resolving ir.dcs.gla.ac.uk (ir.dcs.gla.ac.uk)... 172.16.58.3
Connecting to ir.dcs.gla.ac.uk (ir.dcs.gla.ac.uk)|172.16.58.3|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 506960 (495K) [application/gzip]
Saving to: ‘cran.tar.gz’

cran.tar.gz         100%[===================>] 495.08K  1004KB/s    in 0.5s    

2020-09-16 06:31:36 (1004 KB/s) - ‘cran.tar.gz’ saved [506960/506960]<jupyter_text>IndexingCreate an index for the Cranfield corpus and index all the documents. This is only possible if it isn't created yet.(For more information see the [Elasticsearch documentation](https://elasticsearch-py.readthedocs.io/en/master/api.htmlelasticsearch.client.IndicesClient.create))<jupyter_code>#create index, see https://elasticsearch-py.readthedocs.io/en/master/api.html#elasticsearch.client.IndicesClient.create
cran_index = "cranfield-corpus"
es.indices.create(cran_index)
#index documents, see https://elasticsearch-py.readthedocs.io/en/master/#example-usage
for ID, doc_data in cran_txt_data.items():
  es.index(index=cran_index, id=ID, body=doc_data)
#print new index list
create_response = es.cat.indices()
print(create_response)<jupyter_output><empty_output><jupyter_text>LISA CorpusYou can get the corpus from [this link](http://ir.dcs.gla.ac.uk/resources/test_collections/lisa/).    ParsingFor detailed information about the parsing of this corpus look at [ this Notebook](https://colab.research.google.com/github/pragmalingu/experiments/blob/master/00_Data/LISACorpus.ipynb) or for parsing in generel read [this guide](https://pragmalingu.de/docs/guides/how-to-parse). An overview of the format of the files can be found here: [Data Sets](https://pragmalingu.de/docs/guides/data-comparison)<jupyter_code># download and unzip data
!wget http://ir.dcs.gla.ac.uk/resources/test_collections/lisa/lisa.tar.gz
!tar -xf lisa.tar.gz

# set paths to the dowloaded data as variablesDownload and unzip data.

PATH_TO_LISA_TXT = '/content/'

from collections import defaultdict
import re
import json
from io import StringIO
import numpy as np
import os

# get the text files

file_regex = re.compile('LISA[0-5]')
lisa_files = [i for i in os.listdir(PATH_TO_LISA_TXT) if os.path.isfile(os.path.join(PATH_TO_LISA_TXT,i)) and re.match(file_regex,i)]

txt_entry_marker = re.compile('\*{44}',re.MULTILINE)

def get_data(PATH_TO_FILES, marker):
  """
  Reads multiple files and spilts text into entries at the entry marker.
  The 'marker' contains the regex at which we want to split
  Pops last element since it's empty.
  """
  with open (PATH_TO_FILES,'r') as f:
    text = f.read().replace('     ','')
    lines = re.split(marker,text)
    lines.pop()
  return lines

lisa_txt_list = []
for name in lisa_files: 
  lisa_txt_list.extend(get_data(PATH_TO_LISA_TXT+name, txt_entry_marker))

# process text file

doc_strip = re.compile('\n?Document {1,2}')

lisa_txt_list_stripped = []
lisa_txt_data = defaultdict(dict)

for el in lisa_txt_list:
  lisa_txt_list_stripped.append(re.sub(doc_strip,'', el))

for entry in lisa_txt_list_stripped:
  parts = entry.split('\n')
  empty_index = parts.index('')
  ID = parts[0].strip(' ')
  title = parts[1:empty_index]
  text = parts[empty_index+1:]
  lisa_txt_data[ID]['title'] = ''.join(title)
  lisa_txt_data[ID]['text'] = ''.join(text)<jupyter_output>--2020-10-06 08:42:24--  http://ir.dcs.gla.ac.uk/resources/test_collections/lisa/lisa.tar.gz
Resolving ir.dcs.gla.ac.uk (ir.dcs.gla.ac.uk)... 172.16.58.3
Connecting to ir.dcs.gla.ac.uk (ir.dcs.gla.ac.uk)|172.16.58.3|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1227725 (1.2M) [application/gzip]
Saving to: ‘lisa.tar.gz’

lisa.tar.gz         100%[===================>]   1.17M  1.53MB/s    in 0.8s    

2020-10-06 08:42:25 (1.53 MB/s) - ‘lisa.tar.gz’ saved [1227725/1227725]<jupyter_text>IndexingCreate an index for the Lisa corpus and index all the documents. This is only possible if it isn't created yet.(For more information see the [Elasticsearch documentation](https://elasticsearch-py.readthedocs.io/en/master/api.htmlelasticsearch.client.IndicesClient.create))<jupyter_code>#create index, see https://elasticsearch-py.readthedocs.io/en/master/api.html#elasticsearch.client.IndicesClient.create
lisa_index = "lisa-corpus"
es.indices.create(lisa_index)
#index documents, see https://elasticsearch-py.readthedocs.io/en/master/#example-usage
for ID, doc_data in lisa_txt_data.items():
  es.index(index=lisa_index, id=ID, body=doc_data)
#print new index list
create_response = es.cat.indices()
print(create_response)<jupyter_output>green  open hunspell-cisi-corpus                  EnuNB6nqRBe8F50JqJkLrA 1 0  1460    0   2.6mb   2.6mb
yellow open security-auditlog-2020.07.26          3KQWvUwlRHOYj3AeOYcfkg 1 1    31    0  75.6kb  75.6kb
yellow open security-auditlog-2020.07.27          C3A4S_lhSC-y5NEvhhhAAg 1 1    20    0  36.9kb  36.9kb
yellow open security-auditlog-2020.07.28          sLtcImDYTmafKHDoxhmg8g 1 1    22    0  72.3kb  72.3kb
yellow open bert-text-title-cacm-corpus           yjXirDDnRMatjQAG6dKfvA 1 1  1586    0  57.7mb  57.7mb
yellow open security-auditlog-2020.07.29          s_wFvB3PTeKWfVSZFdlgOw 1 1     9    0 123.3kb 123.3kb
green  open snowball-medline-corpus               OpgXBcIrRnma8hWRj3YeSg 1 0  1032    0     1mb     1mb
yellow open bert-text-adi-corpus                  3IGPlx4iRaODZ2R8vNfKzA 1 1    82    0   1.4mb   1.4mb
green  open stemmer-cacm-corpus                   xDyo3niKQ2GPnqOwqeB1YQ 1 0  3204    0     4mb     4mb
green  open .opendistro_security                  aOOHbiN_QwKao4[...]<jupyter_text>Medline CorpusYou can get the corpus from [this link](http://ir.dcs.gla.ac.uk/resources/test_collections/med/).    ParsingFor detailed information about the parsing of this corpus look at [ this Notebook](https://colab.research.google.com/github/pragmalingu/experiments/blob/master/00_Data/MedlineCorpus.ipynb) or for parsing in generel read [this guide](https://pragmalingu.de/docs/guides/how-to-parse). An overview of the format of the files can be found here: [Data Sets](https://pragmalingu.de/docs/guides/data-comparison)<jupyter_code># download and unzip data
!wget http://ir.dcs.gla.ac.uk/resources/test_collections/medl/med.tar.gz
!tar -xf med.tar.gz

# set paths to the dowloaded data as variables
PATH_TO_MED_TXT = '/content/MED.ALL'

from collections import defaultdict
import re
import json
from io import StringIO
import numpy as np

# get the text and query files

ID_marker = re.compile('\.I')

def get_data(PATH_TO_FILE, marker):
  """
  Reads file and spilts text into entries at the ID marker '.I'.
  First entry is empty, so it's removed.
  'marker' contains the regex at which we want to split
  """
  with open (PATH_TO_FILE,'r') as f:
    text = f.read().replace('\n'," ")
    lines = re.split(marker,text)
    lines.pop(0)
  return lines

med_txt_list = get_data(PATH_TO_MED_TXT, ID_marker)

# process the text file

chunk_start = re.compile('\.W')
med_txt_data = defaultdict(dict)

def fill_dictionary(dictionary, chunk_list, marker, key_name):
  for n in range(0,len(chunk_list)-1):
    line = chunk_list[n+1]
    _ , chunk = re.split(marker,line)
    dictionary[n+1][key_name] = chunk.strip()

fill_dictionary(med_txt_data, med_txt_list, chunk_start, 'text')<jupyter_output><empty_output><jupyter_text>IndexingCreate an index for the Medline corpus and index all the documents. This is only possible if it isn't created yet.(For more information see the [Elasticsearch documentation](https://elasticsearch-py.readthedocs.io/en/master/api.htmlelasticsearch.client.IndicesClient.create))<jupyter_code>#create index, see https://elasticsearch-py.readthedocs.io/en/master/api.html#elasticsearch.client.IndicesClient.create
med_index = "medline-corpus"
es.indices.create(med_index)
#index documents, see https://elasticsearch-py.readthedocs.io/en/master/#example-usage
for ID, doc_data in med_txt_data.items():
  es.index(index=med_index, id=ID, body=doc_data)
#print new index list
create_response = es.cat.indices()
print(create_response)<jupyter_output><empty_output><jupyter_text>NPL CorpusYou can get the corpus from [this link](http://ir.dcs.gla.ac.uk/resources/test_collections/npl/).    ParsingFor detailed information about the parsing of this corpus look at [ this Notebook](https://colab.research.google.com/github/pragmalingu/experiments/blob/master/00_Data/NPLCorpus.ipynb) or for parsing in generel read [this guide](https://pragmalingu.de/docs/guides/how-to-parse). An overview of the format of the files can be found here: [Data Sets](https://pragmalingu.de/docs/guides/data-comparison)<jupyter_code># download and unzip data

!wget http://ir.dcs.gla.ac.uk/resources/test_collections/npl/npl.tar.gz
!tar -xf npl.tar.gz

# set paths to the dowloaded data as variablesDownload and unzip data.

PATH_TO_NPL_TXT = '/content/doc-text'

from collections import defaultdict
import re
import json
from io import StringIO
import numpy as np
import os


# get the text, query and rel files

txt_entry_marker = re.compile('\n   /\n')

def get_data(PATH_TO_FILES, marker):
  """
  Reads multiple files and spilts text into entries at the entry marker.
  The 'marker' contains the regex at which we want to split
  Pops last element since it's empty.
  """
  with open (PATH_TO_FILES,'r') as f:
    text = f.read()
    lines = re.split(marker,text)
    lines.pop()
  return lines

npl_txt_list = get_data(PATH_TO_NPL_TXT, txt_entry_marker)

# process the text file

npl_txt_data = defaultdict(dict)

for entry in npl_txt_list:
  splitted = entry.split('\n')
  splitted = list(filter(None, splitted))
  ID = splitted[0]
  text = ' '.join(map(str, splitted[1:]))
  npl_txt_data[ID]['text'] = text<jupyter_output><empty_output><jupyter_text>IndexingCreate an index for the NPL corpus and index all the documents. This is only possible if it isn't created yet.(For more information see the [Elasticsearch documentation](https://elasticsearch-py.readthedocs.io/en/master/api.htmlelasticsearch.client.IndicesClient.create))<jupyter_code>#create index, see https://elasticsearch-py.readthedocs.io/en/master/api.html#elasticsearch.client.IndicesClient.create
npl_index = "npl-corpus"
es.indices.create(npl_index)
#index documents, see https://elasticsearch-py.readthedocs.io/en/master/#example-usage
for ID, doc_data in npl_txt_data.items():
  es.index(index=npl_index, id=ID, body=doc_data)
#print new index list
create_response = es.cat.indices()
print(create_response)<jupyter_output><empty_output><jupyter_text>Time CorpusYou can get the corpus from [this link](http://ir.dcs.gla.ac.uk/resources/test_collections/time/).    ParsingFor detailed information about the parsing of this corpus look at [ this Notebook](https://colab.research.google.com/github/pragmalingu/experiments/blob/master/00_Data/TimeCorpus.ipynb) or for parsing in generel read [this guide](https://pragmalingu.de/docs/guides/how-to-parse). An overview of the format of the files can be found here: [Data Sets](https://pragmalingu.de/docs/guides/data-comparison)<jupyter_code># download and unzip data

!wget http://ir.dcs.gla.ac.uk/resources/test_collections/time/time.tar.gz
!tar -xf time.tar.gz

# set paths to the dowloaded data as variablesDownload and unzip data.

PATH_TO_TIME_TXT = '/content/TIME.ALL'

from collections import defaultdict
import re
import json
from io import StringIO
import numpy as np
import os

# get the text and query file

txt_entry_marker = re.compile('\*TEXT')

def get_data(PATH_TO_FILES, marker):
  """
  Reads multiple files and spilts text into entries at the entry marker.
  The 'marker' contains the regex at which we want to split
  Pops last element since it's empty.
  """
  with open (PATH_TO_FILES,'r') as f:
    text = f.read()
    lines = re.split(marker,text)
    lines.pop(0)
  return lines

time_txt_list = get_data(PATH_TO_TIME_TXT, txt_entry_marker)

# process text file

page_split = re.compile('PAGE \d{3}')

time_txt_data = defaultdict(dict)
ID = 1
for entry in time_txt_list:
  splitted = re.split(page_split,entry)
  time_txt_data[ID]['text'] = splitted[1]
  ID += 1<jupyter_output><empty_output><jupyter_text>IndexingCreate an index for the Time corpus and index all the documents. This is only possible if it isn't created yet.(For more information see the [Elasticsearch documentation](https://elasticsearch-py.readthedocs.io/en/master/api.htmlelasticsearch.client.IndicesClient.create))<jupyter_code>#create index, see https://elasticsearch-py.readthedocs.io/en/master/api.html#elasticsearch.client.IndicesClient.create
time_index = "time-corpus"
es.indices.create(time_index)
#index documents, see https://elasticsearch-py.readthedocs.io/en/master/#example-usage
for ID, doc_data in time_txt_data.items():
  es.index(index=time_index, id=ID, body=doc_data)
#print new index list
create_response = es.cat.indices()
print(create_response)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1281211327072869, 'fraction_numerical': 0.02107447040838611, 'mean_word_length': 4.368307932786244, 'pattern_counts': {'":': 0, '<': 69, '<?xml version=': 0, '>': 74, 'https://': 50, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 4, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 19, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '222793', 'license': 'MIT', 'n_tokens_mistral': 10401, 'n_tokens_neox': 9816, 'n_words': 2229, 'path': '00_Data/ParsingAndIndexingData.ipynb', 'repo_name': 'pragmalingu/experiments', 'size': '27490'}