<jupyter_start><jupyter_text>Neural search with Transformers What are we going to do?Instead of searching text by compareing characters and words, we will use the power of transfomer models and compare texts in vector sprace.![](https://media.giphy.com/media/xT0xeJpnrWC4XWblEk/giphy.gif)  installing dependencies<jupyter_code>!pip install transformers
from transformers import AutoModel, AutoTokenizer<jupyter_output><empty_output><jupyter_text>loading a model<jupyter_code>model_name = "bert-base-cased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)<jupyter_output><empty_output><jupyter_text>transforming a text to an vector<jupyter_code>inputs = tokenizer("Hello world!", return_tensors="pt")
inputs

inputs = tokenizer(["Hello world!", "funny test"], return_tensors="pt", padding=True,truncation=True)
inputs
outputs = model(**inputs)
outputs<jupyter_output><empty_output><jupyter_text>**last_hidden_state**: Sequence of hidden-states at the output of the last layer of the model.<jupyter_code>outputs["last_hidden_state"].shape<jupyter_output><empty_output><jupyter_text>**pooler_output**: Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer and a Tanh activation function. The Linear layer weights are trained from the next sentence prediction (classification) objective during pretraining.<jupyter_code>outputs["pooler_output"].shape<jupyter_output><empty_output><jupyter_text>loading data We load a data set of news headlines from german newspapers. This data set contains the headlines and the according article urls.After we loaded the data, we need to convert all headlines into vectors.<jupyter_code>!curl -O https://www2.htw-dresden.de/~guhr/dist/feeds.tsv 

!head feeds.tsv
import time
import pandas as pd
import numpy as np

feeds_df = pd.read_csv("feeds.tsv", sep='\t', header=0,encoding="utf-8")
feeds_df.drop(columns=['text'], inplace=True)
feeds_df.drop(columns=['time'], inplace=True)
feeds_df.drop(columns=['id'], inplace=True)
feeds_df.drop(columns=['Unnamed: 5'], inplace=True)
feeds_df.drop(columns=['Unnamed: 6'], inplace=True)
feeds_df.head(5)
# We want to remove the qoutes here in order to get better results.

def remove_quotes(text):
    return text[1:-1]

feeds_df["title"]=feeds_df["title"].map(remove_quotes)
feeds_df["link"]=feeds_df["link"].map(remove_quotes)
feeds_df.head(10)
# Number of entries in our data set
len(feeds_df)<jupyter_output><empty_output><jupyter_text>Processing the data<jupyter_code># since 21257 entries would take a lot of time to process, we just load
# the first 500 articles here. But you are welcome to experiment with this 
# parameter. 

titles = list(feeds_df["title"][:500])
links = list(feeds_df["link"][:500])
import torch
tokens = tokenizer(titles, return_tensors="pt",truncation=True,padding=True)
with torch.no_grad():
    headline_vectors = model(**tokens)["pooler_output"]  
headline_vectors.shape
tokens = tokenizer("Auswirkungen der Corona Pandmie", return_tensors="pt",truncation=True, padding=True)
with torch.no_grad():
  query_vector = model(**tokens)["pooler_output"]
query_vector.shape
result = torch.sum(query_vector * headline_vectors,axis=1) 
result.shape
result<jupyter_output><empty_output><jupyter_text>Ranking the results<jupyter_code>topk = 10
values, indices = torch.topk(result, topk,largest=True)
print(values,indices)
for i in range(0,topk):
  index = indices[i].item()
  value = int(values[i].item())
  print(value,titles[index],links[index])<jupyter_output>458 Viele Corona-Tote: Durchsuchungen in Altenheimen in Mailand https://www.faz.net/aktuell/gesellschaft/gesundheit/coronavirus/viele-corona-tote-durchsuchungen-in-altenheimen-in-mailand-16724777.htm
457 Coronakrise: Condor-Rettung endgültig geplatzt https://www.spiegel.de/wirtschaft/unternehmen/coronakrise-condor-rettung-endgueltig-geplatzt-a-4ebf4558-adc1-48a1-ad52-0c4f5736144a#ref=rs
454 USA: Eine Pressekonferenz, wie sie wohl nur Trump geben kann https://www.sueddeutsche.de/politik/coronavirus-usa-trump-1.487579
454 Südkorea wählt trotz Corona-Krise https://www.wienerzeitung.at/nachrichten/politik/welt/2057223-Suedkorea-waehlt-trotz-Corona-Krise.htm
454 Coronakrise: Shinzo Abe allein zu Haus https://www.sueddeutsche.de/panorama/japan-coronavirus-shinzo-abe-1.487623
454 USA: Mindestens 20 Tote durch Stürme https://www.zeit.de/gesellschaft/zeitgeschehen/2020-04/usa-stuerme-tornados-mississippi-tot
454 Corona-Berichterstattung: Angesteckt https://www.zeit.de/2020/16/coronavirus-berich[...]<jupyter_text>Let's take a look at the vector spaceDownload the two files and upload them into [Tensorflow Projector](https://projector.tensorflow.org/).<jupyter_code># export to tf projector
x_np = headline_vectors.numpy()
x_df = pd.DataFrame(x_np)
x_df.to_csv('vectors.tsv',sep="\t",index=False, header=None,encoding="utf-8")

with open('titles.tsv', 'w') as writer:
  for title in titles:
    writer.write(title[:150]+"...\n")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11971411554496725, 'fraction_numerical': 0.02342664284296208, 'mean_word_length': 7.178571428571429, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 36, 'https://': 10, 'lorem ipsum': 0, 'www.': 8, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '640676', 'license': 'MIT', 'n_tokens_mistral': 1848, 'n_tokens_neox': 1749, 'n_words': 379, 'path': 'assignments/transformer/nlp_3_neural_search.ipynb', 'repo_name': 'oliverguhr/htw-nlp-lecture', 'size': '5037'}