<jupyter_start><jupyter_text>Remove stopwordsFunction to remove stopwords<jupyter_code>from nltk.tokenize import RegexpTokenizer
from nltk import word_tokenize

nltk.download('stopwords')
nltk.download("punkt")

stopwords = set(nltk.corpus.stopwords.words('english'))

def lower_remove_punctuation(text):
    text = text.lower()
    tokenizer = RegexpTokenizer(r'\w+')
    text_tokens = tokenizer.tokenize(text)
    return ' '.join(text_tokens)

def remove_stop_words(text):
    text_tokens = word_tokenize(text)
    tokens_without_sw = [w for w in text_tokens if not w in stopwords and w.isalnum()]
    return ' '.join(tokens_without_sw)<jupyter_output>[nltk_data] Downloading package stopwords to
[nltk_data]     /Users/vladislavkruglikov/nltk_data...
[nltk_data]   Package stopwords is already up-to-date!
[nltk_data] Downloading package punkt to
[nltk_data]     /Users/vladislavkruglikov/nltk_data...
[nltk_data]   Package punkt is already up-to-date!<jupyter_text>LemmatizeFunction to lemmatize text<jupyter_code>from nltk.stem import WordNetLemmatizer

lemmatizer = WordNetLemmatizer()

def lemmatize(text):
    text_tokens = word_tokenize(text)
    lemmatized_tokens = [lemmatizer.lemmatize(w) for w in text_tokens]
    return ' '.join(lemmatized_tokens)<jupyter_output><empty_output><jupyter_text>Title preprocessingRemoing stop words, and apply lemmatization<jupyter_code>title = df['Title']

df['Title preprocessed'] = title.apply(lower_remove_punctuation).apply(remove_stop_words).apply(lemmatize)<jupyter_output><empty_output><jupyter_text>Body preprocessingSince ```Body``` contains many code fragments and we don't want to use code fragments in search we need to get rid of it<jupyter_code>import re

body = df['Body']

def remove_spaces(text):
    return ' '.join(text.split())

def remove_code(text):
    # remove code part
    text = re.sub('<pre>.*<\/pre>', '', text)
    # remove tags
    text = re.sub('<a.*?>', '', text)
    text = re.sub('</a>', '', text)
    text = re.sub('<code>', '', text)
    text = re.sub('</code>', '', text)
    text = re.sub('<p>', '', text)
    text = re.sub('</p>', '', text)
    return text

df['Body preprocessed'] = body.apply(lower_remove_punctuation).apply(remove_spaces).apply(remove_code).apply(remove_stop_words).apply(lemmatize)<jupyter_output><empty_output><jupyter_text>Fit TF-IDF title vectorizerFit TF-IDF vectorizer for title column<jupyter_code>from sklearn.feature_extraction.text import TfidfVectorizer

import pickle

tfidf_title_vectorizer = TfidfVectorizer()

title_preprocessed = df['Title preprocessed']

tfidf_title_vectorizer.fit(title_preprocessed)

pickle.dump(tfidf_title_vectorizer, open('data/tfidf_title_vectorizer.sav', 'wb'))<jupyter_output><empty_output><jupyter_text>Extract keywords from TF-IDF title vectorizerGet top 10 words with highest TF-IDF from each title<jupyter_code>def get_top_tfidf_words(sentence, tfidf_vectorizer, n=5):
    terms = tfidf_vectorizer.get_feature_names()
    sums = tfidf_vectorizer.transform([sentence]).sum(axis=0)
    data = []
    for col, term in enumerate(terms):
        if sums[0, col] > 0:
            data.append((term, sums[0, col]))
    data.sort(key=lambda x: -x[1])
    return ' '.join([i[0] for i in data[:n]])

df['Title preprocessed top 10 keywords'] = title_preprocessed.apply(lambda text: get_top_tfidf_words(text, tfidf_title_vectorizer, n=5))<jupyter_output><empty_output><jupyter_text>Fit TF-IDF body vectorizerFit TF-IDF vectorizer for ```body``` column<jupyter_code>from sklearn.feature_extraction.text import TfidfVectorizer

tfidf_body_vectorizer = TfidfVectorizer()

body_preprocessed = df['Body preprocessed']

tfidf_body_vectorizer.fit(body_preprocessed)

pickle.dump(tfidf_body_vectorizer, open('data/tfidf_body_vectorizer.sav', 'wb'))<jupyter_output><empty_output><jupyter_text>Extract keywords from TF-IDF body vectorizerGet top 30 words with highest TF-IDF from each body<jupyter_code>df['Body preprocessed top 30 keywords'] = body_preprocessed.apply(lambda text: get_top_tfidf_words(text, tfidf_body_vectorizer, n=30))<jupyter_output><empty_output><jupyter_text>Get date scoreThe newest questions have higher score<jupyter_code>df['timestamp'] = pd.DatetimeIndex(df.CreationDate).asi8

date_min = df.timestamp.min()
date_max = df.timestamp.max()

def date_score(date):
    return 1 + (date - date_min) / (date_max - date_min)

df['Date score'] = df.timestamp.apply(date_score)<jupyter_output><empty_output><jupyter_text>Get Vote scoreThe questions with more votes are probably better<jupyter_code>votes = df['Score']

votes_min = votes.min()
votes_max = votes.max()

def votes_score(votes):
    return 1 + (votes - votes_min) / (votes_max - votes_min)

df['Votes score'] = df['Score'].apply(votes_score)<jupyter_output><empty_output><jupyter_text>Calculate TF-IDF vectors for title preprocessed top 10 keywordsWill use this to calculate distance between query TF-IDF vector and database documents TF-IDF vectors to find the most similar<jupyter_code>import numpy as np

from sklearn.metrics.pairwise import cosine_similarity

df['Title preprocessed top 10 keywords TF-IDF vector'] = df['Title preprocessed top 10 keywords'].apply(lambda x: tfidf_title_vectorizer.transform([x]))

def calculate_distance(vec1, vec2):
    cosine_similarities = cosine_similarity(vec1.reshape(1, -1), vec2.reshape(1, -1))
    return cosine_similarities[0][0]
# For example

query = "multiple query one sql"

calculate_distance(df['Title preprocessed top 10 keywords TF-IDF vector'][0], tfidf_title_vectorizer.transform([query]))<jupyter_output><empty_output><jupyter_text>Calculate TF-IDF vectors for body preprocessed top 30 keywordsWill use this to calculate distance between query TF-IDF vector and database documents TF-IDF vectors to find the most similar<jupyter_code>df['Body preprocessed top 30 keywords TF-IDF vector'] = df['Body preprocessed top 30 keywords'].apply(lambda x: tfidf_body_vectorizer.transform([x]))<jupyter_output><empty_output><jupyter_text>Build Inverted Index for titleFunction for creating inverted index to allow us do the first stage of search<jupyter_code>from collections import defaultdict

def build_inverted_index(data: list) -> list:
    inverted_index = defaultdict(list)
    rows = len(data)
    for idx in range(1, rows):
        sample = df.iloc[idx]
        for word in set(data[idx].split()):
            inverted_index[word].append(sample.Id)
    return inverted_index

title_inverted_index = build_inverted_index(df['Title preprocessed top 10 keywords'])
pickle.dump(title_inverted_index, open('data/title_inverted_index.sav', 'wb'))<jupyter_output><empty_output><jupyter_text>Build Inverted Index for bodyCreate different inverted index for body<jupyter_code>body_inverted_index = build_inverted_index(df['Body preprocessed top 30 keywords'])
pickle.dump(body_inverted_index, open('data/body_inverted_index.sav', 'wb'))<jupyter_output><empty_output><jupyter_text>Clear not needed colsSave space by removing columns that we don't need<jupyter_code>df.drop(columns=['CreationDate', 'timestamp', 'Score', 'Title preprocessed', 'Body preprocessed',
                 'Title preprocessed top 10 keywords', 'Body preprocessed top 30 keywords'], inplace=True)<jupyter_output><empty_output><jupyter_text>Look at the final datasetOur dataset after all transformations<jupyter_code>df<jupyter_output><empty_output><jupyter_text>Save preprocessed datasetSave dataset to pickle file, it's better to use pickle because it allows us to save ```TF-IDF vector``` scipy format of column, because pandas will force it to be string<jupyter_code>pickle.dump(df, open('data/Preprocessed Questions.sav', 'wb'))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11129892081653882, 'fraction_numerical': 0.007151215706670134, 'mean_word_length': 6.389048991354467, 'pattern_counts': {'":': 0, '<': 76, '<?xml version=': 0, '>': 78, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '187717', 'license': 'MIT', 'n_tokens_mistral': 2660, 'n_tokens_neox': 2562, 'n_words': 580, 'path': 'src/preprocess.ipynb', 'repo_name': 'LilDataScientist/Search-Engine', 'size': '7691'}