<commit_before>---
layout: post
title: Build your own search Engine
tags: elastic-search lucene tf*idf
permalink: search_engine.html
author: <NAME>
---

]In this post, I will take you through the steps for calculating the $$tf \times idf$$ values for all the words in a given document. To implement this, we use a small dataset (or corpus, as NLPers like to call it) form the [Project Gutenberg Catalog](http://www.gutenberg.org/). This is just a simple toy example on a very small dataset. In real life we use much larger [corpora](https://en.wikipedia.org/wiki/Text_corpus), and need some more sophisticated tools in order to handle large amounts of data. To brush up on the basic concepts of $$tf \times idf$$ you might want to check out my post on the [The Math behind Lucene](http://ticary.com/math_behind_lucene.html).

<!--more-->

For this exercise, we will use the [Project Gutenberg Selections](http://www.gutenberg.org/) which are released as part of [NLTK Data](http://nltk.googlecode.com/svn/trunk/nltk_data/index.xml). NLTK - Natural Language Toolkit - is a python based module for text processing. The corpus - [Project Gutenberg Selections](https://github.com/rutum/tf-idf/tree/master/gutenberg) - contains 18 files. Each file is a complete book. Our task is to calculate the $$tf \times idf$$ of all the words for each of the documents provided. In the end of this exercise we will have 18 documents, with the $$tf \times idf$$ of each word in each of the documents.

Documents with $$tf \times idf$$ values for each word (or token) are often used (with the [vector space model](http://en.wikipedia.org/wiki/Vector_space_model)) to compute the similarity between two documents. Such statistics are quite relevant in Information Retrieval, Search Engines, Document Similarity, and so on.

>NOTE: All the materials needed for this exercise (code + data) can be downloaded from my [github repo](https://github.com/rutum/tf-idf).

The code is written in perl, and is heavy in regular expressions. I have tried my best to document the code, but if you have any issues, or discover bugs, please do not hesitate in contacting me. I will not go into the details of all my code in this post, but will highlight a few features.
<commit_msg>updating title based on kramdown<commit_after>---
layout: post
title: Build your own Search Engine
tags: elastic-search lucene tf*idf
permalink: search_engine.html
author: <NAME>
---

In this post, I will take you through the steps for calculating the $$tf \times idf$$ values for all the words in a given document. To implement this, we use a small dataset (or corpus, as NLPers like to call it) form the [Project Gutenberg Catalog](http://www.gutenberg.org/). This is just a simple toy example on a very small dataset. In real life we use much larger [corpora](https://en.wikipedia.org/wiki/Text_corpus), and need some more sophisticated tools in order to handle large amounts of data. To brush up on the basic concepts of $$tf \times idf$$ you might want to check out my post on the [The Math behind Lucene](http://ticary.com/math_behind_lucene.html).

<!--more-->

For this exercise, we will use the [Project Gutenberg Selections](http://www.gutenberg.org/) which are released as part of [NLTK Data](http://nltk.googlecode.com/svn/trunk/nltk_data/index.xml). NLTK - Natural Language Toolkit - is a python based module for text processing. The corpus - [Project Gutenberg Selections](https://github.com/rutum/tf-idf/tree/master/gutenberg) - contains 18 files. Each file is a complete book. Our task is to calculate the $$tf \times idf$$ of all the words for each of the documents provided. In the end of this exercise we will have 18 documents, with the $$tf \times idf$$ of each word in each of the documents.

Documents with $$tf \times idf$$ values for each word (or token) are often used (with the [vector space model](http://en.wikipedia.org/wiki/Vector_space_model)) to compute the similarity between two documents. Such statistics are quite relevant in Information Retrieval, Search Engines, Document Similarity, and so on.

>NOTE: All the materials needed for this exercise (code + data) can be downloaded from my [github repo](https://github.com/rutum/tf-idf).

The code is written in perl, and is heavy in regular expressions. I have tried my best to document the code, but if you have any issues, or discover bugs, please do not hesitate in contacting me. I will not go into the details of all my code in this post, but will highlight a few features.

{'fraction_non_alphanumeric': 0.0835549645390071, 'fraction_numerical': 0.0017730496453900709, 'mean_word_length': 4.266044340723454, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 9, 'https://': 6, 'lorem ipsum': 0, 'www.': 4, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3099970', 'n_tokens_mistral': 1257, 'n_tokens_neox': 1212, 'n_words': 654}