<commit_before>Last Tuesday [winter storm Stella hit New York](http://www.independent.co.uk/news/world/americas/stella-storm-blizzard-emergency-weather-ice-rain-a7630316.html), and because it was impossible to commute to [RC](https://www.recurse.com/) me and my roommate [@nandajavarma](https://twitter.com/nandajavarma) decided to spend our afternoon coding a [Python script](https://github.com/nandajavarma/document-similarity) to calculate the similarities between different documents.

Document similarity is a huge field in Natural Language Processing, so our purpose was to experiment with some simple concepts and see the results of comparing a few documents. We divided the job into two tasks to be able to cover the whole functionality in one day: my roommate was in charge of extracting the semantics of the documents while I was in charge of calculating the similarities of the resulting extractions.

# Creating a semantical representation

The difference between semantical vs syntactical representation lies in the idea of distance between words based on the likeness of their **meaning** as opposed to their **syntactical composition** (i.e. word order, lexemes or how the String is created).

Let's take a look at these two sentences:
- The **bear** shit in the woods.
- I can't **bear** this shit.

Both sentences are syntactically quite similar but semantically different. The word *bear* has a different connotation in each sentence; in the first one we can link the interpretation of 'bear' with 'animal' or 'dangerous' while in the second one we can link it with words like 'action' or 'upset'.


![alt lsh]({{ site.url }}/assets/images/lsh_post/bear.jpg)

For our script we chose to extract the semantic space from the document content. The script goes through each document word by word and updates the vector representation of the document by adding its semantic relation weight value. Of course, it discards irrelevant words like prepositions or articles.

In addition of the semantic value, we also take into account the [entropy](https://www.quora.com/What-is-high-entropy-data) of each word, grading and updating the word's weight by its ambiguity and uninformativeness. A not very relevant word should have a high entropy, which will manifest ifself in a small update of the document vector representation.

# Calculating similarities<commit_msg>Changed word in LSH post<commit_after>Last Tuesday [winter storm Stella hit New York](http://www.independent.co.uk/news/world/americas/stella-storm-blizzard-emergency-weather-ice-rain-a7630316.html), and because it was impossible to commute to [RC](https://www.recurse.com/) me and my roommate [@nandajavarma](https://twitter.com/nandajavarma) decided to spend our afternoon coding a [Python script](https://github.com/nandajavarma/document-similarity) to calculate the similarities between different documents.

Document similarity is a huge field in Natural Language Processing, so our purpose was to experiment with some simple concepts and see the results of comparing a few documents. We divided the job into two tasks to be able to cover the whole functionality in one day: my roommate was in charge of extracting the semantics of the documents while I was in charge of calculating the similarities of the resulting extractions.

# Creating a semantical representation

The difference between semantical vs syntactical representation lies in the idea of distance between words based on the likeness of their **meaning** as opposed to their **syntactical composition** (i.e. word order, lexemes or how the String is created).

Let's take a look at these two sentences:
- The **bear** shit in the woods.
- I can't **bear** this shit.

Both sentences are syntactically quite similar but semantically different. The word *bear* has a different connotation in each sentence; in the first one we can link the interpretation of 'bear' with 'animal' or 'dangerous' while in the second one we can link it with words like 'handle' or 'upset'.


![alt lsh]({{ site.url }}/assets/images/lsh_post/bear.jpg)

For our script we chose to extract the semantic space from the document content. The script goes through each document word by word and updates the vector representation of the document by adding its semantic relation weight value. Of course, it discards irrelevant words like prepositions or articles.

In addition of the semantic value, we also take into account the [entropy](https://www.quora.com/What-is-high-entropy-data) of each word, grading and updating the word's weight by its ambiguity and uninformativeness. A not very relevant word should have a high entropy, which will manifest ifself in a small update of the document vector representation.

# Calculating similarities
{'fraction_non_alphanumeric': 0.06175173282923756, 'fraction_numerical': 0.0029405587061541692, 'mean_word_length': 4.900867410161091, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 8, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 2, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5559919', 'n_tokens_mistral': 1234, 'n_tokens_neox': 1143, 'n_words': 651}