<commit_before>---
layout: post
title:  "Markov Word Net?"
date:   2017-09-18 12:00:00 -0500
categories: journal
---
NLP research regularly utilizes lexical resources to impart some linguistic intelligence to software. See [Ignatow and Mihalcea, 2017](#references) (chapter 4) for a basic introduction. One very important resource is [WordNet](https://wordnet.princeton.edu/) and it's derived resources, such as [WordNet-Affect](http://wndomains.fbk.eu/wnaffect.html) and [SentiWordNet](http://sentiwordnet.isti.cnr.it/) ([GitHub repo here](https://github.com/kevincobain2000/sentiment_classifier)).  

Text processing in NLP builds on raw text to create a more information-rich corpus. This enhanced corpus serves as a better foundation for analysis of a corpus. See [Ignatow and Mihalcea, 2017](#references) (chapter 5) for an introduction.

## WordNet

WordNet and it's related resources classify words in the English language according to synsets and relations. Each Synsets, or synonym set, is related to other synsets by relations. A few selected WordNet relations are listed below for relation *Y* of word *X*:

| Relation   | Part of Speech | Definition |
| ---------- | -------------- | ---------- |
| `Hypernym` | Noun & Verb    | Hypernym *Y* is a superset of *X* |
| `Hyponym`  | Noun & Verb    | Hyponym *Y* is a subset of *X* |
| `Troponym` | Verb           | Troponym *Y* is a means of doing *X* |
| `Antonym`  | Adjective      |  Antonym *Y* has opposite meaning of *X |<commit_msg>Remove leading cell space in table<commit_after>---
layout: post
title:  "Markov Word Net?"
date:   2017-09-18 12:00:00 -0500
categories: journal
---
NLP research regularly utilizes lexical resources to impart some linguistic intelligence to software. See [Ignatow and Mihalcea, 2017](#references) (chapter 4) for a basic introduction. One very important resource is [WordNet](https://wordnet.princeton.edu/) and it's derived resources, such as [WordNet-Affect](http://wndomains.fbk.eu/wnaffect.html) and [SentiWordNet](http://sentiwordnet.isti.cnr.it/) ([GitHub repo here](https://github.com/kevincobain2000/sentiment_classifier)).  

Text processing in NLP builds on raw text to create a more information-rich corpus. This enhanced corpus serves as a better foundation for analysis of a corpus. See [Ignatow and Mihalcea, 2017](#references) (chapter 5) for an introduction.

## WordNet

WordNet and it's related resources classify words in the English language according to synsets and relations. Each Synsets, or synonym set, is related to other synsets by relations. A few selected WordNet relations are listed below for relation *Y* of word *X*:

| Relation   | Part of Speech | Definition |
| ---------- | -------------- | ---------- |
| `Hypernym` | Noun & Verb    | Hypernym *Y* is a superset of *X* |
| `Hyponym`  | Noun & Verb    | Hyponym *Y* is a subset of *X* |
| `Troponym` | Verb           | Troponym *Y* is a means of doing *X* |
| `Antonym`  | Adjective      | Antonym *Y* has opposite meaning of *X |
{'fraction_non_alphanumeric': 0.123342175066313, 'fraction_numerical': 0.021220159151193633, 'mean_word_length': 3.6920684292379473, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2117548', 'n_tokens_mistral': 979, 'n_tokens_neox': 880, 'n_words': 363}