<jupyter_start><jupyter_text>Critical Introduction to Natural Language Processing Using only the raw text, we'll derive and explore the semantic properties of its words.  Imports Python code in one module gains access to the code in another module by the process of importing it. The import statement is the most common way of invoking the import machinery, but it is not the only way.<jupyter_code>from __future__ import absolute_import, division, print_function<jupyter_output><empty_output><jupyter_text>First we import common system-tools etc. here that are not directly connected to NLP<jupyter_code>import codecs
import glob
import logging
import multiprocessing
import os
import pprint
import re
import nltk
import gensim.models.word2vec as w2v
from gensim.models import KeyedVectors
from gensim.models import Word2Vec
from gensim.utils import simple_preprocess
import sklearn.manifold
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns<jupyter_output>D:\Anaconda3\lib\site-packages\gensim\utils.py:1197: UserWarning: detected Windows; aliasing chunkize to chunkize_serial
  warnings.warn("detected Windows; aliasing chunkize to chunkize_serial")<jupyter_text>You will probably run into an "ModuleNorFoundError" here. THis means that the needed module is not installed on your system.You can do that in the anaconda command prompt:for example: "conda install -c anaconda nltk" or "conda install -c anaconda gensim"  for detailed information refer to https://docs.anaconda.com/anaconda/user-guide/tasks/install-packages/<jupyter_code>%pylab inline<jupyter_output>Populating the interactive namespace from numpy and matplotlib<jupyter_text>**Set up logging**<jupyter_code>logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)<jupyter_output><empty_output><jupyter_text>**Download NLTK tokenizer models (only the first time)**<jupyter_code>nltk.download("punkt")
nltk.download("stopwords")<jupyter_output>[nltk_data] Downloading package punkt to
[nltk_data]     C:\Users\fmx\AppData\Roaming\nltk_data...
[nltk_data]   Package punkt is already up-to-date!
[nltk_data] Downloading package stopwords to
[nltk_data]     C:\Users\fmx\AppData\Roaming\nltk_data...
[nltk_data]   Package stopwords is already up-to-date!<jupyter_text>Set a Text<jupyter_code>sentence = "The oppressed are allowed once every few years to decide which particular representatives of the oppressing class are to represent and repress them."<jupyter_output><empty_output><jupyter_text>**Build your vocabulary (word tokenization)**<jupyter_code>tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')<jupyter_output><empty_output><jupyter_text>Given a character sequence and a defined document unit, tokenization is the task of chopping it up into pieces, called tokens , perhaps at the same time throwing away certain characters, such as punctuation.<jupyter_code>#convert into a list of words
#rtemove unnnecessary,, split into words, no hyphens
#list of words
def sentence_to_wordlist(raw):
    clean = re.sub("[^a-zA-Z]"," ", raw)
    words = clean.split()
    return words
#sentence where each word is tokenized
words = sentence_to_wordlist(sentence)
print(sentence)
print(words)
num_tokens = len(words)
print("The corpus contains {0:,} tokens".format(num_tokens))<jupyter_output>The corpus contains 23 tokens<jupyter_text>Now we sort the words lexically<jupyter_code>vocab = sorted(set(words))
vocab_size = len(vocab)
', '.join(vocab)
print("The size of the vocabulary is {0:,} words".format(vocab_size))<jupyter_output>The size of the vocabulary is 21 words<jupyter_text>Create a one-hot vector representation<jupyter_code>onehot_vectors = np.zeros((num_tokens, vocab_size), int)
print(onehot_vectors)
for i, word in enumerate(words):
    onehot_vectors[i, vocab.index(word)] = 1
' '.join(vocab)
print(onehot_vectors)
pd.DataFrame(onehot_vectors, words, columns=vocab)
df = pd.DataFrame(onehot_vectors, columns=vocab)
df[df == 0] = ''
df<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08827175799652864, 'fraction_numerical': 0.003967270022315894, 'mean_word_length': 6.742802303262956, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 36, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '132807', 'license': 'MIT', 'n_tokens_mistral': 1326, 'n_tokens_neox': 1283, 'n_words': 390, 'path': 'Basic.ipynb', 'repo_name': 'akoenig-mat/bauhauslearningmachines', 'size': '4033'}