<jupyter_start><jupyter_text>Vector Representations **Importing needed libraries**<jupyter_code>import pandas as pd
from urllib.request import urlopen
import spacy

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.preprocessing import Binarizer
from sklearn.metrics.pairwise import cosine_similarity
from sklearn.decomposition import PCA

import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>**Text Vectorization**<jupyter_code># Create the corpus with random sentences

corpus = ["Karma, karma, karma, karma, karma chameleon.",
          "The paintbrush was angry at the color the chameleon chose to use.",
          "She stomped on her fruit loops and thus became a cereal killer.",
          "He hated that he loved what she hated about cereal and her chameleon."
         ]<jupyter_output><empty_output><jupyter_text>**Frequency Count**<jupyter_code># Frequency-count

# Instantiate the object and count the words
vectorizer = CountVectorizer()
vectors = vectorizer.fit_transform(corpus)

# Convert to dense vectors (leave out the zeroes)
print(vectors.todense())<jupyter_output>[[0 0 0 0 0 0 1 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
 [0 0 1 1 0 0 1 1 1 0 0 0 0 0 0 0 0 0 1 0 0 0 3 0 1 1 1 0]
 [0 1 0 0 1 1 0 0 0 1 0 0 1 0 1 1 0 1 0 1 1 0 0 1 0 0 0 0]
 [1 1 0 0 0 1 1 0 0 0 2 2 1 0 0 0 1 0 0 1 0 1 0 0 0 0 0 1]]<jupyter_text>**One Hot Encoding**<jupyter_code># One-hot encoding of word counts
# Initialize the vectorizer and get the word counts
freq   = CountVectorizer()
corpus_freq = freq.fit_transform(corpus)

# Initialize the binarizer and create the binary encoded vector
onehot = Binarizer()
corpus_onehot = onehot.fit_transform(corpus_freq.toarray())

# Display the one-hot encoded vector
corpus_onehot<jupyter_output><empty_output><jupyter_text>**Term frequency-inverse document frequency**<jupyter_code># Instantiate vectorizer object
tfidf = TfidfVectorizer(stop_words='english', max_features=5000)

# Create a vocabulary and get word counts per document
dtm = tfidf.fit_transform(corpus)

# Get feature names to use as dataframe column headers
dtm = pd.DataFrame(dtm.todense(), columns=tfidf.get_feature_names())

# View feature matrix as DataFrame
dtm.head()<jupyter_output>C:\Users\tyler\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\sklearn\utils\deprecation.py:87: FutureWarning: Function get_feature_names is deprecated; get_feature_names is deprecated in 1.0 and will be removed in 1.2. Please use get_feature_names_out instead.
  warnings.warn(msg, category=FutureWarning)<jupyter_text>**Cosine Similarity**- How are two vectors similar to each other<jupyter_code># The text consists of three documents on three different subjects
link = 'https://raw.githubusercontent.com/nwhoffman/NLP_example_text/master/u4s1m2_similarity.txt'
f = urlopen(link)
myfile = f.read()

mystring = str(myfile, 'utf-8')
corpus = mystring.split(';')

# Print out the first 300 characters for each document
for i in [0, 1, 2]:
    print('Document:', i)
    print(corpus[i][0:300])
# Instantiate vectorizer object
tfidf = TfidfVectorizer(stop_words='english', max_features=5000)

# Create a vocabulary and get tfidf values per document
dtm = tfidf.fit_transform(corpus)

# Get feature names to use as DataFrame column headers
dtm = pd.DataFrame(dtm.todense(), columns=tfidf.get_feature_names_out())

# View the feature matrix as a DataFrame
dtm.head()
# Find the cosine similarity of tf-idf vectors
cosine_sim  = cosine_similarity(dtm)

# Turn it into a DataFrame
cosine_sim = pd.DataFrame(cosine_sim)
display(cosine_sim)<jupyter_output><empty_output><jupyter_text>**Word Embedding**The concept of word embedding is related to the mathematical concept of embedding, where a word is represented as a continuous vector in a space with a much lower dimension. Words or phrases (all of the unique words in the corpus) are mapped to vectors in real space. Each word is mapped to a single vector that is learned from applying neural network techniques. Thus, words that are in the same contexts will have similar meanings and similar word vectors.There are a few different ways to generate word vectors. First, we’ll present a brief overview of the popular word2vec tool and then look at the word vectors available in spaCy, generated from the GloVe algorithm.**Word2vec**Word embeddings, in general, are found like processing a large corpus of text to learn the meanings of words. In the process, we don’t want to learn the exact definition of a word but rather some representation of the meaning. We can think of word vectors as a list of weights or scores, where each vector element is a specific dimension. These dimensions represent things like the “peopleness” or “animalness” or “foodness” of a word. The combination of all of these “ness” values is the word vector.The word2vec tool uses two algorithms to find these vectors: the skip-gram algorithm and the CBOW algorithm.**skip-gram**The skip-gram method is named because it uses n-grams that contain gaps where the token is skipped. The surrounding words are predicted based on the input word. This method for generating work vectors works well with small corpora and rare word**CBOW**The continuous-bag-of-words or CBOW method works similarly to the skip-gram method but instead predicts the center word from the surrounding words. This model is faster to train and results in higher accuracy for words that are in use frequently.<jupyter_code>nlp = spacy.load('en_core_web_lg')

# Set the document (this example contains two words)
doc = nlp("drama llama")

# Get the vectors for document
# (the vectors are the average of the two word vectors)
llama_vector = doc.vector

# Look length of the vector (how many dimensions?)
print(llama_vector.shape)

# Display part of the vector
print(llama_vector[:10])
# Find the similarity score for example words

# Compare "llama" and "horse"
print("The similarity score for llama-llama is: ", nlp("llama").similarity(nlp("llama")))
print("The similarity score for llama-horse is: ", nlp("llama").similarity(nlp("horse")))
print("The similarity score for llama-car is: ", nlp("llama").similarity(nlp("car")))<jupyter_output>The similarity score for llama-llama is:  1.0
The similarity score for llama-horse is:  0.37856930047799736
The similarity score for llama-car is:  0.08994985333860218<jupyter_text>**We want to view the vector space using PCA to reduce the vectors dimensionality**<jupyter_code># Define a function to find the vector for a word
def get_word_vectors(words):
    # converts a list of words into their word vectors
    return [nlp(word).vector for word in words]

# Example word list to find vectors for
words = ['python', 'javascript', 'code', 'data', 'linux',
        'llama', 'alpaca', 'cat', 'snake',
        'cook', 'bake', 'cookie', 'clean', 'car']

# Initialize the PCA model and project down to two dimensions
pca = PCA(n_components=2)

# Fit the PCA model on the word list
pca.fit(get_word_vectors(words))

# Apply the transformation learned from the PCA model
word_vecs_2d = pca.transform(get_word_vectors(words))

# Look at the 2-D array (which is just x and y coordinates)
word_vecs_2d
plt.figure(figsize=(10,8))
plt.scatter(word_vecs_2d[:,0], word_vecs_2d[:,1])

# Display the text of the word at each point
for word, coord in zip(words, word_vecs_2d):
    x, y = coord
    plt.text(x+0.05, y, word, size= 15)

# show the plot
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0814539665693818, 'fraction_numerical': 0.02759352613425312, 'mean_word_length': 5.065164923572003, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 30, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 16, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '569918', 'license': 'MIT', 'n_tokens_mistral': 2518, 'n_tokens_neox': 2289, 'n_words': 947, 'path': 'Machine Learning/Natural Language Processing/Vector Representations.ipynb', 'repo_name': 'Tyler9937/Data-Science-and-Computer-Science-Fundamentals', 'size': '7538'}