<jupyter_start><jupyter_text>Introduction to machine learning. Natural language processing.___   *<NAME>*  About me___* Contact: https://www.linkedin.com/in/radoslav-petkov-8a4a53144/* Sofia University, Computer Science, Bsc (2nd year)* Has been working for Sirma since the end of 2014  Summary___* Supervised vs Unsupervised machine learning* Representation of words and sentences* Autoencoders* Sequence to sequence models* Memory Networks  Supervised vs Unsupervised![](un_supervised.png)  Supervised___ Example of simple linear model: $ y= X*W + b$   Regression___ $y \in /R $<jupyter_code>from sklearn.datasets import load_boston
import pandas as pd
import numpy as np
boston = load_boston()

dataset = pd.DataFrame(data=boston["data"], columns=boston["feature_names"])
dataset["Target"] = boston["target"]

print(boston["DESCR"])
dataset[dataset["Target"] < 10][:2]
dataset[(dataset["Target"] > 10) & (dataset["Target"] < 30)][:2]
dataset[dataset["Target"] > 45][:2]<jupyter_output><empty_output><jupyter_text>Classification___ $y \in [0..c]$<jupyter_code>from sklearn.datasets import load_iris

iris = load_iris()
pd_iris = pd.DataFrame(data=iris["data"], columns=iris["feature_names"])
pd_iris["Type"] = iris["target"]
print(iris["DESCR"])
pd_iris [ pd_iris["Type"] == 0][:2]
pd_iris [ pd_iris["Type"] == 1][:2]
pd_iris [ pd_iris["Type"] == 2][:2]<jupyter_output><empty_output><jupyter_text>![](http://www.strelitzia-bg.com/images/products/8577_1.jpg) ![](https://upload.wikimedia.org/wikipedia/commons/thumb/b/b1/Mature_flower_diagram_bg.svg/2000px-Mature_flower_diagram_bg.svg.png)  Loss, the function for optimization----Loss functions measure how bad (based on the labeled data) our model predicts and we want to find good enough weights that minimize it.* Mean Squared Error (MSE): $\frac{1}{n}\sum_{i=1}^{n}(y_i - \hat{y_i})^2$* LogLoss: $- \frac{1}{n} \sum\limits_{i=1}^n (y_i \cdot ln(\hat{y_i}) + (1-y_i) \cdot ln(1-\hat{y_i}) )$  Unsupervised $ y$ is missing<jupyter_code># Code source: <NAME>
# Modified for documentation by <NAME>
# License: BSD 3 clause
# with slight modifications by me

import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

from sklearn.cluster import KMeans
from sklearn import datasets
plt.close("all")
np.random.seed(5)

iris = datasets.load_iris()
X = iris.data
y = iris.target

estimators = [('k_means_iris_8', KMeans(n_clusters=8)),
              ('k_means_iris_5', KMeans(n_clusters=5)),
                       ('k_means_iris_3', KMeans(n_clusters=3))]

titles = ['8 clusters', '5 clusters', '3 clusters']
fig, axes = plt.subplots(ncols=2, nrows=2, figsize=(12,12))

for i, (name, est) in enumerate(estimators):
    ax = fig.add_subplot(2, 2, i+1, projection='3d')
    est.fit(X)
    labels = est.labels_

    ax.scatter(X[:, 3], X[:, 0], X[:, 2],
               c=labels.astype(np.float), edgecolor='k')

    ax.w_xaxis.set_ticklabels([])
    ax.w_yaxis.set_ticklabels([])
    ax.w_zaxis.set_ticklabels([])
    ax.set_xlabel('Petal width')
    ax.set_ylabel('Sepal length')
    ax.set_zlabel('Petal length')
    ax.set_title(titles[i])
    ax.dist = 12

# Plot the ground truth
ax = fig.add_subplot(2, 2, 4, projection='3d')
for name, label in [('Setosa', 0),
                    ('Versicolour', 1),
                    ('Virginica', 2)]:
    ax.text3D(X[y == label, 3].mean(),
              X[y == label, 0].mean(),
              X[y == label, 2].mean() + 2, name,
              horizontalalignment='center',
              bbox=dict(alpha=.2, edgecolor='w', facecolor='w'))
# Reorder the labels to have colors matching the cluster results
y = np.choose(y, [1, 2, 0]).astype(np.float)
ax.scatter(X[:, 3], X[:, 0], X[:, 2], c=y, edgecolor='k')

ax.w_xaxis.set_ticklabels([])
ax.w_yaxis.set_ticklabels([])
ax.w_zaxis.set_ticklabels([])
ax.set_xlabel('Petal width')
ax.set_ylabel('Sepal length')
ax.set_zlabel('Petal length')
ax.set_title('Ground Truth')
ax.dist = 12

fig.axes
for ax in fig.axes:
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
# fig.subplots_adjust(hspace=0)
plt.show()<jupyter_output><empty_output><jupyter_text>What is overfitting, underfitting and what is good fit?* Overfitting - high score on train and low on test* Underfitting - low score on both train and test* Generalization - high score on both train and test  Natural-language processing (NLP) How the computer works with text?![](natural-language-processing.jpg) *   What is one hot encoding of words?Simply said, it is representation where each word is vector with size equal to the vocabulary size and there is 1 at the index equal to the word index.Imagine you have the following corpus of words: *Machine*, *learning*, *rocks*.***One representation would be :***Machine -> [1 0 0],  learning -> [0 1 0], rocks -> [0 0 1] *  What about Bag of Words?Each sentence is vector with size equal to the vocablary size storing the occurances of each word in the sentence.There are several additional modifications such as:* **TF-IDF (Term frequency - Inverse document frequency) **    We assign weights of each word instead of just occurances.The weights tend to filter out common terms so the lower the weight is the more occurances the word has.    TF is the ration between the count of the word in the current sentence and all words in the sentence.    IDF is the logarithmically scaled fraction of all documents in the corpus with the documents that contain the given word.        TF-IDF has various weighting schemes, for example :    $ \mbox{tf-idf}_{t,d} = (1 +\log \mbox{tf}_{t,d}) \cdot \log \frac{N}{\mbox{df}_t} $ * **Hashing**    Instead of using dictionary to vectorize, a special hash function is used.  Examples___Lets have the following corpus: *machine*, *learning*, *rocks*, *this*, *robot*And the following sentences:* *machine learning rocks** *this machine learning rocks** *this robot rocks*<jupyter_code>from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfVectorizer

corpus = ["this computer", "this machine", "this robot", "this text"]
corpus_transformed = CountVectorizer().fit(corpus)
corpus_tf_idf  = TfidfVectorizer().fit(corpus)
print(corpus_transformed.vocabulary_)
for (sentence, transf) in zip(corpus, corpus_transformed.transform(corpus).toarray()):
    print(sentence, transf)
print(corpus_tf_idf.vocabulary_)
for (sentence, transf) in zip(corpus, corpus_tf_idf.transform(corpus).toarray()):
    print(sentence, transf)<jupyter_output>{'computer': 0, 'robot': 2, 'this': 4, 'machine': 1, 'text': 3}
this computer [0.88654763 0.         0.         0.         0.46263733]
this machine [0.         0.88654763 0.         0.         0.46263733]
this robot [0.         0.         0.88654763 0.         0.46263733]
this text [0.         0.         0.         0.88654763 0.46263733]<jupyter_text>*   Hmm, now the fancy word vectorsVectors from lattent space formed by the corpus used for training unsupervised model.They can catch the 'semantic meaning of a word'.![](word_embedding2.jpg) ![](word_embedding.png)<jupyter_code>from gensim.models import KeyedVectors

word_vectors = KeyedVectors.load_word2vec_format('/media/radoslav/6906F83679A14133/Download/glove/GoogleNews-vectors-negative300.bin', binary=True)
word_vectors.wv.most_similar(positive=['walking', 'swam'], negative=['swimming'])[0]
word_vectors.wv.most_similar(positive=['France', 'Sofia'], negative=['Paris'])[0]
word_vectors.wv.most_similar(positive=['queen', 'man'], negative=['woman'])[0]<jupyter_output><empty_output><jupyter_text>Now lets try what such a model can do on one of those IQ quizzes<jupyter_code>word_vectors.wv.doesnt_match(["apple", "banana", "orange", "bread"])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13778867242936396, 'fraction_numerical': 0.028254418784672945, 'mean_word_length': 4.666666666666667, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 32, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '378822', 'license': 'MIT', 'n_tokens_mistral': 2784, 'n_tokens_neox': 2582, 'n_words': 730, 'path': 'SirmaTechLab/Intro_To_ML_NLP/.ipynb_checkpoints/Intro to machine learning. NLP-checkpoint.ipynb', 'repo_name': 'radpet/stuff', 'size': '7776'}