<jupyter_start><jupyter_text>IntroductionThis notebook contains a text classification tutorial according to:https://towardsdatascience.com/applying-machine-learning-to-classify-an-unsupervised-text-document-e7bb6265f52  ImportsImport libraries and write settings here.<jupyter_code># Data manipulation
import pandas as pd
import numpy as np

# Options for pandas
pd.options.display.max_columns = 50
pd.options.display.max_rows = 30
pd.options.display.float_format = '{:,.4f}'.format

# Display all cell outputs
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = 'all'

from IPython import get_ipython
ipython = get_ipython()

# autoreload extension
if 'autoreload' not in ipython.extension_manager.loaded:
    %load_ext autoreload

%autoreload 2

# Visualizations
import seaborn as sns
#import plotly.plotly as py
#import plotly.graph_objs as go
#from plotly.offline import iplot, init_notebook_mode
#init_notebook_mode(connected=True)

import cufflinks as cf
cf.go_offline(connected=True)
cf.set_config_file(theme='white')<jupyter_output>/home/luca/anaconda3/envs/tensorflow/lib/python3.6/site-packages/plotly/graph_objs/_deprecations.py:558: DeprecationWarning:

plotly.graph_objs.YAxis is deprecated.
Please replace it with one of the following more specific types
  - plotly.graph_objs.layout.YAxis
  - plotly.graph_objs.layout.scene.YAxis


/home/luca/anaconda3/envs/tensorflow/lib/python3.6/site-packages/plotly/graph_objs/_deprecations.py:531: DeprecationWarning:

plotly.graph_objs.XAxis is deprecated.
Please replace it with one of the following more specific types
  - plotly.graph_objs.layout.XAxis
  - plotly.graph_objs.layout.scene.XAxis<jupyter_text>Custom imports<jupyter_code>from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.cluster import KMeans
import numpy as np
import pandas as pd<jupyter_output><empty_output><jupyter_text>Analysis/ModelingDo work here Create a text document with two kinds of sentences related with either *cricket* or *travelling*<jupyter_code>document = ["This is the most beautiful place in the world.",
            "This man has more skills to show in cricket than any other game.",
            "Hi there! how was your ladakh trip last month?",
            "There was a player who had scored 200 + runs in single cricket innings in his career.",
            "I have got the opportunity to travel to Paris next year for my internship.",
            "May be he is better than you in batting but you are much better than him in bowling.",
            "That was really a great day for me when I was there at Lavasa for the whole night.",
            "That’s exactly I wanted to become, a highest ratting batsmen ever with top scores.",
            "Does it really matter wether you go to Thailand or Goa, its just you have spend your holidays.",
            "Why don’t you go to Switzerland next year for your 25th Wedding anniversary?",
            "Travel is fatal to prejudice, bigotry, and narrow mindedness., and many of our people need it sorely on these accounts.",
            "Stop worrying about the potholes in the road and enjoy the journey.",
            "No cricket team in the world depends on one or two players. The team always plays to win.",
            "Cricket is a team game. If you want fame for yourself, go play an individual game.",
            "Because in the end, you won’t remember the time you spent working in the office or mowing your lawn. Climb that goddamn mountain.",
            "Isn’t cricket supposed to be a team sport? I feel people should decide first whether cricket is a team game or an individual sport."]<jupyter_output><empty_output><jupyter_text>Information extractionIn order to extract information from the document we adopt a well known method called **frequency-inverse document frequency (td-idf)**. This method allows to extract words' importance in the document by converting the textual representation into a *Vector Space Model (VSM)*.**Note:** this approach is used also, for example, in Google searches.  TF-IDFTF\*IDF is an information retrieval technique that weighs a term’s frequency (TF) and its inverse document frequency (IDF). Each word or term has its respective TF and IDF score. The product of the TF and IDF scores of a term is called the TF*IDF weight of that term.It is used to weigh a keyword in any content and assign the importance to that keyword based on the number of times it appears in the document:$$w_{i,j} = tf_{i,j} \times log \left( \frac{N}{df_i} \right)$$<jupyter_code>vectorizer = TfidfVectorizer(stop_words='english')
X = vectorizer.fit_transform(document)
vectorizer
X<jupyter_output><empty_output><jupyter_text>ClusteringOnce the information is extracted from the document and it is coded into a numeric vector, it is then possible to proceed with the clustering of sentences based on their content.In this case we use a simple **K-Means** algorithm.<jupyter_code># Set the number of clusters to output
true_k = 2

# Define the model with initialization
model = KMeans(n_clusters=true_k, init='k-means++', max_iter=100, n_init=1)

# Fit the model
model.fit(X)
# Get the resulting centroids
order_centroids = model.cluster_centers_.argsort()[:, ::-1]

# Get features
terms = vectorizer.get_feature_names()
df = pd.DataFrame(order_centroids, columns=terms , index=['Group_0', 'Group_1']) 
df.T.head(10)
for i in range(true_k):
    print("\nCluster %d:" % i),
    for ind in order_centroids[i, :20]:
        print(' %s' % terms[ind])<jupyter_output>Cluster 0:<jupyter_text>ResultsOnce we have trained the algorithm, then we can also predict group's belonging for a new sentence simply by assigning it to the closest centroid.<jupyter_code>X = vectorizer.transform(["Nothing is easy in cricket. Maybe when you watch it on TV, it looks easy. But it is not. You have to use your brain and time the ball."])
predicted = model.predict(X)
print(predicted)<jupyter_output>[1]<jupyter_text>So, here we have got the prediction as [1] which means it belongs to the cluster 1 which is related with Cricket and hence our test sentence is also talking about cricket fact. So our prediction is correct.<jupyter_code>X = vectorizer.transform(["Baseball and cricket are different as a sport."])
predicted = model.predict(X)
print(predicted)<jupyter_output>[0]
{'fraction_non_alphanumeric': 0.07626851560037819, 'fraction_numerical': 0.007406240151276395, 'mean_word_length': 4.514335360556038, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '371562', 'license': 'Apache-2.0', 'n_tokens_mistral': 1885, 'n_tokens_neox': 1810, 'n_words': 743, 'path': 'text_clustering_tutorial/Text clustering tutorial.ipynb', 'repo_name': 'clissa/transfer_errors', 'size': '6346'}