<jupyter_start><jupyter_text>NLP Cheat Sheet - Introduction - Overview - Python - Starter KitIntroduction to Natural Language Processing (NLP) tools, frameworks, concepts, resources for PythonDemo: [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/janlukasschroeder/nlp-cheat-sheet-python/master?filepath=NLP%2BCheat%2BSheet.ipynb) NLP Python Libraries- [spacy](https://spacy.io/)- [NLTK](https://www.nltk.org/) - similar to spacy, supports more models, simpl GUI model download `nltk.download()`- [gensim](https://radimrehurek.com/gensim) - topic modelling, accessing corpus, similarity calculations between query and indexed docs, SparseMatrixSimilarity, Latent Semantic Analysis- [lexnlp](https://github.com/LexPredict/lexpredict-lexnlp) - information retrieval and extraction for real, unstructured legal text- [Holmes](https://github.com/msg-systems/holmes-extractorthe-basic-idea) - information extraction, document classification, search in documents- [Pytorch-Transformers - includes BERT, GPT2, XLNet](https://huggingface.co/pytorch-transformers/index.html)Uncased model is better unless you know that case information is important for your task (e.g., Named Entity Recognition or Part-of-Speech tagging)  General- PyTorch is an open source machine learning library based on the Torch library, used for applications such as computer vision and natural language processing- Tensorflow- Keras NLP Algortihms- [GPT-2](https://openai.com/blog/better-language-models/) - generate fake news, text summaries- [BERT](https://github.com/google-research/bert)- [FinBERT](https://github.com/ProsusAI/finBERT) - analyze sentiment of financial text- [XLnet](https://github.com/zihangdai/xlnet)- [ERNIE](https://github.com/PaddlePaddle/ERNIE/) Datasets- [Gutenberg Corpus](https://block.pglaf.org/germany.shtml) - contains 25,000 free electronic books. `from nltk.corpus import gutenberg`- [OntoNotes 5](https://github.com/ontonotes/conll-formatted-ontonotes-5.0) - corpus comprising various genres of text (news, conversational telephone speech, weblogs, usenet newsgroups, broadcast, talk shows) in three languages (English, Chinese, and Arabic) with structural information (syntax and predicate argument structure) and shallow semantics (word sense linked to an ontology and coreference).- [wiki_en_tfidf.mm in gensim](https://radimrehurek.com/gensim/wiki.htmllatent-semantic-analysis) 3.9M documents, 100K features (distinct tokens) and 0.76G non-zero entries in the sparse TF-IDF matrix. The Wikipedia corpus contains about 2.24 billion tokens in total.- [GPT-2 Dataset](https://github.com/openai/gpt-2-output-dataset)- [Brown corpus](http://icame.uib.no/brown/bcm-los.html) - contains text from 500 sources, and the sources have been categorized by genre, such as news, editorial, and so on.- [Reuters Corpus - 10,788 news documents totaling 1.3 million words](https://www.nltk.org/book/ch02.html)- [Newsfilter.io stock market news corpus](https://developers.newsfilter.io/) - contains over 4 million press releases, earnings reports, FDA drug approvals, analyst ratings, merger agreements and many more covering all US companies listed on NASDAQ, NYSE, AMEX- [Kaggle - All the news, 143K articles](https://www.kaggle.com/snapcrack/all-the-news)- [Kaggle - Daily news for stock market prediction](https://www.kaggle.com/aaron7sun/stocknews)- [CNN News](https://drive.google.com/uc?export=download&id=0BwmD_VLjROrfTHk4NFg2SndKcjQ)- [AG News - PyTorch integrated](https://pytorch.org/tutorials/beginner/text_sentiment_ngrams_tutorial.html) Installation:spacy (good for beginners; use NLTK for bigger projects)```shellpip install spacypython -m spacy download en  python -m spacy download en_core_web_lg```LexNLP (good for dealing with legal and financial documents; [installation guide here](https://github.com/LexPredict/lexpredict-contraxsuite/blob/master/documentation/Installation%20and%20Configuration/Installation%20and%20Configuration%20Guide.pdf))```shellpip install https://github.com/LexPredict/lexpredict-lexnlp/archive/master.zippython  to open REPL console>>> import nltk>>> nltk.download()  download all packages```   Concepts---  Word embeddings (=word vectors)Visualizing word vectors using PCA. Paper: https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf![word-embs](https://www.tensorflow.org/images/linear-relationships.png)- Word embeddings are vector representation of words. - Example sentence: word embeddings are words converted into numbers.- A word in this sentence may be “Embeddings” or “numbers ” etc.- A dictionary may be the list of all unique words in the sentence, eg [‘Word’,’Embeddings’,’are’,’Converted’,’into’,’numbers’]- A vector representation of a word may be a one-hot encoded vector where 1 stands for the position where the word exists and 0 everywhere else.  Example- numbers = [0,0,0,0,0,1] - converted = [0,0,0,1,0,0]** Either use pre-trained word vectors or train our own**  Pre-trained word embeddings:- Word2Vec (Google, 2013), uses Skip Gram and CBOW- [Vectors trained on Google News (1.5GB)](https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit) - vocabulary of 3 million words trained on around 100 billion words from the google news dataset- GloVe (Stanford)- [Stanford Named Entity Recognizer (NER)](https://nlp.stanford.edu/software/CRF-NER.shtml)- [LexPredict: pre-trained word embedding models for legal or regulatory text](https://github.com/LexPredict/lexpredict-lexnlp)- [LexNLP legal models](https://github.com/LexPredict/lexpredict-legal-dictionary) - US GAAP, finaical common terms, US federal regulators, common law Create word vectors yourself```pythonimport gensimword2vev_model = gensim.models.word2vec.Word2Vec(sentence_list)```https://www.analyticsvidhya.com/blog/2017/06/word-embeddings-count-word2veec/  How to create word vectors?- Count-based methods compute the statistics of how often some word co-occurs with its neighbor words in a large text corpus, and then map these count-statistics down to a small, dense vector for each word. - Predictive models directly try to predict a word from its neighbors in terms of learned small, dense embedding vectors (considered parameters of the model).  - Example: Word2vec (Google) 1. Count based word embeddings Count Vector (= Document Term Matrix)![img](https://s3-ap-south-1.amazonaws.com/av-blog-media/wp-content/uploads/2017/06/04164920/count-vector.png) TF-IDFTerm Frequency - Inverse Document Frequency- Term frequency equals the number of times a word appears in a document divided by the total number of words in the document. - Inverse document frequency calculates the weight of rare words in all documents in the corpus, with rare words having a high IDF score, and words that are present in all documents in a corpus having IDF close to zero.(sklearn) in Python has a function TfidfVectorizer() that will compute the TF-IDF values for you<jupyter_code>from sklearn.feature_extraction.text import TfidfVectorizer

# Write a function for cleaning strings and returning an array of ngrams
def ngrams_analyzer(string):
    string = re.sub(r'[,-./]', r'', string)
    ngrams = zip(*[string[i:] for i in range(5)])  # N-Gram length is 5
    return [''.join(ngram) for ngram in ngrams]

# Construct your vectorizer for building the TF-IDF matrix
vectorizer = TfidfVectorizer(analyzer=ngrams_analyzer)

# Credits: https://towardsdatascience.com/group-thousands-of-similar-spreadsheet-text-cells-in-seconds-2493b3ce6d8d<jupyter_output><empty_output><jupyter_text>Co-Occurrence Vector 2. Prediction based word embeddings- Uses Neural Networks- CBOW predicts target words (e.g. 'mat') from source context words ('the cat sits on the')- Skip-gram does the inverse and predicts source context-words from the target words CBOW (Continuous Bag of words) Skip GramSkip – gram follows the same topology as of CBOW. It just flips CBOW’s architecture on its head. The aim of skip-gram is to predict the context given a word Outcome![out](https://github.com/sanketg10/deep-learning-repo/raw/6b207e326cc937930a0512a8c599e86e48c297b2/embeddings/assets/skip_gram_net_arch.png)  Bag of Words<jupyter_code># John likes to watch movies. Mary likes movies too.
BoW1 = {"John":1,"likes":2,"to":1,"watch":1,"movies":2,"Mary":1,"too":1};<jupyter_output><empty_output><jupyter_text>spacy<jupyter_code>import spacy
# Import dataset
nlp = spacy.load("en")
# Import large dataset. Needs to be downloaded first.
# nlp = spacy.load("en_core_web_lg")<jupyter_output><empty_output><jupyter_text>Stop WordsStop words are the very common words like ‘if’, ‘but’, ‘we’, ‘he’, ‘she’, and ‘they’.We can usually remove these words without changing the semantics of a text and doing so often (but not always) improves the performance of a model.<jupyter_code># spacy: Removing stop words
spacy_stopwords = spacy.lang.en.stop_words.STOP_WORDS

print('spacy: Number of stop words: %d' % len(spacy_stopwords))
# nltk: Removing stop words 
from nltk.corpus import stopwords
english_stop_words = stopwords.words('english')

print('ntlk: Number of stop words: %d' % len(english_stop_words))
text = 'Larry Page founded Google in early 1990.'
doc = nlp(text)
tokens = [token.text for token in doc if not token.is_stop]
print('Original text: %s' % (text))
print()
print(tokens)<jupyter_output>Original text: Larry Page founded Google in early 1990.

['Larry', 'Page', 'founded', 'Google', 'early', '1990', '.']<jupyter_text>SpansPart of a given text. So doc[2:4] is a span starting at token 2, up to – but not including! – token 4.Docs: https://spacy.io/api/span<jupyter_code>doc = nlp("Larry Page founded Google in early 1990.")
span = doc[2:4]
span.text
[(spans) for spans in doc]<jupyter_output><empty_output><jupyter_text>Token and TokenizationSegmenting text into words, punctuation etc.- Sentence tokenization- Word tokenizationDocs: https://spacy.io/api/token<jupyter_code>doc = nlp("Larry Page founded Google in early 1990.")
[token.text for token in doc]
# Load OpenAI GPT-2 using PyTorch Transformers
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2Model.from_pretrained('gpt2')
# https://huggingface.co/pytorch-transformers/serialization.html<jupyter_output><empty_output><jupyter_text>Chunks and ChunkingSegments and labels multi-token sequences.- Each of these larger boxes is called a chunk. - Like tokenization, which omits whitespace, chunking usually selects a subset of the tokens. - The pieces produced by a chunker do not overlap in the source text.Segmentation and Labeling at both the Token and Chunk LevelsTag Representation of Chunk StructuresTree Representation of Chunk StructuresCredits: https://www.nltk.org/book/ch07.html  Chinks and ChinkingChink is a sequence of tokens that is not included in a chunk.Credits: https://www.nltk.org/book/ch07.html  Part-of-speech (POS) TaggingAssigning word types to tokens like verb or noun.POS tagging should be done straight after tokenization and before any words are removed so that sentence structure is preserved and it is more obvious what part of speech the word belongs to.<jupyter_code>text = "Asian shares skidded on Tuesday after a rout in tech stocks put Wall Street to the sword"
doc = nlp(text)
[(x.orth_, x.pos_, spacy.explain(x.pos_)) for x in [token for token in doc]]
[(x.orth_, x.tag_, spacy.explain(x.tag_)) for x in [token for token in doc]]
# using nltk
import nltk

tokens = nltk.word_tokenize(text)
pos_tags = nltk.pos_tag(tokens)
pos_tags<jupyter_output><empty_output><jupyter_text>BILUO tagging- BEGIN - The first token of a multi-token entity.- IN - An inner token of a multi-token entity.- LAST - The final token of a multi-token entity.- UNIT - A single-token entity.- OUT - A non-entity token.<jupyter_code>[(token, token.ent_iob_, token.ent_type_) for token in doc]<jupyter_output><empty_output><jupyter_text>StemmingStemming is the process of reducing words to their root form.Examples:- cats, catlike, catty → cat- fishing, fished, fisher → fishThere are two types of stemmers in NLTK: [Porter Stemmer](https://tartarus.org/martin/PorterStemmer/) and [Snowball stemmers](https://tartarus.org/martin/PorterStemmer/)[Credits](https://stackabuse.com/python-for-nlp-tokenization-stemming-and-lemmatization-with-spacy-library/)<jupyter_code>import nltk
from nltk.stem.porter import *

stemmer = PorterStemmer()
tokens = ['compute', 'computer', 'computed', 'computing']
for token in tokens:
    print(token + ' --> ' + stemmer.stem(token))<jupyter_output>compute --> comput
computer --> comput
computed --> comput
computing --> comput<jupyter_text>LemmatizationAssigning the base form of word, for example: - "was" → "be"- "rats" → "rat"<jupyter_code>doc = nlp("Was Google founded in early 1990?")
[(x.orth_, x.lemma_) for x in [token for token in doc]]<jupyter_output><empty_output><jupyter_text>Sentence DetectionFinding and segmenting individual sentences.<jupyter_code>doc = nlp("<NAME> founded Google in early 1990. <NAME> joined.")
[sent.text for sent in doc.sents]<jupyter_output><empty_output><jupyter_text>Dependency Parsing	Assigning syntactic dependency labels, describing the relations between individual tokens, like subject or object.<jupyter_code>doc = nlp("We are reading a text.")
# Dependency labels
[(x.orth_, x.dep_, spacy.explain(x.dep_)) for x in [token for token in doc]]
# Syntactic head token (governor)
[token.head.text for token in doc]<jupyter_output><empty_output><jupyter_text>Base noun phrases<jupyter_code>doc = nlp("I have a red car")
[chunk.text for chunk in doc.noun_chunks]<jupyter_output><empty_output><jupyter_text>Named Entity Recognition (NER)What is NER? Labeling "real-world" objects, like persons, companies or locations.2 popular approaches:- Rule-based- ML-based:    - Multi-class classification    - Conditional Random Field (probabilistic graphical model)Datasets:- [Kaggle, IOB, POS tags](https://www.kaggle.com/abhinavwalia95/entity-annotated-corpus/home)Credits: https://medium.com/@yingbiao/ner-with-bert-in-action-936ff275bc73Entities supported by spacy:- PERSON	People, including fictional.- NORP	Nationalities or religious or political groups.- FAC	Buildings, airports, highways, bridges, etc.- ORG	Companies, agencies, institutions, etc.- GPE	Countries, cities, states.- LOC	Non-GPE locations, mountain ranges, bodies of water.- PRODUCT	Objects, vehicles, foods, etc. (Not services.)- EVENT	Named hurricanes, battles, wars, sports events, etc.- WORK_OF_ART	Titles of books, songs, etc.- LAW	Named documents made into laws.- LANGUAGE	Any named language.- DATE	Absolute or relative dates or periods.- TIME	Times smaller than a day.- PERCENT	Percentage, including ”%“.- MONEY	Monetary values, including unit.- QUANTITY	Measurements, as of weight or distance.- ORDINAL	“first”, “second”, etc.- CARDINAL	Numerals that do not fall under another type. Alternatives to spacy[LexNLP](https://lexpredict-lexnlp.readthedocs.io/en/latest/modules/extract/extract.htmlpattern-based-extraction-methods) entities:- acts, e.g., “section 1 of the Advancing Hope Act, 1986”- amounts, e.g., “ten pounds” or “5.8 megawatts”- citations, e.g., “10 U.S. 100” or “1998 S. Ct. 1”- companies, e.g., “Lexpredict LLC”- conditions, e.g., “subject to …” or “unless and until …”- constraints, e.g., “no more than” or “- copyright, e.g., “(C) Copyright 2000 Acme”- courts, e.g., “Supreme Court of New York”- CUSIP, e.g., “392690QT3”- dates, e.g., “June 1, 2017” or “2018-01-01”- definitions, e.g., “Term shall mean …”- distances, e.g., “fifteen miles”- durations, e.g., “ten years” or “thirty days”- geographic and geopolitical entities, e.g., “New York” or “Norway”- money and currency usages, e.g., “$5” or “10 Euro”- percents and rates, e.g., “10%” or “50 bps”- PII, e.g., “212-212-2121” or “999-999-9999”- ratios, e.g.,” 3:1” or “four to three”- regulations, e.g., “32 CFR 170”- trademarks, e.g., “MyApp (TM)”- URLs, e.g., “http://acme.com/”Stanford NER entities:- Location, Person, Organization, Money, Percent, Date, TimeNLTK- NLTK maximum entropy classifier<jupyter_code>doc = nlp("<NAME> founded Google in the US in early 1990.")
# Text and label of named entity span
[(ent.text, ent.label_) for ent in doc.ents]
doc = nlp('European authorities fined Google a record $5.1 billion on Wednesday for abusing its power in the mobile phone market and ordered the company to alter its practices')
[(X.text, X.label_) for X in doc.ents]
from collections import Counter

labels = [x.label_ for x in doc.ents]
Counter(labels)
[(X, X.ent_iob_, X.ent_type_) for X in doc]
# Show Begin and In entities
items = [x.text for x in doc.ents]
print(items)
Counter(items).most_common(3)
import lexnlp.extract.en as lexnlp
import nltk
text = "There are ten cows in the 2 acre pasture."
print(list(lexnlp.amounts.get_amounts(text)))
import lexnlp.extract.en.acts
text = "test section 12 of the VERY Important Act of 1954."
lexnlp.extract.en.acts.get_act_list(text)<jupyter_output><empty_output><jupyter_text>Text ClassificationTwo types:- binary classification (text only belongs to one class)- multi-class classification (text can belong to multiple classes)Assigning categories or labels to a whole document, or parts of a document.Approach:- calculate document vectors for each document- use kNN to calculate clusters based on document vectors- each cluster represents a class of documents that are similar to each other<jupyter_code># Credits: https://pytorch.org/tutorials/beginner/text_sentiment_ngrams_tutorial.html
import re
from torchtext.data.utils import ngrams_iterator
from torchtext.data.utils import get_tokenizer

ag_news_label = {1 : "World",
                 2 : "Sports",
                 3 : "Business",
                 4 : "Sci/Tec"}

def predict(text, model, vocab, ngrams):
    tokenizer = get_tokenizer("basic_english")
    with torch.no_grad():
        text = torch.tensor([vocab[token]
                            for token in ngrams_iterator(tokenizer(text), ngrams)])
        output = model(text, torch.tensor([0]))
        return output.argmax(1).item() + 1

ex_text_str = "MEMPHIS, Tenn. – Four days ago, <NAME> was \
    enduring the season’s worst weather conditions on Sunday at The \
    Open on his way to a closing 75 at Royal Portrush, which \
    considering the wind and the rain was a respectable showing. \
    Thursday’s first round at the WGC-FedEx St. Jude Invitational \
    was another story. With temperatures in the mid-80s and hardly any \
    wind, the Spaniard was 13 strokes better in a flawless round. \
    Thanks to his best putting performance on the PGA Tour, Rahm \
    finished with an 8-under 62 for a three-stroke lead, which \
    was even more impressive considering he’d never played the \
    front nine at TPC Southwind."

vocab = train_dataset.get_vocab()
model = model.to("cpu")

print("This is a %s news" %ag_news_label[predict(ex_text_str, model, vocab, 2)])

# Output: This is a Sports news<jupyter_output><empty_output><jupyter_text>CountVectorizer - Convert a collection of text documents to a matrix of token counts- [skikitLearn](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html)  SimilarityHow similar are two documents, sentences, token or spans?Cosine similarity (also known as: L2-normalized dot product of vectors) is a formula used to calculate how similar two given word vectors are.How to calculate Cosine similarity?- spacy (see example below)- scikit: [sklearn.metrics.pairwise.cosine_similarity](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.pairwise.cosine_similarity.htmlsklearn.metrics.pairwise.cosine_similarity)  Cosine SimilarityFigure below shows three word vectors and Cosine distance (=similarity) between - "I hate cats" and "I love dogs" (result: not very similar)- "I love dogs" and "I love, love, love, .. dogs" (result: similar)[Credits](https://towardsdatascience.com/group-thousands-of-similar-spreadsheet-text-cells-in-seconds-2493b3ce6d8d)  L2 NormLength of a word vector. Also known as **Euclidean norm**.Example:- length of "I like cats" is 4.7<jupyter_code>doc1 = nlp("I like cats")
doc2 = nlp("I like dogs")
# Compare 2 documents
doc1.similarity(doc2)
# "cats" vs "dogs"
doc1[2].similarity(doc2[2])
# "I" vs "like dogs"
doc1[0].similarity(doc2[1:3])
doc = nlp("I like cats")
# L2 norm of "I like cats"
doc.vector_norm
# L2 norm of "cats"
doc[2].vector_norm
# Vector representation of "cats"
doc[2].vector
# can also be done using sklearn's linear kernel (equivilant to cosine similarity)<jupyter_output><empty_output><jupyter_text>n-grams: Unigram, bigrams, trigrams- Unigram = one word, eg the, and, of, hotel- Bigrams = two consecutive words, eg the hotel, in seattle, the city- Trigrams = three consecutive words, eg easy access to, high speed internet, the heart ofCredits: https://towardsdatascience.com/building-a-content-based-recommender-system-for-hotels-in-seattle-d724f0a32070  Get all unigrams<jupyter_code>from sklearn.feature_extraction.text import CountVectorizer

def get_top_n_words(corpus, n=None):
    vec = CountVectorizer(stop_words='english').fit(corpus)
    bag_of_words = vec.transform(corpus)
    sum_words = bag_of_words.sum(axis=0) 
    words_freq = [(word, sum_words[0, idx]) for word, idx in vec.vocabulary_.items()]
    words_freq = sorted(words_freq, key = lambda x: x[1], reverse=True)
    return words_freq[:n]

text = ["The quick brown fox jumped over the lazy dog."]

common_words = get_top_n_words(text, 20) # or use df['desc'] 

df2 = pd.DataFrame(common_words, columns = ['desc' , 'count'])
df2.groupby('desc').sum()['count'].sort_values().plot(kind='barh', title='Top 20 words in hotel description after removing stop words')<jupyter_output><empty_output><jupyter_text>Get all bigrams<jupyter_code>def get_top_n_bigram(corpus, n=None):
    vec = CountVectorizer(ngram_range=(2, 2), stop_words='english').fit(corpus)
    bag_of_words = vec.transform(corpus)
    sum_words = bag_of_words.sum(axis=0) 
    words_freq = [(word, sum_words[0, idx]) for word, idx in vec.vocabulary_.items()]
    words_freq =sorted(words_freq, key = lambda x: x[1], reverse=True)
    return words_freq[:n]
common_words = get_top_n_bigram(df['desc'], 20)
df4 = pd.DataFrame(common_words, columns = ['desc' , 'count'])
df4.groupby('desc').sum()['count'].sort_values(ascending=False).iplot(kind='bar', yTitle='Count', linecolor='black', title='Top 20 bigrams in hotel description After removing stop words')<jupyter_output><empty_output><jupyter_text>Get all trigrams<jupyter_code>def get_top_n_trigram(corpus, n=None):
    vec = CountVectorizer(ngram_range=(3, 3), stop_words='english').fit(corpus)
    bag_of_words = vec.transform(corpus)
    sum_words = bag_of_words.sum(axis=0) 
    words_freq = [(word, sum_words[0, idx]) for word, idx in vec.vocabulary_.items()]
    words_freq =sorted(words_freq, key = lambda x: x[1], reverse=True)
    return words_freq[:n]
common_words = get_top_n_trigram(df['desc'], 20)
df6 = pd.DataFrame(common_words, columns = ['desc' , 'count'])
df6.groupby('desc').sum()['count'].sort_values(ascending=False).iplot(kind='bar', yTitle='Count', linecolor='black', title='Top 20 trigrams in hotel description after removing stop words')<jupyter_output><empty_output><jupyter_text>Visualization<jupyter_code>from spacy import displacy
doc = nlp("This is a sentence")
displacy.render(doc, style="dep")
doc = nlp("Larry Page founded Google in the US in early 1990.")
displacy.render(doc, style="ent")<jupyter_output><empty_output><jupyter_text>Inspired by: https://www.datacamp.com/community/blog/spacy-cheatsheet  Kernels Used by - Support Vector Machines (SVMs)- Principal Component Analysis (PCA)Useful for- classification tasksAlso known as- kernel function- similarity functionOpposite of kernels: vectorsSource:- [Wikipedia](https://en.wikipedia.org/wiki/Kernel_method)  Linear KernelLinear Kernel is used when the data is Linearly separable, that is, it can be separated using a single Line.Compute the linear kernel between X and Y: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.pairwise.linear_kernel.htmlsklearn.metrics.pairwise.linear_kernel  Non-linear Kernel[Credits](https://en.wikipedia.org/wiki/Kernel_method)  Spearman's Rank Correlation CoefficientCredits: https://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient  kNNk-nearest neighbors algoritmUseful for- classification  Text Summarization- [How to Make a Text Summarizer](https://github.com/llSourcell/How_to_make_a_text_summarizer)- [How to Prepare News Articles for Text Summarization](https://machinelearningmastery.com/prepare-news-articles-text-summarization/)  Sentiment AnalysisIs text fact or opinion? Only perform sentiment analysis on opinion, not facts.Sentiments:- positive- neutral- negative2 ways:- rule-based uses lexicon with polarity score per word. Count positive and negative words. Doesn't provide training data.- automatic using machine learning (=classification problem). Needs training data.Sentiment analysis can be performed with ntlk's `SentimentIntensityAnalyzer`See: https://www.nltk.org/api/nltk.sentiment.htmlmodule-nltk.sentiment.vaderLearning resources: - https://www.youtube.com/watch?v=3Pzni2yfGUQ- https://towardsdatascience.com/sentiment-analysis-with-python-part-1-5ce197074184<jupyter_code>from nltk.sentiment.vader import SentimentIntensityAnalyzer
# nltk.download('vader_lexicon')<jupyter_output>/Users/jan/PycharmProjects/playground/nlp-cheat-sheet/venv/lib/python3.6/site-packages/nltk/twitter/__init__.py:20: UserWarning: The twython library has not been installed. Some functionality from the twitter package will not be available.
  warnings.warn("The twython library has not been installed. "<jupyter_text>Logistic RegressionA classification model that uses a sigmoid function to convert a linear model's raw prediction () into a value between 0 and 1. You can interpret the value between 0 and 1 in either of the following two ways:- As a probability that the example belongs to the positive class in a binary classification problem.- As a value to be compared against a classification threshold. If the value is equal to or above the classification threshold, the system classifies the example as the positive class. Conversely, if the value is below the given threshold, the system classifies the example as the negative class.https://developers.google.com/machine-learning/glossary/logistic-regression  RNNRecurrent neural networks- Size changes depending on input/output (in contrast to neural network like CNN) LSTMLong Short-Term MermoyToDo<jupyter_code>from keras.layers.recurrent import LSTM
from keras.models import Sequential
from keras.layers.core import Dense, Activation, Dropout, RepeatVector, Merge, TimeDistributedDense<jupyter_output><empty_output><jupyter_text>Levenshtein distance<jupyter_code>import Levenshtein<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10847847885022469, 'fraction_numerical': 0.01574627697106993, 'mean_word_length': 5.2945301542777, 'pattern_counts': {'":': 7, '<': 94, '<?xml version=': 0, '>': 105, 'https://': 62, 'lorem ipsum': 0, 'www.': 12, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '52817', 'license': 'FTL', 'n_tokens_mistral': 8815, 'n_tokens_neox': 8267, 'n_words': 2698, 'path': 'Resources/Penyisihan Resources/NLP-Cheat-Sheet.ipynb', 'repo_name': 'ardhanidzaky/gemastikxiv', 'size': '26942'}