<jupyter_start><jupyter_text>TOC 0. **Text Preprocesing**1. **Feature Engineering** (Parsing, Entity Extraction, Statistical Features such as Tf-IDF, Word Embeddings - gentle intro, later more), N-grams and more.2. **Stemming**3. **Word Embeddings**4. **Part of speech Tagging**5. **Named entity Disabiguation**6. **Named entity Recognition**7. **Sentiment Analysis**8. **Semantic Text Similarity**9. **Language Identification**10. **Text Summarization**  1. Text PreprocessingThis involves three simple steps, namely:- Denoising- Normalizing Lexicon- Standardizing Objects**DeNoising**Here we get rid of language stopwords -- as you've seen in other tutorials that sometimes they're helpful(multi-lingual stuff), other times not.Let's Denoise some stuff...<jupyter_code>noise_stuff = ['is', 'the', 'a', '...']
def denoise(input_text):
    words = input_text.split()
    noise_free_words = [word for word in words if word not in noise_stuff]
    noise_free_text = " ".join(noise_free_words)
    return noise_free_text
denoise('the cat is a real clown and it drives me ...')<jupyter_output><empty_output><jupyter_text>**Normalizing Lexicon**Here we touched also briefly about words such beauty, beautiful, beautify, which are all variations of beauty or more specifically "beaut" - which is what Stemming does- **Stemming** : This is a rudimentary rule-based process that strips the suffixes- **Lemmetization**: does something else. It goes step-by-step in an organized manner to to obtain root of the word.Let's take a look...<jupyter_code># We import them both
from nltk.stem.wordnet import WordNetLemmatizer
from nltk.stem.porter import PorterStemmer

# Assign
lemm = WordNetLemmatizer()
stemm = PorterStemmer()

word = "beautifying"
print(lemm.lemmatize(word, 'v'))
print(stemm.stem(word))<jupyter_output>beautify
beautifi<jupyter_text>**Standardizing Object**Text data might contain word or phrases which do not exist in dictionaries and may not find their way to search engines etc. Think of acronyms, hashtags, some other new trending stuff on the internet. [Note: urban dictionary does capture a lot of stuff lately though]<jupyter_code>lookup_dict = {'rt': 'Retweet', 'dm':'Direct Message', 'lol':"Laughing out loud", 'wtf': 'What the funk'}

def lookup_words(input_text):
    words = input_text.split()
    new_words = []
    for word in words:
        if word.lower() in lookup_dict:
            word = lookup_dict[word.lower()]
        new_words.append(word)
        new_text = " ".join(new_words)
        return new_text
lookup_words("wtf dude I sent you a dm but you never replied")<jupyter_output><empty_output><jupyter_text>2. Feature EngineeringHere we will discuss and play around a bit with the following:- Parsing- Entity extraction- Term Frequency / Inverse Document Frequency (TF-IDF)- Word Embedding (gentle intro, more later)-   What are N-grams Coming soon<jupyter_code>def generate_ngrams(text, n):
    words = text.split()
    output = []  
    for i in range(len(words)-n+1):
        output.append(words[i:i+n])
    return output

generate_ngrams('this is a sample text', 2)
# [['this', 'is'], ['is', 'a'], ['a', 'sample'], , ['sample', 'text']] 
# Play around with parameters N<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11499227202472952, 'fraction_numerical': 0.004945904173106646, 'mean_word_length': 4.872958257713249, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '533324', 'license': 'Apache-2.0', 'n_tokens_mistral': 1022, 'n_tokens_neox': 978, 'n_words': 347, 'path': 'deep-learning/NLP/NLP Tutorials - Extended & ultimate Version 😀.ipynb', 'repo_name': 'timgates42/Artificial-Intelligence-Deep-Learning-Machine-Learning-Tutorials', 'size': '3235'}