<jupyter_start><jupyter_text>TEXT PREPROCESSING - TOKENIZATION Text preprocessing is a very integral part of any Natural Language Processing. Although we currently have immensely powerful methods of text preprocessing, an intuition about the traditional methods will be a solid foundation to understanding the things to come. The inital text preprocessing concept we will be discussing is **Tokenization**. It is bascially breaking down text into smaller pieces - a.k.a tokens. This helps in developing a NLP model or understanding the semantics of the textual data. _Here is an example of tokenization_| Before Tokenization | After Tokenization || --- | --- | | "This is a test sentence" | 'This' 'is' 'a' 'test' 'sentence' |We will be using NLTK and Keras Libraries to illustrate the concepts. Will be adding Gensim, Spacy, TextBlob examples later on.<jupyter_code>#Natural Language Tool Kit (NLTK) is a NLP package for text data processing
import nltk

paragraph = """Paragraphs are the building blocks of papers. Many students define paragraphs \
in terms of length. A paragraph is a group of at least five sentences. Paragraph \
is half a page long, etc."""

#nltk.sent_tokenize() lists the sentences in raw data
sentences = nltk.sent_tokenize(paragraph)
print(sentences)
#nltk.word_tokenize() lists the sentences in raw data
words = nltk.word_tokenize(paragraph)
print(words)
#nltk.tweet_tokenize() for tokenizing tweets - handles emojis and hashtags

tweet = " Martinelli to start today <3. Hope he recreates magic at #StamfordBridge #ARSCHE #COYG"

tweet_tokenizer = nltk.TweetTokenizer()
tweet_tokens = tweet_tokenizer.tokenize(tweet)
print(tweet_tokens)

normal_tokens = nltk.word_tokenize(tweet)
print(normal_tokens)
#nltk.RegexpTokenizer() for splits the text into tokens based on regex

sample_sentence = "I earned $400 as winnings of Microsoft Hackathon."
regex_tokenizer = nltk.RegexpTokenizer('\w+|\$[\d\.]+|\S+')
regex_tokens = regex_tokenizer.tokenize(sample_sentence)
print(regex_tokens)

normal_tokens = nltk.word_tokenize(sample_sentence)
print(normal_tokens)
#Tokenization can also be done by using keras - a deep learning API written in Python
import keras
from keras.preprocessing.text import text_to_word_sequence

words = text_to_word_sequence(paragraph)
print(words)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07361170899698666, 'fraction_numerical': 0.0017219113215669393, 'mean_word_length': 5.64, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '229579', 'license': 'MIT', 'n_tokens_mistral': 697, 'n_tokens_neox': 670, 'n_words': 268, 'path': 'natural-language-processing/text-preprocessing-level-1/tokenization.ipynb', 'repo_name': 'naagarjunsa/data-science-portfolio', 'size': '2323'}