<jupyter_start><jupyter_text>Using spacy: [Installation instructions for spacy](https://spacy.io/docs/usage/)<jupyter_code>import spacy
# model meta data
spacy.info('en_core_web_sm')
# preload the language model
nlp = spacy.load('en_core_web_sm')
review_df['text'][:2]
# Keeping it in a pandas dataframe
doc_df = review_df['text'].apply(nlp)

type(doc_df)
doc_df[:3]
type(doc_df[0])
doc_df[2]
# spacy gives you both fine grained (.pos_) + coarse grained (.tag_) parts of speech    
for doc in doc_df[2]:
    print(doc.text, doc.pos_, doc.tag_)
# spaCy also does noun chunking for us

print([chunk for chunk in doc_df[2].noun_chunks])<jupyter_output>[I, WVM, the recommendation, a couple, friends, who, them, the past, they, a nice job, I, a fan, <NAME>, time, my move, They, all my things, the old place, less than 2.5 hours, my (heavy) furniture, perfect condition, they, extra care, the wood floors, the process, I, I, I, WVM]<jupyter_text>Using [Textblob](https://textblob.readthedocs.io/en/dev/)<jupyter_code>from textblob import TextBlob<jupyter_output><empty_output><jupyter_text>The default tagger in TextBlob uses the PatternTagger, the same as [pattern](https://www.clips.uantwerpen.be/pattern), which is fine for our example. To use the NLTK tagger, we can specify the pos_tagger when we call TextBlob. More [here](http://textblob.readthedocs.io/en/dev/advanced_usage.htmladvanced).<jupyter_code>blob_df = review_df['text'].apply(TextBlob)

type(blob_df)
type(blob_df[2])
blob_df[2].tags
# blobs in TextBlob also have noun phrase extraction

print([np for np in blob_df[2].noun_phrases])<jupyter_output>['wvm', 'nice job', 'evan', 'cody', 'past weekend', 'old place', 'perfect condition', 'extra care', 'wood floors', 'wvm']
{'fraction_non_alphanumeric': 0.13437139561707034, 'fraction_numerical': 0.006343713956170703, 'mean_word_length': 6.081632653061225, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 12, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '200189', 'license': 'MIT', 'n_tokens_mistral': 635, 'n_tokens_neox': 614, 'n_words': 180, 'path': '03.02_Chunking_and_POS_Tagging.ipynb', 'repo_name': 'sondywoo/feature-engineering-book', 'size': '1738'}