<jupyter_start><jupyter_text>This study guide should reinforce and provide practice for all of the concepts you have seen in the past week. There are a mix of written questions and coding exercises, both are equally important to prepare you for the sprint challenge as well as to be able to speak on these topics comfortably in interviews and on the job.If you get stuck or are unsure of something remember the 20 minute rule. If that doesn't help, then research a solution with google and stackoverflow. Only once you have exausted these methods should you turn to your Team Lead - they won't be there on your SC or during an interview. That being said, don't hesitate to ask for help if you truly are stuck.Have fun studying!<jupyter_code>import pandas as pd
df = pd.read_csv('https://raw.githubusercontent.com/bundickm/Study-Guides/master/data/cannabis.csv')
print('Shape:', df.shape, '\n')
df.head()<jupyter_output>Shape: (2351, 6)<jupyter_text>Tokens  Definitions Define the following terms in your own words, do not simply copy and paste a definition found elsewhere but reword it to be understandable and memorable to you. *Double click the markdown to add your definitions.***Natural Language Processing**: `Your Answer Here`**Token**: `Your Answer Here`**Corpus**: `Your Answer Here`**Stopwords**: `Your Answer Here`**Statistical Trimming**: `Your Answer Here`**Stemming**: `Your Answer Here`**Lemmatization**: `Your Answer Here`**Vectorization**: `Your Answer Here`  Questions of Understanding 1. What are at least 4 common cleaning tasks you need to do when creating tokens? 1. `Your Answer Here` 2. `Your Answer Here` 3. `Your Answer Here` 4. `Your Answer Here`2. Why is it important to apply custom stopwords to our dataset in addition to the ones that come in a library like spaCy?```Your Answer Here```3. Explain the tradeoffs between statistical trimming, stemming, and lemmatizing.```Your Answer Here```4. Why do we need to vectorize our documents?```Your Answer Here```  Practice Problems Write a function to tokenize the `Description` column. Make sure to include the following:- Return the tokens in an iterable structure- Normalize the case- Remove non-alphanumeric characters such as punctuation, whitespace, unicode, etc.- Apply stopwords and make sure to add stopwords specific to this dataset- Lemmatize the tokens before returning them<jupyter_code><jupyter_output><empty_output><jupyter_text>Apply your function to `Description` and save the resulting tokens in a new column, `Tokens`<jupyter_code><jupyter_output><empty_output><jupyter_text>Use the function below to create a `word_count` dataframe based off the `df['Tokens']` column you created.<jupyter_code>def count(docs):
        word_counts = Counter()
        appears_in = Counter()
        total_docs = len(docs)

        for doc in docs:
            word_counts.update(doc)
            appears_in.update(set(doc))

        temp = zip(word_counts.keys(), word_counts.values())
        wc = pd.DataFrame(temp, columns = ['word', 'count'])

        wc['rank'] = wc['count'].rank(method='first', ascending=False)
        total = wc['count'].sum()

        wc['pct_total'] = wc['count'].apply(lambda x: x / total)
        
        wc = wc.sort_values(by='rank')
        wc['cul_pct_total'] = wc['pct_total'].cumsum()

        t2 = zip(appears_in.keys(), appears_in.values())
        ac = pd.DataFrame(t2, columns=['word', 'appears_in'])
        wc = ac.merge(wc, on='word')

        wc['appears_in_pct'] = wc['appears_in'].apply(lambda x: x / total_docs)
        
        return wc.sort_values(by='rank')<jupyter_output><empty_output><jupyter_text>Run the line of code below, and then explain how to interpret the graph.```Your Answer Here```<jupyter_code>sns.lineplot(x='rank', y='cul_pct_total', data=word_count);<jupyter_output><empty_output><jupyter_text>Vectorization  Definitions Define the following terms in your own words, do not simply copy and paste a definition found elsewhere but reword it to be understandable and memorable to you. *Double click the markdown to add your definitions.***Vectorization**: `Your Answer Here`**Document Term Matrix (DTM)**: `Your Answer Here`**Latent Semantic Analysis**: `Your Answer Here`**Term Frequency - Inverse Document Frequency (TF-IDF)**: `Your Answer Here`**Word Embedding**: `Your Answer Here`**N-Gram**: `Your Answer Here`**Skip-Gram**: `Your Answer Here`  Questions of Understanding 1. Why do we need to vectorize our documents?```Your Answer Here```2. How is TF-IDF different from simple word frequency? Why do we use TF-IDF over word frequency?```Your Answer Here```3. Why might we choose a word embedding approach over a bag-of-words approach when it comes to vectorization?```Your Answer Here```  Practice Problems Use the dataframe `df` above to complete the following. Vectorize the `Tokens` column.<jupyter_code><jupyter_output><empty_output><jupyter_text>Build a Nearest Neighbors model from your dataframe and then find the 5 nearest neighbors to the strain "100-OG"<jupyter_code><jupyter_output><empty_output><jupyter_text>You will be putting together a classification model below, but before you do you'll need a baseline. Run the line of code below and then find the normalized value counts for the `Rating` column in `df`.<jupyter_code>df['Rating'] = df['Rating'].round().astype(int)<jupyter_output><empty_output><jupyter_text>What is the baseline accuracy?```Your Answer Here```Visualize the rating counts from above<jupyter_code><jupyter_output><empty_output><jupyter_text>Use your vectorized tokens in the `df` dataframe to train a classification model<jupyter_code><jupyter_output><empty_output><jupyter_text>Predict the score of the fake strain description below.```'Afgooey, also known as Afgoo, is a potent indica strain that is believed to descend from an Afghani indica and Maui Haze. Its sativa parent may lend Afgoo some uplifting, creative qualities, but this strain undoubtedly takes after its indica parent as it primarily delivers relaxing, sleepy effects alongside its earthy pine flavor. Growers hoping to cultivate Afgoo may have a better chance of success indoors, but this indica can also thrive in Mediterranean climates outdoors.'```<jupyter_code><jupyter_output><empty_output><jupyter_text>Topic Modeling  Questions of Understanding 1. What is Latent Dirichlet Allocation? What is another name for LDA in NLP?```Your Answer Here```2. How do interpret the results of a topic modeling output?```Your Answer Here```  Practice Problems Find the top 5 topics of the `Description` column using LDA<jupyter_code><jupyter_output><empty_output><jupyter_text>In a short paragraph, explain how to interpret the first topic your model came up with. If your topic words are difficult to interpret, explain how you could clean up the descriptions to improve your topics```Your Answer Here``` Use `pyLDAvis` to create a visualization to help you interpret your topic modeling results<jupyter_code><jupyter_output><empty_output><jupyter_text>Explain how to interpret the results of `pyLDAvis````Your Answer Here``` Create at least 1 more visualization to help you interpret the results of your topic modeling<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09054353854686634, 'fraction_numerical': 0.004021075984470327, 'mean_word_length': 5.316112084063048, 'pattern_counts': {'":': 0, '<': 56, '<?xml version=': 0, '>': 56, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '201673', 'license': 'MIT', 'n_tokens_mistral': 2067, 'n_tokens_neox': 1957, 'n_words': 848, 'path': 'Unit_4_Sprint_1_Natural_Language_Processing_Study_Guide.ipynb', 'repo_name': 'WhyseRabbit/Study-Guides', 'size': '7212'}