<jupyter_start><jupyter_text>Lambda School Data Science*Unit 4, Sprint 1, Module 3*---  Document Classification (Assignment)This notebook is for you to practice skills during lecture.Today's guided module project and assignment will be different. You already know how to do classification. You ready know how to extract features from documents. So? That means you're ready to combine and practice those skills in a kaggle competition. We we will open with a five minute sprint explaining the competition, and then give you 25 minutes to work. After those twenty five minutes are up, I will give a 5-minute demo an NLP technique that will help you with document classification (*and **maybe** the competition*).Today's all about having fun and practicing your skills. Sections* Part 1: Text Feature Extraction & Classification Pipelines* Part 2: Latent Semantic Indexing* Part 3: Word Embeddings with Spacy* Part 4: Post Lecture Assignment  Text Feature Extraction & Classification Pipelines (Learn)  Follow Along What you should be doing now:1. Join the Kaggle Competition2. Download the data3. Train a model (try using the pipe method I just demoed)  Load Competition Data<jupyter_code>import pandas as pd

# You may need to change the path
train = pd.read_csv('./data/train.csv')
test = pd.read_csv('./data/test.csv')
train.head()<jupyter_output><empty_output><jupyter_text>Define Pipeline Components<jupyter_code>vect = ...
clf = ...

pipe = Pipeline([('vect', vect), ('clf', clf)])<jupyter_output><empty_output><jupyter_text>Define Your Search SpaceYou're looking for both the best hyperparameters of your vectorizer and your classification model.<jupyter_code>parameters = {
    'vect__max_df': (0.75, 1.0),
    'clf__max_depth':(5,10,15,20)
}

grid_search = GridSearchCV(pipe,parameters, cv=5, n_jobs=4, verbose=1)
grid_search.fit(..., ...)<jupyter_output><empty_output><jupyter_text>Make a Submission File*Note:* In a typical Kaggle competition, you are only allowed two submissions a day, so you only submit if you feel you cannot achieve higher test accuracy. For this competition the max daily submissions are capped at **20**. Submit for each demo and for your assignment.<jupyter_code># Predictions on test sample
pred = grid_search.predict(test['description'])
submission = pd.DataFrame({'id': test['id'], 'ratingCategory':pred})
submission['ratingCategory'] = submission['ratingCategory'].astype('int64')
# Make Sure the Category is an Integer
submission.head()
subNumber = 0
# Save your Submission File
# Best to Use an Integer or Timestamp for different versions of your model

submission.to_csv(f'./data/submission{subNumber}.csv', index=False)
subNumber += 1<jupyter_output><empty_output><jupyter_text>ChallengeYou're trying to achieve a minimum of 70% Accuracy on your model.  Latent Semantic Indexing (Learn)  Follow Along1. Join the Kaggle Competition2. Download the data3. Train a model & try:     - Creating a Text Extraction & Classification Pipeline    - Tune the pipeline with a `GridSearchCV` or `RandomizedSearchCV`    - Add some Latent Semantic Indexing (lsi) into your pipeline. *Note:* You can grid search a nested pipeline, but you have to use double underscores ie `lsi__svd__n_components`4. Make a submission to Kaggle   Define Pipeline Components<jupyter_code>lsi = ...
vect = ...
clf = ...

pipe = Pipeline([('lsi', lsi), ('clf', clf)])<jupyter_output><empty_output><jupyter_text>Define Your Search SpaceYou're looking for both the best hyperparameters of your vectorizer and your classification model.<jupyter_code>parameters = {
    'lsi__svd__n_components': [10,100,250],
    'vect__max_df': (0.75, 1.0),
    'clf__max_depth':(5,10,15,20)
}

grid_search = GridSearchCV(pipe,parameters, cv=5, n_jobs=4, verbose=1)
grid_search.fit(..., ...)<jupyter_output><empty_output><jupyter_text>Make a Submission File<jupyter_code># Predictions on test sample
pred = grid_search.predict(test['description'])
submission = pd.DataFrame({'id': test['id'], 'ratingCategory':pred})
submission['ratingCategory'] = submission['ratingCategory'].astype('int64')
# Make Sure the Category is an Integer
submission.head()
# Save your Submission File
# Best to Use an Integer or Timestamp for different versions of your model

submission.to_csv(f'./data/submission{subNumber}.csv', index=False)
subNumber += 1<jupyter_output><empty_output><jupyter_text>ChallengeContinue to apply Latent Semantic Indexing (LSI) to various datasets.   Word Embeddings with Spacy (Learn)  Follow Along<jupyter_code># Apply to your Dataset

from sklearn.model_selection import RandomizedSearchCV
from sklearn.ensemble import GradientBoostingClassifier

from scipy.stats import randint

param_dist = {
    
    'max_depth' : randint(3,10),
    'min_samples_leaf': randint(2,15)
}
# Continue Word Embedding Work Here<jupyter_output><empty_output><jupyter_text>Make a Submission File<jupyter_code># Predictions on test sample
pred = ...predict(test['description'])
submission = pd.DataFrame({'id': test['id'], 'ratingCategory':pred})
submission['ratingCategory'] = submission['ratingCategory'].astype('int64')
# Make Sure the Category is an Integer
submission.head()
# Save your Submission File
# Best to Use an Integer or Timestamp for different versions of your model

submission.to_csv(f'./data/submission{subNumber}.csv', index=False)
subNumber += 1<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10124697561883492, 'fraction_numerical': 0.013958682300390842, 'mean_word_length': 5.845859872611465, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '90714', 'license': 'MIT', 'n_tokens_mistral': 1666, 'n_tokens_neox': 1579, 'n_words': 558, 'path': 'module3-document-classification/LS_DS_413_Document_Classification_Assignment.ipynb', 'repo_name': 'EvidenceN/Unit4_Lambda_DS', 'size': '5373'}