<jupyter_start><jupyter_text>2. Use the LabelEncoder to convert the group names to numeric labels<jupyter_code>from sklearn import preprocessing
le = preprocessing.LabelEncoder()
le.fit(newsgroups_df['group'])
le.transform(newsgroups_df['group'])
newsgroups_df['newsgroups_label'] = le.transform(newsgroups_df['group'])
newsgroups_df.head(3)
newsgroups_df.tail(3)<jupyter_output><empty_output><jupyter_text>3. Pick out 10 words or phrases to use as manually created features. Doing an 80/20 train/test split, how well does a Naive Bayes classifier do? atheismgraphicsmotorcyclesbaseballhockeycryptmedspacegunsmideast<jupyter_code>newsgroups_df['has_atheism'] = newsgroups_df['content'].str.contains('atheism')
newsgroups_df['has_graphics'] = newsgroups_df['content'].str.contains('graphics')
newsgroups_df['has_motorcycles'] = newsgroups_df['content'].str.contains('motocycles')
newsgroups_df['has_baseball'] = newsgroups_df['content'].str.contains('baseball')
newsgroups_df['has_hockey'] = newsgroups_df['content'].str.contains('hockey')
newsgroups_df['has_crypt'] = newsgroups_df['content'].str.contains('crypt')
newsgroups_df['has_med'] = newsgroups_df['content'].str.contains('med')
newsgroups_df['has_space'] = newsgroups_df['content'].str.contains('space')
newsgroups_df['has_guns'] = newsgroups_df['content'].str.contains('guns')
newsgroups_df['has_mideast'] = newsgroups_df['content'].str.contains('mideast')
newsgroups_df.columns
from sklearn.cross_validation import train_test_split

nbx_train, nbx_test, nby_train, nby_test = train_test_split(
    newsgroups_df[['has_atheism', 'has_graphics', 'has_motorcycles', 'has_baseball',
       'has_hockey', 'has_crypt', 'has_med', 'has_space', 'has_guns',
       'has_mideast']], # the first is our FEATURES
    newsgroups_df['newsgroups_label'], # the second parameter is the LABEL (0-16, southern us, brazilian, anything really)
    test_size=0.2) # 80% training, 20% testing
from sklearn import naive_bayes

clf = naive_bayes.BernoulliNB()
clf.fit(nbx_train, nby_train)
clf.score(nbx_train, nby_train)
clf.score(nbx_test, nby_test)<jupyter_output><empty_output><jupyter_text>4. Use a CountVectorizer to automatically create your list of features. Doing an 80/20 train/test split, how well can a Naive Bayes classifier do?<jupyter_code>from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer()
#kernel dies without max features
vectorizer.fit(newsgroups_df['content'])
all_word_features = vectorizer.transform(newsgroups_df['content'])
all_word_features
x_train, x_test, y_train, y_test = train_test_split(
    all_word_features,
    newsgroups_df['content'], 
    test_size=0.2)
clf = naive_bayes.BernoulliNB()
clf.fit(x_train, y_train)
clf.score(x_train, y_train)
clf.score(x_test, y_test)
# So Naive Bayes did better with the manually selected features.<jupyter_output><empty_output><jupyter_text>5. PUSH THAT SCORE UP! You can adjust ngrams, max_features and any other options of the vectorizer, or try a decision tree or any other type of classifier. bc I couldn't run the Count Vectorizer without max_features and ngrams don't make sense here, bc it is more about single words(?), I would like to try Random Forests<jupyter_code>from sklearn.ensemble import RandomForestClassifier
tree_clf = RandomForestClassifier()
tree_clf.fit(nbx_train, nby_train)
print("Training score:", tree_clf.score(nbx_train, nby_train))

print("Testing score:", tree_clf.score(nbx_test, nby_test))
#Same results for Naive Bayes and Random forest?<jupyter_output><empty_output><jupyter_text>6. Write 15 sentences that, when run against the predictor, are put in 15 separate newsgroups (list the names of the newsgroups).<jupyter_code>sentences = [
    "I believe atheism is also a religion",
    "I love graphics",
    "We rode a motorcycle on the weekend",
    "I never played or watched baseball",
    "Who likes hockey, really?!"
    "You would want to encrypt your emails",
    "I really love the pharma industry for all the medicine provided",
    "How does medical care in space work out?",
    "Do you have a collection of guns?",
    "What's your favourite country in the mideast?",
    "The US should pass a anti-gun law to control weapon sales",
    "You should teach how to crypt in schools",
    "Have you ever fired guns?",
    "These are really cool graphics. How did you make them?",
    "Mideast politics are really complicated."
]
sentences_words_features = vectorizer.transform(sentences)
sentences_words_features
clf = naive_bayes.BernoulliNB()
clf.fit(vectorizer.transform(newsgroups_df['content']), newsgroups_df['newsgroups_label'])
predictions = clf.predict(sentences_words_features)
predictions
le.inverse_transform(predictions)
# This is predicting badly, but it is also with the training data, so it is consistent in a way. 
# I'd assume that this is a case for improving training data, to aslo improve the resule 
# (ie go for word combinations instead of single words.)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10362072414482897, 'fraction_numerical': 0.00640128025605121, 'mean_word_length': 6.633587786259542, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '316202', 'license': 'MIT', 'n_tokens_mistral': 1684, 'n_tokens_neox': 1564, 'n_words': 447, 'path': '11 Naive Bayes/homework/.ipynb_checkpoints/Gruen_Gianna_11_1-checkpoint.ipynb', 'repo_name': 'barjacks/algorithms_mine', 'size': '4999'}