<jupyter_start><jupyter_text>Exploring Text Data (2) PyConUK talk abstractData set of abstracts for the PyConUK 2016 talks (retrieved 14th Sept 2016 from https://github.com/PyconUK/2016.pyconuk.org)The data can be found in `../data/pyconuk2016/{keynotes,workshops,talks}/*`There are 101 abstracts  Load the dataFirstly, we load all the data into the `documents` dictionaryWe also merge the documents into one big string, `corpus_all_in_one`, for convenience<jupyter_code>import os

data_dir = os.path.join('..', 'data', 'pyconuk2016')

talk_types = ['keynotes', 'workshops', 'talks']

all_talk_files = [os.path.join(data_dir, talk_type, fname)
                  for talk_type in talk_types
                  for fname in os.listdir(os.path.join(data_dir, talk_type))]

documents = {}
for talk_fname in all_talk_files:
    bname = os.path.basename(talk_fname)
    talk_title = os.path.splitext(bname)[0]
    with open(talk_fname, 'r') as f:
        content = f.read()
        documents[talk_title] = content
        
corpus_all_in_one = ' '.join([doc for doc in documents.values()])
        
print("Number of talks: {}".format(len(all_talk_files)))
print("Corpus size (char): {}".format(len(corpus_all_in_one)))
%matplotlib inline
import matplotlib.pyplot as plt
from wordcloud import WordCloud

cloud = WordCloud(max_words=100)
cloud.generate_from_text(corpus_all_in_one)

plt.figure(figsize=(12,8))
plt.imshow(cloud)
plt.axis('off')
plt.show()
all_talk_files[0]
%cat {all_talk_files[0]}

# For a list of magics type:
# %lsmagic
documents = {}
for talk_fname in all_talk_files:
    bname = os.path.basename(talk_fname)
    talk_title = os.path.splitext(bname)[0]
    with open(talk_fname, 'r') as f:
        content = ""
        for line in f:
            if line.startswith('title:'):
                line = line[6:]
            if line.startswith('subtitle:') \
            or line.startswith('speaker:') \
            or line.startswith('---'):
                continue
            content += line
        documents[talk_title] = content

corpus_all_in_one = ' '.join([doc for doc in documents.values()])
%matplotlib inline
import matplotlib.pyplot as plt
from wordcloud import WordCloud

cloud = WordCloud(max_words=100)
cloud.generate_from_text(corpus_all_in_one)

plt.figure(figsize=(12,8))
plt.imshow(cloud)
plt.axis('off')
plt.show()
from collections import Counter
import string
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords

stop_list = stopwords.words('english') + list(string.punctuation)

document_frequency = Counter()

for talk_id, content in documents.items():
    try:  # py3
        tokens = word_tokenize(content)
    except UnicodeDecodeError:  # py27
        tokens = word_tokenize(content.decode('utf-8'))
    unique_tokens = [token.lower() for token in set(tokens)
                     if token.lower() not in stop_list]
    document_frequency.update(unique_tokens)

for word, freq in document_frequency.most_common(20):
    print("{}\t{}".format(word, freq))
# print(stop_list)
for item in ['will', "'ll", 'll']:
    print("{} in stop_list == {}".format(item, item in stop_list))
from nltk import ngrams

try:
    all_tokens = [t for t in word_tokenize(corpus_all_in_one)]
except UnicodeDecodeError:
    all_tokens = [t for t in word_tokenize(corpus_all_in_one.decode('utf-8'))]

bigrams = ngrams(all_tokens, 2)
trigrams = ngrams(all_tokens, 3)

bi_count = Counter(bigrams)
tri_count = Counter(trigrams)

for phrase, freq in bi_count.most_common(20):
    print("{}\t{}".format(phrase, freq))
    
for phrase, freq in tri_count.most_common(20):
    print("{}\t{}".format(phrase, freq))<jupyter_output><empty_output><jupyter_text>Term Frequency (TF)TF provides a weight of a term within a document, based on the term frequencyTF(term, doc) = count(term in doc)TF(term, doc) = count(term in doc) / len(doc) Inverse Document Frequency (IDF)IDF provides a weight of a term across the collection, based on the document frequency of such termIDF(term) = log( N / DF(term) )IDF(term) = log( 1 + N / DF(term) )  Introducing sklearnSo far, we have used some homemade implementation to count wordsWhat if we need something more involved?sklearn (http://scikit-learn.org/) is one of the main libraries for Machine Learning in PythonWith an easy-to-use interface, it provides support for a variety of Machine Learning modelsWe're going to use it to tackle a Text Classification problem<jupyter_code>from random import randint

winner = randint(1, 36)

print("And the winner is ... {}".format(winner))
from nltk import pos_tag
from nltk.tokenize import word_tokenize

s = "The quick brown fox juped over the dog"
tokens = word_tokenize(s)
tokens

pos_tag(tokens)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11458773784355179, 'fraction_numerical': 0.012684989429175475, 'mean_word_length': 4.204620462046204, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '9341', 'license': 'MIT', 'n_tokens_mistral': 1643, 'n_tokens_neox': 1560, 'n_words': 439, 'path': 'notebooks/02 pyconuk_exploratory_analysis.ipynb', 'repo_name': 'bibs440v/NLP', 'size': '4730'}