<jupyter_start><jupyter_text>Q1. Country namesFor the first question you will be making use of regex. In particular you have a list of countries and you'll have to answer some very specific questions about that list.Start by loading the defining the path to this list<jupyter_code>path = "data/countries.txt"<jupyter_output><empty_output><jupyter_text>The first thing you will build is a wrapper that will apply a regex pattern into a given file, and return a list of results found matching that pattern. Implement it in the function below:<jupyter_code>def find_all_in_file(pattern, path):
    """
    Function that returns all matches of a certain pattern in a certain text.
    
    Args:
    pattern - regex pattern
    path - path to the file
    """
    
    # YOUR CODE HERE
    raise NotImplementedError()<jupyter_output><empty_output><jupyter_text>Make sure this function is working with the following tests:<jupyter_code>assert find_all_in_file(pattern="^P.+?$", path=path)[8] == "Portugal"
assert find_all_in_file(pattern="^.+?a$", path=path)[18] == "Croatia"
assert len(find_all_in_file(pattern="^.+?ca$", path=path)) == 4<jupyter_output><empty_output><jupyter_text>Q1.a)Now that you prepared your wrapper, let's move on to the actual expressions. The first thing we are looking for is for countries with loooong names. In particular we want you to find all the countries with more than 15 letters. Use the wrapper you defined above and assign its return to a variable `ret`.<jupyter_code># ret_long = find_all_in_file(...)

# YOUR CODE HERE
raise NotImplementedError()
print("Number of countries with more than 15 or more letters: ", len(ret_long))
assert len(ret_long) == 16<jupyter_output><empty_output><jupyter_text>Q1.b)Now, find out how many countries:* Start with a vowel* Start with a consonant<jupyter_code># ret_vowel = find_all_in_file(...)

# YOUR CODE HERE
raise NotImplementedError()
print("Number of countries that start with vowels: " , len(ret_vowel))
assert len(ret_vowel) == 36
# ret_consonant = find_all_in_file(...)

# YOUR CODE HERE
raise NotImplementedError()
print("Number of countries that start with consonants: " , len(ret_consonant))
assert len(ret_consonant) == 160<jupyter_output><empty_output><jupyter_text>Q1.c)Next, find how many countries are composed by only one word and end in `ia`. You'll want to have a list with countries such as `Croatia`, `Serbia`, etc.<jupyter_code># ret_ia = find_all_in_file(...)

# YOUR CODE HERE
raise NotImplementedError()
print("Number of variants of countries ending in \"ia\": " , len(ret_ia))
assert "Serbia" in ret_ia
assert "Croatia" in ret_ia
assert len(ret_ia) == 35<jupyter_output><empty_output><jupyter_text>Q1.d)Finally, find the countries which have at least four consecutive consonants, without taking into account the first letter (Hint: you can assume the first letter is capitalized). So, it should match things like `Abcdf`.<jupyter_code># ret_bcdf = find_all_in_file(...)

# YOUR CODE HERE
raise NotImplementedError()
print("Number of countries matched: " , len(ret_bcdf))
assert len(ret_bcdf) == 3
assert hashlib.sha256(' '.join(ret_bcdf).encode()).hexdigest() == '7da1a15074b9245ae3b88fb92fc5c484243003a084d03280bf11d9346d768869'<jupyter_output><empty_output><jupyter_text>Q2. A Study in ScarletFor this following questions we will be looking at <NAME>'s ["A Study in Scarlet"](https://en.wikipedia.org/wiki/A_Study_in_Scarlet) (which you might have seen adapted to tv in ["A Study in Pink"](https://en.wikipedia.org/wiki/A_Study_in_Pink)). We will be performing common preprocessing operations on this text, as it is a common task in Natural Language Processing. Start by downloading the data and loading it into a list of sentences:<jupyter_code>path = "data/sherlock.txt"
data =  [line.strip('\n') for line in open(path, 'r', encoding='utf8') if len(line)>1]<jupyter_output><empty_output><jupyter_text>Q2.a)First tokenize the data. Implement the function to receive an NLTK-style tokenizer and return the token list for each sentence:<jupyter_code>def apply_tokenizer(data, tokenizer):
    """
    Returns a list of lists, with the tokens of given text. I.e
    for an input ['Abc def', 'Ghi jkl mn'] it returns [['Abc', 'def'], ['Ghi', 'jkl', 'mn']]
    
    Args:
    data - list with the data
    tokenizer - nltk tokenizer
    """
    
    # YOUR CODE HERE
    raise NotImplementedError()
tokenizer = WordPunctTokenizer()
data_tok = apply_tokenizer(data=data, tokenizer=tokenizer)

assert len(data_tok) == 3770
assert len([w for s in data_tok for w in s]) == 51648
assert data_tok[8] == ['I','could','join','it',',','the','second','Afghan','war','had','broken','out','.','On','landing','at']<jupyter_output><empty_output><jupyter_text>Q2.b)The second step you will implement is lowercasing the data.<jupyter_code>def apply_lowercase(data):
    """
    Returns a list of lists, with all the tokens lowecased.
    
    Args:
    data - list with tokenized data
    """

    # YOUR CODE HERE
    raise NotImplementedError()
data_lc = apply_lowercase(data=apply_tokenizer(data=data, tokenizer=tokenizer))

assert len(data_lc) == 3770
assert len([w for s in data_lc for w in s]) == 51648
assert data_lc[8] == ['i','could','join','it',',','the','second','afghan','war','had','broken','out','.','on','landing','at']<jupyter_output><empty_output><jupyter_text>Q2.c)Now implement a function that filters the stopwords.NOTE: Stopwords adapted from [here](https://gist.github.com/sebleier/554280). (Notice what we added some specific things, like ?" and ." to the stopwords. This was shown to be a limitation of the nltk tokenizer so it will be removed that way, instead of the more conventional way. This goes to show that there are more powerful tokenizers that you should use in the case you have to perform tokenization in the future.)<jupyter_code>def apply_filter_stopwords(data, stopwords_fp):
    """
    Returns a list of lists, with no stopwords.
    
    Args:
    data - list with the tokenized data
    stopwords_fp - path to the stopwords file
    """
    
    # Create the list of stopwords from the file
    # stopwords = ...
    # YOUR CODE HERE
    raise NotImplementedError()
    
    # Filter the stopwords from the text
    # data_filt = ...
    # YOUR CODE HERE
    raise NotImplementedError()
    
    return data_filt

stopwords_fp = "data/english_stopwords.txt"
data_filt_sw = apply_filter_stopwords(data=apply_lowercase(apply_tokenizer(data, tokenizer)), 
                                      stopwords_fp=stopwords_fp)
assert len(data_filt_sw) == 3770
assert len([w for s in data_filt_sw for w in s]) == 27733
assert data_filt_sw[8] == ['could', 'join', ',', 'second', 'afghan', 'war', 'broken', '.', 'landing']<jupyter_output><empty_output><jupyter_text>Q2.d)After filtering stopwords, we want to remove punctuation from the text as well. Make use of `string.punctuation` to do so.<jupyter_code>def apply_filter_punkt(data):
    """
    Returns a list of lists, with no punctuation.
    
    Args:
    data - list with the tokenized data
    """

    # YOUR CODE HERE
    raise NotImplementedError()
data_filt_punkt = apply_filter_punkt(data=apply_tokenizer(data, tokenizer))

assert len(data_filt_punkt) == 3770
assert len([w for s in data_filt_punkt for w in s]) == 46362
assert data_filt_punkt[8] == ['I','could','join','it','the','second','Afghan','war','had','broken','out','On','landing','at']<jupyter_output><empty_output><jupyter_text>Q2.e)The last preprocessing step you are going to implement is stemming.<jupyter_code>def apply_stemmer(data, stemmer):
    """
    Returns a list of lists, with stemmed data.
    
    Args:
    data - list with the tokenized data
    stemmer - instance of stemmer to use
    """

    # YOUR CODE HERE
    raise NotImplementedError()
stemmer = SnowballStemmer("english")
data_stems = apply_stemmer(data=apply_lowercase(apply_tokenizer(data, tokenizer)),
                           stemmer=stemmer)

assert len(data_stems) == 3770
assert len([w for s in data_stems for w in s]) == 51648
assert data_stems[8][-2] == 'land'<jupyter_output><empty_output><jupyter_text>Q2.f)Finally, join everything in a function, that applies the steps in the following order, in :* Tokenization* Lowercasing* Filtering stopwords* Filtering punctuation* Stemming<jupyter_code># Custom transformer to implement sentence cleaning
class TextCleanerTransformer(TransformerMixin):
    def __init__(self, tokenizer, stemmer, regex_list, lower=True, remove_punct=True, stopwords=[]):
        self.tokenizer = tokenizer
        self.stemmer = stemmer
        self.regex_list = regex_list
        self.lower = lower
        self.remove_punct = remove_punct
        self.stopwords = stopwords
    
    def clean_sentences(self, sentences):
                
        # Split sentence into list of words
        # sentences_tokens = ...
        # YOUR CODE HERE
        raise NotImplementedError()
        
        # Lowercase
        if self.lower:
            # sentences_tokens = ...
            # YOUR CODE HERE
            raise NotImplementedError()
            
        # Remove punctuation
        if self.remove_punct:
            # sentences_tokens = ...
            # YOUR CODE HERE
            raise NotImplementedError()

        if self.stopwords:
            # sentences_tokens = ...
            # YOUR CODE HERE
            raise NotImplementedError()
    
        # Stem words
        if self.stemmer:
            # sentences_tokens = ...
            # YOUR CODE HERE
            raise NotImplementedError()

        # Join list elements into string
        sentences_prep = [" ".join(tokens).strip() for tokens in sentences_tokens]
        return sentences_prep

text_cleaner = TextCleanerTransformer(
    regex_list=[],
    tokenizer=tokenizer, 
    stemmer=stemmer,
    lower=True, 
    remove_punct=True, 
    stopwords=stopwords_fp
)

data_preprocessed = text_cleaner.clean_sentences(data)
assert len(data_preprocessed) == 3770
assert len([w for s in data_preprocessed for w in s.split()]) == 22447
assert data_preprocessed[8] == 'could join second afghan war broken land'
assert data_preprocessed[15] == 'noth misfortun disast remov brigad'<jupyter_output><empty_output><jupyter_text>Q3. Movie reviewsWe will now use what we've learned to explore movie reviews. We will start by analysing the dataset, then we will apply the preprocessing you implemented above, and finally we will see how it affects a classification task.  Q3.a)To get some stats on the dataset, we will start by implementing your own function to get the list of n-grams from a list of tokens. Complete the function below:<jupyter_code>def ngrams(data, n):
    """
    Returns list of tuples for all the n-grams
    
    Args:
    data - list of tokenized data (flattened)
    n - the n in n-grams
    """

    # YOUR CODE HERE
    raise NotImplementedError()
assert ngrams("The actress won the oscar".split(), 2) == [('The', 'actress'), ('actress', 'won'), ('won', 'the'), ('the', 'oscar')]
assert ngrams("The actress won the oscar".split(), 3) == [('The', 'actress', 'won'), ('actress', 'won', 'the'), ('won', 'the', 'oscar')]
assert ngrams("The actress won the oscar".split(), 4) == [('The', 'actress', 'won', 'the'), ('actress', 'won', 'the', 'oscar')]<jupyter_output><empty_output><jupyter_text>Q3.b)We will now see in our dataset what are the most common n-grams. Load the data and find how many unique bi-grams, tri-grams and four-grams we have. Also, take advantage of `Counter` and `most_common()` to find the most common tri-gram. Merge together the words of the most common trigram to get one single string. (Hint: look at python's `join` function, exemplefied below when joining the full text)<jupyter_code># Load the dataset
df = pd.read_csv('data/imdb_sentiment.csv')

# Get the text and split into full list of words
docs = df['text']
full_text = ' '.join([d.strip() for d in docs])
words = full_text.split(' ')<jupyter_output><empty_output><jupyter_text>Implement below the code to get the sets of unigrams, bigrams, trigrams and fourgrams, and to<jupyter_code># unigrams = ...
# bigrams = ...
# trigrams = ...
# fourgrams = ...
# most_common_trigram = ...
#
# YOUR CODE HERE
raise NotImplementedError()
n_unigrams = str(len(unigrams))
n_bigrams = str(len(bigrams))
n_trigrams = str(len(trigrams))
n_fourgrams = str(len(fourgrams))

print('Found {} unigrams'.format(n_unigrams))
assert hashlib.sha256(n_unigrams.encode()).hexdigest() == '1ae2d8247d3ad491c79aed034828ba78b21e25438a6e9a61f252eb566e39e877'

print('Found {} bigrams'.format(n_bigrams))
assert hashlib.sha256(n_bigrams.encode()).hexdigest() == '7d2d487bcdf890f05578da49f574e3e8f22f7420f752071a24eb49759de5adf8'

print('Found {} trigrams'.format(n_trigrams))
assert hashlib.sha256(n_trigrams.encode()).hexdigest() == '8c54e3c7087ab053a77d56c60408fd47837081fdea817b7cc9e68f134cef969d'

print('Found {} fourgrams'.format(n_fourgrams))
assert hashlib.sha256(n_fourgrams.encode()).hexdigest() == '<KEY>'

print('Most common trigram is "{}"'.format(most_common_trigram))
assert hashlib.sha256(most_common_trigram.encode()).hexdigest() == '28b6f04107ef3f1120975abf58ca8d08d20243beea929999b203f0add941fe16'<jupyter_output><empty_output><jupyter_text>Q3.c)Let's now process a sample of our dataset with the previous Q2 preprocessing, and get a Bag of Words representation. Start by using your text cleaner to get a preprocessed version of this dataset.Note: if you didn't finish the text cleaner above, jump to the TF-IDF implementation directly, where you can load the BoW from a file.<jupyter_code>text_cleaner = TextCleanerTransformer(
    regex_list=[],
    tokenizer=tokenizer, 
    stemmer=None,
    lower=True, 
    remove_punct=True, 
    stopwords=stopwords_fp
)

docs_preprocessed = text_cleaner.clean_sentences(docs[:200])<jupyter_output><empty_output><jupyter_text>We can get a vocabulary, vectorize our dataset and convert it into a BoW<jupyter_code>def build_vocabulary(docs):
    vocabulary = Counter()

    for doc in docs:
        words = doc.split()
        vocabulary.update(words)
    
    return OrderedDict(vocabulary.most_common())

def vectorize(docs):
    vocabulary = build_vocabulary(docs)
    vectors = []
    for doc in docs:
        words = doc.split()
        vector = np.array([doc.count(word) for word in vocabulary])
        vectors.append(vector)
    
    return (vocabulary, vectors)

def build_df(docs):
    vocab, vectors = vectorize(docs)
    return pd.DataFrame(vectors, columns=vocab)

BoW = build_df(docs_preprocessed)
BoW.head()<jupyter_output><empty_output><jupyter_text>You will now implement one of TF-IDFs variation to compute from the bag of words the more relevant words. The formulation you should use is one you've learned before:$$ tfidf _{t, d} =(tf_{t,d})*(log_2{(1 + \frac{N}{df_{t}})})  $$Implement the TF-IDF below:<jupyter_code>def tfidf(BoW_df):
    """
    Returns pandas dataframe of a tfidf representation from a BoW representation dataframe.

    Args:
    BoW_df - dataframe with document word counts (Bag of Words)
    """
    # tf = (...)
    
    # def _idf(column):
    #   return (...)
    
    # tf_idf = (...)
    
    # return tf_idf

    # YOUR CODE HERE
    raise NotImplementedError()<jupyter_output><empty_output><jupyter_text>Let's now apply it to our previous BoW (note: load the BoW first if you could not use your text cleaner)<jupyter_code>BoW = pd.read_csv('data/imdb_sentiment_bow_sample.csv')
BoW.head()
relevance = tfidf(BoW)

assert(math.isclose(relevance['movie'][0], 0.009717385023827248),
       math.isclose(relevance['film'][10], 0.019778475747522496),
       math.isclose(relevance['nice'][16], 0.010851136310680626),
       math.isclose(relevance['good'][128], 0.00989061193998239))<jupyter_output><empty_output><jupyter_text>Q3.d)Now, let's use scikit-learn to get to a similar matrix and relevance numbers. Load the full processed dataset:<jupyter_code># Load the dataset
df_preprocessed = pd.read_csv('data/imdb_sentiment_processed.csv')

# Get the processed text 
docs = df_preprocessed['text']<jupyter_output><empty_output><jupyter_text>Start by transforming your documents into a matrix of tf-idf scores using sklearn. Make use of the `CountVectorizer` and the `TfidfTransformer` provided by scikitlearn. Implement a function that provided with a list of documents returns the word term frequency matrix and the corresponding vocabulary:<jupyter_code>def build_word_term_frequency_matrix(docs):
    """
    Returns the matrix of word and tf-idf scores 
    
    Args:
    docs - list of documents in dataset
    """
    # vectorizer = ...
    # word_count_matrix = ...
    # vocabulary = ...
    # YOUR CODE HERE
    raise NotImplementedError()

    # tfidf = ...
    # word_term_frequency_matrix = ...
    # YOUR CODE HERE
    raise NotImplementedError()

    return (word_term_frequency_matrix, vocabulary)<jupyter_output><empty_output><jupyter_text>Now get the corresponding string of the most important word of this document (with index `321`) according to TF-IDF.<jupyter_code>index = 321

word_term_frequency_matrix, vocabulary = build_word_term_frequency_matrix(docs)

max_word_idx = word_term_frequency_matrix[index].argmax()
inv_vocab = {v: k for k, v in vocabulary.items()}
most_relevant_word = inv_vocab[max_word_idx]

assert(most_relevant_word == 'dull')<jupyter_output><empty_output><jupyter_text>Q3.e)Finally, let's try to classify the sentiment of these movie reviews. Build a Pipeline to classify a review as positive or negative. Use `MultinomialNB` as your final classifier, train it and get an accuracy score above 86% on the imdb validation dataset, by choosing the best set of parameters of `CountVectorizer()` and `TfidfTransformer()`, according to what we learned in Part III.Hint: Try to use more than unigrams! Also, remember what we said about stopwords and feature space size in Part III of the Learning Notebooks?<jupyter_code># Split in train and validation
train_df, validation_df = train_test_split(df_preprocessed, test_size=0.3, random_state=42)

# Encode the labels
le = preprocessing.LabelEncoder()
le.fit(train_df['sentiment'].values)

train_df['sentiment'] = le.transform(train_df['sentiment'].values)
validation_df['sentiment'] = le.transform(validation_df['sentiment'].values)
def train_and_validate(train_df, validation_df):
    """
    Train a model using sklearn's Pipeline and return it along with its 
    current accuracy in the validation set. Assume the documents are already 
    preprocessed
    
    Args:
    train_df - dataframe with training docs
    validation_df - dataframe with validation docs
    """
    
    # Build the pipeline
    # text_clf = Pipeline(...)
    
    # Train the classifier
    # (...)

    # predicted = (...)
    # acc = (...)
    
    # return text_clf, acc
    
    # YOUR CODE HERE
    raise NotImplementedError()
_, acc = train_and_validate(train_df, validation_df)
print("Accuracy: {}".format(acc))
assert(acc >= 0.86)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10723860589812333, 'fraction_numerical': 0.02328759922199443, 'mean_word_length': 4.342319573153609, 'pattern_counts': {'":': 1, '<': 103, '<?xml version=': 0, '>': 105, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '496944', 'license': 'MIT', 'n_tokens_mistral': 6450, 'n_tokens_neox': 5981, 'n_words': 1821, 'path': 'S04 - Text Classification/BLU07 - Feature Extraction/Exercise notebook.ipynb', 'repo_name': 'jtiagosg/batch3-students', 'size': '19098'}