<jupyter_start><jupyter_text>Exercise 2.2 GoalThe performance of machine learning systems directly depends on the quality of input features. In this exercise, you will investigate the impact of individual features on a system for named entity recognition: what does the inclusion of each individual feature do to the results? And what happens when they are combined? AcknowledgementThis exercise made use of examples from the following exercise (in the HLT course):https://github.com/cltl/ma-hlt-labs/Lab3: machine learning ProcedureThis notebook will provide the code for running the experiments outlined above. You will only need to make minor adaptations to run the feature ablation analysis. This notebook was developed for another more introductory course where students did not need to generate their own features. Please take that into account while reading this code (i.e. you can use this as an example, but it will not work one-to-one on your own data).The notebooks and set up have been designed for educational purposes: design choices are based on clearly illustrating what is going on and facilitating the exercises. The DataThe data of the original assignment been preprocessed to make some useful features directly availabe (as you were recommended to do as well in Assignment 2).The format of the conll files provided to the students in this original exercise was:Token  Preceding_token  Capitalization  POS-tag  Chunklabel  GoldlabelThe first lines look like this:-DOCSTART-      FULLCAP -X-     -X-     O  EU              FULLCAP NNP     B-NP    B-ORGrejects EU      LOWCASE VBZ     B-VP    OPreceding_token: This column provides the token preceding the current token. (This is an empty space if there is no previous token).Capitalization: This column provides information on the capitalization of the token.  PackagesWe will make use of the following packages:* scikit-learn : provides lots of useful implementations for machine learning (and has relatively good documentation!)* csv: a light-weight package to deal with data represented in csv (or related formats such as tsv)* gensim: a useful package for working with word embeddings* numpy: a packages that (among others) provides useful datastructures and operations to work with vectorsSome notes on design decisions (feel free to ignore these if this is all new to you):* We are using csv rather than (the more common) pandas for working with the conll files, because pandas standardly applies type conversion, which we do not want when dealing with text that contains numbers (fixing this will make the code look more complex).* scikit-learn provides several machine learning algorithms, but this is not the focus of this exercise. We are using logistic regression, because it serves the purpose of our experiments and is relatively efficient.<jupyter_code>#this cell imports all the modules we'll need. Make sure to run this once before running the other cells


#sklearn is scikit-learn
import sklearn
import csv
import gensim
import numpy as np
import pandas as pd

from sklearn import metrics
from sklearn.feature_extraction import DictVectorizer
from sklearn.linear_model import LogisticRegression<jupyter_output><empty_output><jupyter_text>Part 1: Traditional FeaturesIn this first part, we will explore the impact of various features on named entity recognition.We will use so-called traditional features, where the feature values (strings) are presented by one-hot encoding  Step 1: A Basic ClassifierWe will first walk through the process of creating and evaluating a simple classifier that only uses the token itself as a feature. In the next step, we will run evaluations on this basic system.This is generally a good way to start experimenting: first walk through the entire experimental process with a very basic, easy to create system to see if everything works, there are no problems with the data etc. You can then build up from there towards a more sophististicated system.<jupyter_code>#functions for feature extraction and training a classifier


## For documentation on how to create input representations of features in scikit-learn:
# https://scikit-learn.org/stable/modules/feature_extraction.html

#Setting some variables that we will use multiple times
trainfile = '../data/conll_train_cap.txt'
testfile = '../data/conll_test_cap.txt'

def extract_features_token_only_and_labels(conllfile):
    '''Function that extracts features and gold label from preprocessed conll (here: tokens only).
    
    :param conllfile: path to the (preprocessed) conll file
    :type conllfile: string
    
    
    :return features: a list of dictionaries, with key-value pair providing the value for the feature `token' for individual instances
    :return labels: a list of gold labels of individual instances
    '''
    
    features = []
    labels = []
    conllinput = open(conllfile, 'r')
    #delimiter indicates we are working with a tab separated value (default is comma)
    #quotechar has as default value '"', which is used to indicate the borders of a cell containing longer pieces of text
    #in this file, we have only one token as text, but this token can be '"', which then messes up the format. We set quotechar to a character that does not occur in our file
    csvreader = csv.reader(conllinput, delimiter='\t',quotechar='|')
    for row in csvreader:
        #I preprocessed the file so that all rows with instances should contain 6 values, the others are empty lines indicating the beginning of a sentence
        if len(row) == 6:
            #structuring feature value pairs as key-value pairs in a dictionary
            #the first column in the conll file represents tokens
            feature_value = {'Token': row[0]}
            features.append(feature_value)
            #The last column provides the gold label (= the correct answer). 
            labels.append(row[-1])
    
    return features, labels



def create_vectorizer_and_classifier(features, labels):
    '''
    Function that takes feature-value pairs and gold labels as input and trains a logistic regression classifier
    
    :param features: feature-value pairs
    :param labels: gold labels
    :type features: a list of dictionaries
    :type labels: a list of strings
    
    :return lr_classifier: a trained LogisticRegression classifier
    :return vec: a DictVectorizer to which the feature values are fitted. 
    '''
    
    vec = DictVectorizer()
    #fit creates a mapping between observed feature values and dimensions in a one-hot vector, transform represents the current values as a vector 
    tokens_vectorized = vec.fit_transform(features)
    lr_classifier = LogisticRegression(solver='saga')
    lr_classifier.fit(tokens_vectorized, labels)
    
    return lr_classifier, vec

#extract features and labels:
feature_values, labels = extract_features_token_only_and_labels(trainfile) 
#create vectorizer and trained classifier:
lr_classifier, vectorizer = create_vectorizer_and_classifier(feature_values, labels)<jupyter_output><empty_output><jupyter_text>Step 2: EvaluationWe will now run a basic evaluation of the system on a test file. Two important properties of the test file:1. the test file and training file are independent sets (if they contain identical examples, this is coincidental)2. the test file is preprocessed in the exact same way as the training file The first function runs our classifier on the test data.The second function prints out a confusion matrix (comparing predictions and gold labels per class). You can find more information on confusion matrices here: https://www.geeksforgeeks.org/confusion-matrix-machine-learning/The third function prints out the macro precision, recall and f-score of the system<jupyter_code>def get_predicted_and_gold_labels_token_only(testfile, vectorizer, classifier):
    '''
    Function that extracts features and runs classifier on a test file returning predicted and gold labels
    
    :param testfile: path to the (preprocessed) test file
    :param vectorizer: vectorizer in which the mapping between feature values and dimensions is stored
    :param classifier: the trained classifier
    :type testfile: string
    :type vectorizer: DictVectorizer
    :type classifier: LogisticRegression()
    
    
    
    :return predictions: list of output labels provided by the classifier on the test file
    :return goldlabels: list of gold labels as included in the test file
    '''
    
    #we use the same function as above (guarantees features have the same name and form)
    sparse_feature_reps, goldlabels = extract_features_token_only_and_labels(testfile)
    #we need to use the same fitting as before, so now we only transform the current features according to this mapping (using only transform)
    test_features_vectorized = vectorizer.transform(sparse_feature_reps)
    predictions = classifier.predict(test_features_vectorized)
    
    return predictions, goldlabels

def print_confusion_matrix(predictions, goldlabels):
    '''
    Function that prints out a confusion matrix
    
    :param predictions: predicted labels
    :param goldlabels: gold standard labels
    :type predictions, goldlabels: list of strings
    '''
    
    
    
    #based on example from https://datatofish.com/confusion-matrix-python/ 
    data = {'Gold':    goldlabels, 'Predicted': predictions    }
    df = pd.DataFrame(data, columns=['Gold','Predicted'])

    confusion_matrix = pd.crosstab(df['Gold'], df['Predicted'], rownames=['Gold'], colnames=['Predicted'])
    print (confusion_matrix)


def print_precision_recall_fscore(predictions, goldlabels):
    '''
    Function that prints out precision, recall and f-score
    
    :param predictions: predicted output by classifier
    :param goldlabels: original gold labels
    :type predictions, goldlabels: list of strings
    '''
    
    precision = metrics.precision_score(y_true=goldlabels,
                        y_pred=predictions,
                        average='macro')

    recall = metrics.recall_score(y_true=goldlabels,
                     y_pred=predictions,
                     average='macro')


    fscore = metrics.f1_score(y_true=goldlabels,
                 y_pred=predictions,
                 average='macro')

    print('P:', precision, 'R:', recall, 'F1:', fscore)
    
#vectorizer and lr_classifier are the vectorizer and classifiers created in the previous cell.
#it is important that the same vectorizer is used for both training and testing: they should use the same mapping from values to dimensions
predictions, goldlabels = get_predicted_and_gold_labels_token_only(testfile, vectorizer, lr_classifier)
print_confusion_matrix(predictions, goldlabels)
print_precision_recall_fscore(predictions, goldlabels)<jupyter_output><empty_output><jupyter_text>Step 3: A More Elaborate SystemNow that we have run a basic experiment, we are going to investigate alternatives. In this exercise, we only focus on features. We will continue to use the same logistic regression classifier throughout the exercise.We want to investigate the impact of individual features. We will thus use a function that allows us to specify whether we include a specific feature or not. The features we have at our disposal are:* the token itself (as used above)* the preceding token* the capitalization indication (see above for values that this takes)* the pos-tag of the token* the chunklabel of the chunk the token is part of<jupyter_code># the functions with multiple features and analysis

#defines the column in which each feature is located (note: you can also define headers and use csv.DictReader)
feature_to_index = {'Token': 0, 'Prevtoken': 1, 'Cap': 2, 'Pos': 3, 'Chunklabel': 4}


def extract_features_and_gold_labels(conllfile, selected_features):
    '''Function that extracts features and gold label from preprocessed conll (here: tokens only).
    
    :param conllfile: path to the (preprocessed) conll file
    :type conllfile: string
    
    
    :return features: a list of dictionaries, with key-value pair providing the value for the feature `token' for individual instances
    :return labels: a list of gold labels of individual instances
    '''
    
    features = []
    labels = []
    conllinput = open(conllfile, 'r')
    #delimiter indicates we are working with a tab separated value (default is comma)
    #quotechar has as default value '"', which is used to indicate the borders of a cell containing longer pieces of text
    #in this file, we have only one token as text, but this token can be '"', which then messes up the format. We set quotechar to a character that does not occur in our file
    csvreader = csv.reader(conllinput, delimiter='\t',quotechar='|')
    for row in csvreader:
        #I preprocessed the file so that all rows with instances should contain 6 values, the others are empty lines indicating the beginning of a sentence
        if len(row) == 6:
            #structuring feature value pairs as key-value pairs in a dictionary
            #the first column in the conll file represents tokens
            feature_value = {}
            for feature_name in selected_features:
                row_index = feature_to_index.get(feature_name)
                feature_value[feature_name] = row[row_index]
            features.append(feature_value)
            #The last column provides the gold label (= the correct answer). 
            labels.append(row[-1])
    return features, labels

def get_predicted_and_gold_labels(testfile, vectorizer, classifier, selected_features):
    '''
    Function that extracts features and runs classifier on a test file returning predicted and gold labels
    
    :param testfile: path to the (preprocessed) test file
    :param vectorizer: vectorizer in which the mapping between feature values and dimensions is stored
    :param classifier: the trained classifier
    :type testfile: string
    :type vectorizer: DictVectorizer
    :type classifier: LogisticRegression()
    
    
    
    :return predictions: list of output labels provided by the classifier on the test file
    :return goldlabels: list of gold labels as included in the test file
    '''
    
    #we use the same function as above (guarantees features have the same name and form)
    features, goldlabels = extract_features_and_gold_labels(testfile, selected_features)
    #we need to use the same fitting as before, so now we only transform the current features according to this mapping (using only transform)
    test_features_vectorized = vectorizer.transform(features)
    predictions = classifier.predict(test_features_vectorized)
    
    return predictions, goldlabels

#define which from the available features will be used (names must match key names of dictionary feature_to_index)
all_features = ['Token','Prevtoken','Cap','Pos','Chunklabel']

sparse_feature_reps, labels = extract_features_and_gold_labels(trainfile, all_features)
#we can use the same function as before for creating the classifier and vectorizer
lr_classifier, vectorizer = create_vectorizer_and_classifier(sparse_feature_reps, labels)
#when applying our model to new data, we need to use the same features
predictions, goldlabels = get_predicted_and_gold_labels(testfile, vectorizer, lr_classifier, all_features)
print_confusion_matrix(predictions, goldlabels)
print_precision_recall_fscore(predictions, goldlabels)<jupyter_output><empty_output><jupyter_text>Step 4: Feature Ablation AnalysisIf all worked well, the system that made use of all features worked better than the system with just the tokens.We now want to know which of the features contributed to this improved: do we want to include all features?Or just some?We can investigate this using *feature ablation analysis*. This means that we systematically test what happens if we add or remove a specific feature. Ideally, we investigate all possible combinations.The cell below illustrates how you can use the code above to investigate a system with three features. You can modify the selected features to try out different combinations. You can either do this manually and rerun the cell or write a function that creates list of all combinations you want to tests and runs them one after the other.Include your results in the report of this week.<jupyter_code># example of system with just one additional feature
#define which from the available features will be used (names must match key names of dictionary feature_to_index)
selected_features = ['Token','Prevtoken','Pos']

feature_values, labels = extract_features_and_gold_labels(trainfile, selected_features)
#we can use the same function as before for creating the classifier and vectorizer
lr_classifier, vectorizer = create_vectorizer_and_classifier(feature_values, labels)
#when applying our model to new data, we need to use the same features
predictions, goldlabels = get_predicted_and_gold_labels(testfile, vectorizer, lr_classifier, selected_features)
print_confusion_matrix(predictions, goldlabels)
print_precision_recall_fscore(predictions, goldlabels)<jupyter_output><empty_output><jupyter_text>Part 2: One-hot versus EmbeddingsIn this second part of the exercise, we will compare results using one-hot encodings to pretrained word embeddings. One-hot representationIn one-hot representation, each feature value is represented by an n-dimensional vector, where n corresponds to the number of possible values the feature can take. In our system, the Token feature can take the value of each token that occurs at least once in the corpus. This feature thus uses a vector with the size of the vocabulary in the corpus. Each possible value is associated with a specific dimension. If this value is represented, that dimension will receive the value 1 and all other dimensions will have the value 0.The system receive a concatenation of all feature representations as input. What does one-hot look like?We will start with an illustration of a one-hot representation. We will use the capitalization feature for this: it has 6 possible values and is therefore represented by a 6-dimensional vector. If you would like a more precise look, you may consider creating a toy example of a few lines, in which the capitalization feature has different values.<jupyter_code># create classifier with caps feature only and print vectorizer, then with token only (but you see less)

selected_features = ['Cap']

feature_values, labels = extract_features_and_gold_labels(trainfile, selected_features)

#creating a vectorizing
vectorizer = DictVectorizer()
#fitting the values to dimensions (creating a mapping) and transforming the current observations according to this mapping
capitalization_vectorized = vectorizer.fit_transform(feature_values)
print(capitalization_vectorized.toarray())<jupyter_output><empty_output><jupyter_text>Using word embeddingsWe are now going to use word embeddings to represent tokens. We load a pretrained distributional semantic model.You can use the same model as in Exercise 2.1. We tested the exercise with the same model (GoogleNews negative sampling 300 dimensions) as Exercise 2.1 as well.Note: loading the model may take a while. You probably want to run that only once.<jupyter_code># this step takes a while
word_embedding_model = gensim.models.KeyedVectors.load_word2vec_format('../models/GoogleNews-vectors-negative300.bin.gz', binary=True)  
def extract_embeddings_as_features_and_gold(conllfile,word_embedding_model):
    '''
    Function that extracts features and gold labels using word embeddings
    
    :param conllfile: path to conll file
    :param word_embedding_model: a pretrained word embedding model
    :type conllfile: string
    :type word_embedding_model: gensim.models.keyedvectors.Word2VecKeyedVectors
    
    :return features: list of vector representation of tokens
    :return labels: list of gold labels
    '''
    labels = []
    features = []
    
    conllinput = open(conllfile, 'r')
    csvreader = csv.reader(conllinput, delimiter='\t',quotechar='|')
    for row in csvreader:
        if len(row) == 6:
            if row[0] in word_embedding_model:
                vector = word_embedding_model[row[0]]
            else:
                vector = [0]*300
            features.append(vector)
            labels.append(row[-1])
    return features, labels

def create_classifier(features, labels):
    '''
    Function that creates classifier from features represented as vectors and gold labels
    
    :param features: list of vector representations of tokens
    :param labels: list of gold labels
    :type features: list of vectors
    :type labels: list of strings
    
    :returns trained logistic regression classifier
    '''
    
    
    lr_classifier = LogisticRegression(solver='saga')
    lr_classifier.fit(features, labels)
    
    return lr_classifier
    
    
def label_data_using_word_embeddings(testfile, word_embedding_model, classifier):
    '''
    Function that extracts word embeddings as features and gold labels from test data and runs a classifier
    
    :param testfile: path to test file
    :param word_embedding_model: distributional semantic model
    :param classifier: trained classifier
    :type testfile: string
    :type word_embedding_model: gensim.models.keyedvectors.Word2VecKeyedVectors
    :type classifier: LogisticRegression
    
    :return predictions: list of predicted labels
    :return labels: list of gold labels
    '''
    
    dense_feature_representations, labels = extract_embeddings_as_features_and_gold(testfile,word_embedding_model)
    predictions = classifier.predict(dense_feature_representations)
    
    return predictions, labels


# I printing announcements of where the code is at (since some of these steps take a while)

print('Extracting dense features...')
dense_feature_representations, labels = extract_embeddings_as_features_and_gold(trainfile,word_embedding_model)
print('Training classifier....')
classifier = create_classifier(dense_feature_representations, labels)
print('Running evaluation...')
predicted, gold = label_data_using_word_embeddings(testfile, word_embedding_model, classifier)
print_confusion_matrix(predictions, goldlabels)
print_precision_recall_fscore(predicted, gold)<jupyter_output><empty_output><jupyter_text>Including the preceding tokenWe can include the preceding token as a feature in a similar way. We simply concatenate the two vectors.<jupyter_code>def extract_embeddings_of_current_and_preceding_as_features_and_gold(conllfile,word_embedding_model):
    '''
    Function that extracts features and gold labels using word embeddings for current and preceding token
    
    :param conllfile: path to conll file
    :param word_embedding_model: a pretrained word embedding model
    :type conllfile: string
    :type word_embedding_model: gensim.models.keyedvectors.Word2VecKeyedVectors
    
    :return features: list of vector representation of tokens
    :return labels: list of gold labels
    '''
    labels = []
    features = []
    
    conllinput = open(conllfile, 'r')
    csvreader = csv.reader(conllinput, delimiter='\t',quotechar='|')
    for row in csvreader:
        if len(row) == 6:
            if row[0] in word_embedding_model:
                vector1 = word_embedding_model[row[0]]
            else:
                vector1 = [0]*300
            if row[1] in word_embedding_model:
                vector2 = word_embedding_model[row[1]]
            else:
                vector2 = [0]*300
            features.append(np.concatenate((vector1,vector2)))
            labels.append(row[-1])
    return features, labels
    
    
def label_data_using_word_embeddings_current_and_preceding(testfile, word_embedding_model, classifier):
    '''
    Function that extracts word embeddings as features (of current and preceding token) and gold labels from test data and runs a trained classifier
    
    :param testfile: path to test file
    :param word_embedding_model: distributional semantic model
    :param classifier: trained classifier
    :type testfile: string
    :type word_embedding_model: gensim.models.keyedvectors.Word2VecKeyedVectors
    :type classifier: LogisticRegression
    
    :return predictions: list of predicted labels
    :return labels: list of gold labels
    '''
    
    features, labels = extract_embeddings_of_current_and_preceding_as_features_and_gold(testfile,word_embedding_model)
    predictions = classifier.predict(features)
    
    return predictions, labels



print('Extracting dense features...')
features, labels = extract_embeddings_of_current_and_preceding_as_features_and_gold(trainfile,word_embedding_model)
print('Training classifier...')
#we can use the same function as for just the tokens itself
classifier = create_classifier(features, labels)
print('Running evaluation...')
predicted, gold = label_data_using_word_embeddings_current_and_preceding(testfile, word_embedding_model, classifier)
print_confusion_matrix(predictions, goldlabels)
print_precision_recall_fscore(predicted, gold)<jupyter_output><empty_output><jupyter_text>A mixed systemThe code below combines traditional features with word embeddings. Note that we only include features with a limited range of possible values. Combining one-hot token representations (using highly sparse dimensions) with dense representations is generally not a good idea.<jupyter_code>def extract_word_embedding(token, word_embedding_model):
    '''
    Function that returns the word embedding for a given token out of a distributional semantic model and a 300-dimension vector of 0s otherwise
    
    :param token: the token
    :param word_embedding_model: the distributional semantic model
    :type token: string
    :type word_embedding_model: gensim.models.keyedvectors.Word2VecKeyedVectors
    
    :returns a vector representation of the token
    '''
    if token in word_embedding_model:
        vector = word_embedding_model[token]
    else:
        vector = [0]*300
    return vector


def extract_feature_values(row, selected_features):
    '''
    Function that extracts feature value pairs from row
    
    :param row: row from conll file
    :param selected_features: list of selected features
    :type row: string
    :type selected_features: list of strings
    
    :returns: dictionary of feature value pairs
    '''
    feature_values = {}
    for feature_name in selected_features:
        r_index = feature_to_index.get(feature_name)
        feature_values[feature_name] = row[r_index]
        
    return feature_values
    
    
def create_vectorizer_traditional_features(feature_values):
    '''
    Function that creates vectorizer for set of feature values
    
    :param feature_values: list of dictionaries containing feature-value pairs
    :type feature_values: list of dictionairies (key and values are strings)
    
    :returns: vectorizer with feature values fitted
    '''
    vectorizer = DictVectorizer()
    vectorizer.fit(feature_values)
    
    return vectorizer
        
    
def combine_sparse_and_dense_features(dense_vectors, sparse_features):
    '''
    Function that takes sparse and dense feature representations and appends their vector representation
    
    :param dense_vectors: list of dense vector representations
    :param sparse_features: list of sparse vector representations
    :type dense_vector: list of arrays
    :type sparse_features: list of lists
    
    :returns: list of arrays in which sparse and dense vectors are concatenated
    '''
    
    combined_vectors = []
    sparse_vectors = np.array(sparse_features.toarray())
    
    for index, vector in enumerate(sparse_vectors):
        combined_vector = np.concatenate((vector,dense_vectors[index]))
        combined_vectors.append(combined_vector)
    return combined_vectors
    

def extract_traditional_features_and_embeddings_plus_gold_labels(conllfile, word_embedding_model, vectorizer=None):
    '''
    Function that extracts traditional features as well as embeddings and gold labels using word embeddings for current and preceding token
    
    :param conllfile: path to conll file
    :param word_embedding_model: a pretrained word embedding model
    :type conllfile: string
    :type word_embedding_model: gensim.models.keyedvectors.Word2VecKeyedVectors
    
    :return features: list of vector representation of tokens
    :return labels: list of gold labels
    '''
    labels = []
    dense_vectors = []
    traditional_features = []
    
    conllinput = open(conllfile, 'r')
    csvreader = csv.reader(conllinput, delimiter='\t',quotechar='|')
    for row in csvreader:
        if len(row) == 6:
            token_vector = extract_word_embedding(row[0], word_embedding_model)
            pt_vector = extract_word_embedding(row[1], word_embedding_model)
            dense_vectors.append(np.concatenate((token_vector,pt_vector)))
            #mixing very sparse representations (for one-hot tokens) and dense representations is a bad idea
            #we thus only use other features with limited values
            other_features = extract_feature_values(row, ['Cap','Pos','Chunklabel'])
            traditional_features.append(other_features)
            #adding gold label to labels
            labels.append(row[-1])
            
    #create vector representation of traditional features
    if vectorizer is None:
        #creates vectorizer that provides mapping (only if not created earlier)
        vectorizer = create_vectorizer_traditional_features(traditional_features)
    sparse_features = vectorizer.transform(traditional_features)
    combined_vectors = combine_sparse_and_dense_features(dense_vectors, sparse_features)
    
    return combined_vectors, vectorizer, labels

def label_data_with_combined_features(testfile, classifier, vectorizer, word_embedding_model):
    '''
    Function that labels data with model using both sparse and dense features
    '''
    feature_vectors, vectorizer, goldlabels = extract_traditional_features_and_embeddings_plus_gold_labels(testfile, word_embedding_model, vectorizer)
    predictions = classifier.predict(feature_vectors)
    
    return predictions, goldlabels


print('Extracting Features...')
feature_vectors, vectorizer, gold_labels = extract_traditional_features_and_embeddings_plus_gold_labels(trainfile, word_embedding_model)
print('Training classifier....')
lr_classifier = create_classifier(feature_vectors, gold_labels)
print('Running the evaluation...')
predictions, goldlabels = label_data_with_combined_features(testfile, lr_classifier, vectorizer, word_embedding_model)
print_confusion_matrix(predictions, goldlabels)
print_precision_recall_fscore(predictions, goldlabels)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.05658841532901377, 'fraction_numerical': 0.0028326767166997687, 'mean_word_length': 4.213715837718554, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '507181', 'license': 'Apache-2.0', 'n_tokens_mistral': 8066, 'n_tokens_neox': 7726, 'n_words': 3240, 'path': 'code/assignment3/sample_code_features_ablation_analysis.ipynb', 'repo_name': 'guedesbruna/ma-ml4nlp-labs', 'size': '30713'}