<jupyter_start><jupyter_text>CS 224n Assignment 3: Dependency ParsingIn this assignment, you will build a neural dependency parser using PyTorch. In Part 1, you will learnabout two general neural network techniques (Adam Optimization and Dropout) that you will use to buildthe dependency parser in Part 2. In Part 2, you will implement and train the dependency parser, beforeanalyzing a few erroneous dependency parses. 1. Machine Learning & Neural Networks (8 points)(^1) Kingma and Ba, 2015,https://arxiv.org/pdf/1412.6980.pdf(^2) The actual Adam update uses a few additional tricks that are less important, but we won’t worry about them here.(^3) Srivastava et al., 2014,https://www.cs.toronto.edu/ ̃hinton/absps/JMLRdropout.pdfi. (2 points) What mustγequal in terms of pdrop? Briefly justify your answer.ii. (2 points) Why should we apply dropout during training but not during evaluation?  2. Neural Transition-Based Dependency Parsing (42 points)In this section, you’ll be implementing a neural-network based dependency parser, with the goal of maximizing performance on the UAS (Unlabeled Attachemnt Score) metric.A dependency parser analyzes the grammatical structure of a sentence, establishing relationships betweenheadwords, and words which modify those heads. Your implementation will be atransition-basedparser,which incrementally builds up a parse one step at a time. At every step it maintains a partial parse,which is represented as follows:- A stack of words that are currently being processed.- A buffer of words yet to be processed.- A list of dependencies predicted by the parser.Initially, the stack only contains ROOT, the dependencies list is empty, and the buffer contains all wordsof the sentence in order. At each step, the parser applies a transition to the partial parse until its bufferis empty and the stack size is 1. The following transitions can be applied:```- SHIFT: removes the first word from the buffer and pushes it onto the stack.- LEFT-ARC: marks the second (second most recently added) item on the stack as a dependent of    the first item and removes the second item from the stack.- RIGHT-ARC: marks the first (most recently added) item on the stack as a dependent of the second    item and removes the first item from the stack.``` On each step, your parser will decide among the three transitions using a neural network classifier.(a) (6 points) Go through the sequence of transitions needed for parsing the sentence“I parsed thissentence correctly”. The dependency tree for the sentence is shown below. At each step, give theconfiguration of the stack and buffer, as well as what transition was applied this step and what newdependency was added (if any). The first three steps are provided below as an example.![dependency diagram](./img/a3-2-a-1.png) ![dependency diagram](./img/a3-2-a-1.png)Stack | Buffer | New  dependency | Transition----  |  ----   | ----| ----[ROOT] | [I, parsed, this, sentence, correctly] | | Initial Configuration[ROOT, I]| [parsed, this, sentence, correctly]| | SHIFT[ROOT, I, parsed] | [this, sentence, correctly] | | SHIFT[ROOT, parsed] | [this, sentence, correctly] | parsed→I |LEFT-ARC[ROOT, parsed, this] | [ sentence, correctly] |  | SHIFT[ROOT, parsed, this, sentence] | [ correctly] |  | SHIFT[ROOT, parsed, sentence] | [ correctly] | sentence→this | LEFT-ARC[ROOT, parsed] | [ correctly] | parsed→sentence | RIGHT-ARC[ROOT, parsed, correctly] | [] |  | SHIFT[ROOT, parsed] | [] | parsed→correctly | RIGHT-ARC[ROOT] | [] | ROOT→parsed | RIGHT-ARC (b) (2 points) A sentence containing n words will be parsed in how many steps (in terms of n )? Brieflyexplain why.  >2N + 1    1 initial configuration    Then, For each word:        1 Step for shifting from Buffer to Stack        1 Steps for applying transition / dependency    Total = 2N + 1> Complexity = O(N) (c) (6 points) Implement the init and parse step functions in the PartialParse class in`parser_transitions.py`. This implements the transition mechanics your parser will use. Youcan run basic (non-exhaustive) tests by running python `parser_transitions.py partc`.<jupyter_code>class PartialParse(object):
    def __init__(self, sentence):
        """Initializes this partial parse.

        @param sentence (list of str): The sentence to be parsed as a list of words.
                                        Your code should not modify the sentence.
        """
        # The sentence being parsed is kept for bookkeeping purposes. Do not alter it in your code.
        self.sentence = sentence

        ### YOUR CODE HERE (3 Lines)
        ### Your code should initialize the following fields:
        ###     self.stack: The current stack represented as a list with the top of the stack as the
        ###                 last element of the list.
        ###     self.buffer: The current buffer represented as a list with the first item on the
        ###                  buffer as the first item of the list
        ###     self.dependencies: The list of dependencies produced so far. Represented as a list of
        ###             tuples where each tuple is of the form (head, dependent).
        ###             Order for this list doesn't matter.
        ###
        ### Note: The root token should be represented with the string "ROOT"
        ###
        self.stack = ["ROOT"]
        self.buffer = self.sentence.copy()
        self.dependencies = []

        ### END YOUR CODE


    def parse_step(self, transition):
        """Performs a single parse step by applying the given transition to this partial parse

        @param transition (str): A string that equals "S", "LA", or "RA" representing the shift,
                                left-arc, and right-arc transitions. You can assume the provided
                                transition is a legal transition.
        """
        ### YOUR CODE HERE (~7-10 Lines)
        
        ### TODO:
        ###     Implement a single parsing step, i.e. the logic for the following as
        ###     described in the pdf handout:
        ###         1. Shift
        ###         2. Left Arc
        ###         3. Right Arc
        if transition == 'S':
            # remove last item in the stack
            self.stack.append(self.buffer.pop(0))
        elif transition == 'LA':
            #  [..., n-1, n] :  n -> n-1
            popped = self.stack.pop(-2)
            self.dependencies.append((self.stack[-1], popped))
        elif transition == 'RA':
            #  [..., n-1, n] :  n-1 -> n
            popped = self.stack.pop(-1)
            self.dependencies.append(( self.stack[-1], popped))
            
        ### END YOUR CODE

    def parse(self, transitions):
        """Applies the provided transitions to this PartialParse

        @param transitions (list of str): The list of transitions in the order they should be applied

        @return dsependencies (list of string tuples): The list of dependencies produced when
                                                        parsing the sentence. Represented as a list of
                                                        tuples where each tuple is of the form (head, dependent).
        """
        for transition in transitions:
            self.parse_step(transition)
            logger.debug(f'performing transition {transition} for sentence {self.sentence}')
        return self.dependencies<jupyter_output><empty_output><jupyter_text>(d) (6 points) Our network will predict which transition should be applied next to a partial parse. Wecould use it to parse a single sentence by applying predicted transitions until the parse is complete.However, neural networks run much more efficiently when making predictions aboutbatchesof dataat a time (i.e., predicting the next transition for any different partial parses simultaneously). Wecan parse sentences in minibatches with the following algorithm.---```Algorithm 1 Minibatch Dependency ParsingInput: sentences, a list of sentences to be parsed andmodel, our model that makes parse decisionsInitializepartialparsesas a list of PartialParses, one for each sentence insentencesInitializeunfinishedparsesas a shallow copy ofpartialparseswhile unfinished_parses is not empty do    Take the first batch size parses in unfinished_parsesas a minibatch    Use themodelto predict the next transition for each partial parse in the minibatch    Perform a parse step on each partial parse in the minibatch with its predicted transition    Remove the completed (empty buffer and stack of size 1) parses from unfinished_parsesend whileReturn:The dependencies for each (now completed) parse in partial_parses.```---Implement this algorithm in the minibatch parse function in `parser_transitions.py`. Youcan run basic (non-exhaustive) tests by running python `parser_transitions.py partd`.**Note**: You will need minibatch parse to be correctly implemented to evaluate the model you willbuild in part (e). However, you do not need it to train the model, so you should be able to completemost of part (e) even if minibatch parse is not implemented yet.<jupyter_code>def minibatch_parse(sentences, model, batch_size):
    """Parses a list of sentences in minibatches using a model.

    @param sentences (list of list of str): A list of sentences to be parsed
                                            (each sentence is a list of words and each word is of type string)
    @param model (ParserModel): The model that makes parsing decisions. It is assumed to have a function
                                model.predict(partial_parses) that takes in a list of PartialParses as input and
                                returns a list of transitions predicted for each parse. That is, after calling
                                    transitions = model.predict(partial_parses)
                                transitions[i] will be the next transition to apply to partial_parses[i].
    @param batch_size (int): The number of PartialParses to include in each minibatch


    @return dependencies (list of dependency lists): A list where each element is the dependencies
                                                    list for a parsed sentence. Ordering should be the
                                                    same as in sentences (i.e., dependencies[i] should
                                                    contain the parse for sentences[i]).
    """
    dependencies = []

    ### YOUR CODE HERE (~8-10 Lines)
    ### TODO:
    ###     Implement the minibatch parse algorithm as described in the pdf handout
    ###
    ###     Note: A shallow copy (as denoted in the PDF) can be made with the "=" sign in python, e.g.
    ###                 unfinished_parses = partial_parses[:].
    ###             Here `unfinished_parses` is a shallow copy of `partial_parses`.
    ###             In Python, a shallow copied list like `unfinished_parses` does not contain new instances
    ###             of the object stored in `partial_parses`. Rather both lists refer to the same objects.
    ###             In our case, `partial_parses` contains a list of partial parses. `unfinished_parses`
    ###             contains references to the same objects. Thus, you should NOT use the `del` operator
    ###             to remove objects from the `unfinished_parses` list. This will free the underlying memory that
    ###             is being accessed by `partial_parses` and may cause your code to crash.


    # init parser
    partial_parses = [PartialParse(s) for s in sentences]

    # make a shallow copy of partial_parses to track unfinished parses
    unfinished_parses = partial_parses[:]
    dependencies = [[] for _ in partial_parses]

    while unfinished_parses:
        for start  in range(0, len(unfinished_parses), batch_size):
            end = min(start + batch_size, len(unfinished_parses))
            minibatch = unfinished_parses[start:end]
            transitions = model.predict(minibatch)
            logger.debug(transitions)
            for e in range(len(transitions)):
                minibatch[e].parse([transitions[e]])

        # Remove finished parses from unfinished_parses
        remove = []
        for p in unfinished_parses:
            # if there are not items left in buffer and stack is  just 'ROOT', parse finished
            if (not p.buffer) and len(p.stack) == 1:
                dependencies[partial_parses.index(p)] = p.dependencies.copy()
                remove.append(p)
        for p in remove:
            unfinished_parses.remove(p)

    ### END YOUR CODE

    return dependencies<jupyter_output><empty_output><jupyter_text>We are now going to train a neural network to predict, given the state of the stack, buffer, anddependencies, which transition should be applied next. First, the model extracts a feature vectorrepresenting the current state. We will be using the feature set presented in the original neuraldependency parsing paper:A Fast and Accurate Dependency Parser using Neural Networks.^4 Thefunction extracting these features has been implemented for you inutils/parserutils.py.This feature vector consists of a list of tokens (e.g., the last word in the stack, first word in the buffer,dependent of the second-to-last word in the stack if there is one, etc.). They can be representedas a list of integers [w 1 ,w 2 ,...,wm] wheremis the number of features and each 0≤wi<|V|isthe index of a token in the vocabulary (|V|is the vocabulary size). First our network looks up anembedding for each word and concatenates them into a single input vector:$ x= [Ew 1 ,...,Ewm] \in R dm$where $E∈R|V|×d $ is an embedding matrix with each rowEwas the vector for a particular word w.(^4) Chen and Manning, 2014,http://cs.stanford.edu/people/danqi/papers/emnlp2014.pdf![dependency diagram](./img/model-arch.png)We then compute our prediction as:>$h= ReLU(xW+b 1 ) \\l=hU+b 2 \\\hat{y}= softmax(l)$wherehis referred to as the hidden layer,lis referred to as the logits,ˆy is referred to as thepredictions, and ReLU(z) = max(z,0)). We will train the model to minimize cross-entropy loss:> $J(θ) =CE(y,yˆ) =− \sum_i y_i \cdot \log(\hat y_i) $To compute the loss for the training set, we average thisJ(θ) across all training examples. (e) (10 points) In `parsermodel.py` you will find skeleton code to implement this simple neural net-work using PyTorch. Complete the init ,embedding lookup and forward functions toimplement the model. Then complete the train for epoch and train functions within the`run.py` file.Finally execute python run.py to train your model and compute predictions on test data fromPenn Treebank (annotated with Universal Dependencies). Make sure to turn off debug setting bysettingdebug=False in themainfunction ofrun.py.Hints:- When debugging, setdebug=Truein themainfunction ofrun.py. This will cause the code    to run over a small subset of the data, so that training the model won’t take as long. Make    sure to setdebug=Falseto run the full model once you are done debugging.- When running withdebug=True, you should be able to get a loss smaller than 0.2 and a UAS    larger than 65 on the dev set (although in rare cases your results may be lower, there is some    randomness when training).- It should take about1 hourto train the model on the entire the training dataset, i.e., when    debug=False.- When running withdebug=False, you should be able to get a loss smaller than 0.08 on the    train set and an Unlabeled Attachment Score larger than 87 on the dev set. For comparison,    the model in the original neural dependency parsing paper gets 92.5 UAS. If you want, you    can tweak the hyperparameters for your model (hidden layer size, hyperparameters for Adam,    number of epochs, etc.) to improve the performance (but you are not required to do so).Deliverables:- Working implementation of the neural dependency parser inparsermodel.py. (We’ll lookat and run this code for grading).- Report the best UAS your model achieves on the dev set and the UAS it achieves on the test set.<jupyter_code>class ParserModel(nn.Module):
    """ Feedforward neural network with an embedding layer and single hidden layer.
    The ParserModel will predict which transition should be applied to a
    given partial parse configuration.

    PyTorch Notes:
        - Note that "ParserModel" is a subclass of the "nn.Module" class. In PyTorch all neural networks
            are a subclass of this "nn.Module".
        - The "__init__" method is where you define all the layers and their respective parameters
            (embedding layers, linear layers, dropout layers, etc.).
        - "__init__" gets automatically called when you create a new instance of your class, e.g.
            when you write "m = ParserModel()".
        - Other methods of ParserModel can access variables that have "self." prefix. Thus,
            you should add the "self." prefix layers, values, etc. that you want to utilize
            in other ParserModel methods.
        - For further documentation on "nn.Module" please see https://pytorch.org/docs/stable/nn.html.
    """
    def __init__(self, embeddings, n_features=36,
        hidden_size=200, n_classes=3, dropout_prob=0.5):
        """ Initialize the parser model.

        @param embeddings (Tensor): word embeddings (num_words, embedding_size)
        @param n_features (int): number of input features
        @param hidden_size (int): number of hidden units
        @param n_classes (int): number of output classes
        @param dropout_prob (float): dropout probability
        """
        super(ParserModel, self).__init__()
        self.n_features = n_features
        self.n_classes = n_classes
        self.dropout_prob = dropout_prob
        self.embed_size = embeddings.shape[1]
        self.hidden_size = hidden_size
        self.pretrained_embeddings = nn.Embedding(embeddings.shape[0], self.embed_size)
        self.pretrained_embeddings.weight = nn.Parameter(torch.tensor(embeddings))

        ### YOUR CODE HERE (~5 Lines)
        ### TODO:
        ###     1) Construct `self.embed_to_hidden` linear layer, initializing the weight matrix
        ###         with the `nn.init.xavier_uniform_` function with `gain = 1` (default)
        ###     2) Construct `self.dropout` layer.
        ###     3) Construct `self.hidden_to_logits` linear layer, initializing the weight matrix
        ###         with the `nn.init.xavier_uniform_` function with `gain = 1` (default)
        ###
        ### Note: Here, we use Xavier Uniform Initialization for our Weight initialization.
        ###         It has been shown empirically, that this provides better initial weights
        ###         for training networks than random uniform initialization.
        ###         For more details checkout this great blogpost:
        ###             http://andyljones.tumblr.com/post/110998971763/an-explanation-of-xavier-initialization
        ### Hints:
        ###     - After you create a linear layer you can access the weight
        ###       matrix via:
        ###         linear_layer.weight
        ###
        ### Please see the following docs for support:
        ###     Linear Layer: https://pytorch.org/docs/stable/nn.html#torch.nn.Linear
        ###     Xavier Init: https://pytorch.org/docs/stable/nn.html#torch.nn.init.xavier_uniform_
        ###     Dropout: https://pytorch.org/docs/stable/nn.html#torch.nn.Dropout

        self.embed_to_hidden = nn.Linear(self.embed_size * self.n_features, self.hidden_size)
        nn.init.xavier_uniform_(self.embed_to_hidden.weight,gain = 1)

        self.dropout = nn.Dropout(self.dropout_prob)
        self.hidden_to_logits = nn.Linear(self.hidden_size, self.n_classes)
        nn.init.xavier_uniform_(self.hidden_to_logits.weight, gain = 1)

        ### END YOUR CODE

    def embedding_lookup(self, t):
        """ Utilize `self.pretrained_embeddings` to map input `t` from input tokens (integers)
            to embedding vectors.

            PyTorch Notes:
                - `self.pretrained_embeddings` is a torch.nn.Embedding object that we defined in __init__
                - Here `t` is a tensor where each row represents a list of features. Each feature is represented by an integer (input token).
                - In PyTorch the Embedding object, e.g. `self.pretrained_embeddings`, allows you to
                    go from an index to embedding. Please see the documentation (https://pytorch.org/docs/stable/nn.html#torch.nn.Embedding)
                    to learn how to use `self.pretrained_embeddings` to extract the embeddings for your tensor `t`.

            @param t (Tensor): input tensor of tokens (batch_size, n_features)

            @return x (Tensor): tensor of embeddings for words represented in t
                                (batch_size, n_features * embed_size)
        """
        ### YOUR CODE HERE (~1-3 Lines)
        ### TODO:
        ###     1) Use `self.pretrained_embeddings` to lookup the embeddings for the input tokens in `t`.
        ###     2) After you apply the embedding lookup, you will have a tensor shape (batch_size, n_features, embedding_size).
        ###         Use the tensor `view` method to reshape the embeddings tensor to (batch_size, n_features * embedding_size)
        ###
        ### Note: In order to get batch_size, you may need use the tensor .size() function:
        ###         https://pytorch.org/docs/stable/tensors.html#torch.Tensor.size
        ###
        ###  Please see the following docs for support:
        ###     Embedding Layer: https://pytorch.org/docs/stable/nn.html#torch.nn.Embedding
        ###     View: https://pytorch.org/docs/stable/tensors.html#torch.Tensor.view
        x = self.pretrained_embeddings(t)
        # logger.debug(f'size before reshape {x.size()}')
        # print(f'size before reshape {x.size()}')

        x = x.view(x.size()[0],-1)
        # logger.debug(f'size after reshape {x.size()}')
        # print(f'size after reshape {x.size()}')
        ### END YOUR CODE
        return x


    def forward(self, t):
        """ Run the model forward.

            Note that we will not apply the softmax function here because it is included in the loss function nn.CrossEntropyLoss

            PyTorch Notes:
                - Every nn.Module object (PyTorch model) has a `forward` function.
                - When you apply your nn.Module to an input tensor `t` this function is applied to the tensor.
                    For example, if you created an instance of your ParserModel and applied it to some `t` as follows,
                    the `forward` function would called on `t` and the result would be stored in the `output` variable:
                        model = ParserModel()
                        output = model(t) # this calls the forward function
                - For more details checkout: https://pytorch.org/docs/stable/nn.html#torch.nn.Module.forward

        @param t (Tensor): input tensor of tokens (batch_size, n_features)

        @return logits (Tensor): tensor of predictions (output after applying the layers of the network)
                                 without applying softmax (batch_size, n_classes)
        """
        ###  YOUR CODE HERE (~3-5 lines)
        ### TODO:
        ###     1) Apply `self.embedding_lookup` to `t` to get the embeddings
        ###     2) Apply `embed_to_hidden` linear layer to the embeddings
        ###     3) Apply relu non-linearity to the output of step 2 to get the hidden units.
        ###     4) Apply dropout layer to the output of step 3.
        ###     5) Apply `hidden_to_logits` layer to the output of step 4 to get the logits.
        ###
        ### Note: We do not apply the softmax to the logits here, because
        ### the loss function (torch.nn.CrossEntropyLoss) applies it more efficiently.
        ###
        ### Please see the following docs for support:
        ###     ReLU: https://pytorch.org/docs/stable/nn.html?highlight=relu#torch.nn.functional.relu
        # print(f'size forward reshape {t.size()}')

        embed = self.embedding_lookup(t)
        hidden = self.embed_to_hidden(embed)
        x = self.hidden_to_logits(hidden)
        x = self.dropout(x)
        logits = F.relu(x)


        ### END YOUR CODE
        return logits
from utils import parser_utils

parser, embeddings, train_data, dev_data, test_data = parser_utils.load_and_preprocess_data(False)

import parser_model
model = parser_model.ParserModel(embeddings)
model
# load trained model for playing

model_file = '/Users/mupadhye/git/mandarup/lab/courses/cs224n/hw/a3/results/model-20190907_225211/model.weights'

# assign model to parser
parser.model = model
    
print("Restoring the best model weights found on the dev set")
parser.model.load_state_dict(torch.load(model_file))
print("Final evaluation on test set",)
parser.model.eval()

UAS, dependencies = parser.parse(test_data)<jupyter_output>2919736it [00:00, 43968578.48it/s]<jupyter_text>Analyze parsed dependenciesLet's Take a look at 1st sentence in test data<jupyter_code># dependency prediction on first sentence
test_dep = dependencies[0]
test_dep
# first sentence
test_data[0]
# id2tok to get human readable
test_sent = [parser.id2tok[w] for w in test_data[0]['word']]
test_sent
for h,w in test_dep:
    print(test_sent[h], ' --> ', test_sent[w])<jupyter_output>,  -->  no
monday  -->  black
monday  -->  n't
monday  -->  was
monday  -->  it
monday  -->  ,
monday  -->  .
<ROOT>  -->  monday<jupyter_text>Randomly selected sentence<jupyter_code>def analyze_dep(sentence, dependencies):
    print(f'sentence: {sentence}')
    print('-'*20)
    print('DEPENDENCIES')
    print('-'*20)
    for h,w in dependencies:
        print(f'\t{sentence[h]}  -->  {sentence[w]}')
        

def id2words(word_ids, id2tok):
    return [id2tok[w] for w in word_ids]
        
def analyze_random_sent(test_data, parser, dependencies):
    rndix = np.random.randint(0, len(test_data))
    word_ids = test_data[rndix]['word']
    sent = id2words(word_ids, parser.id2tok)
    sent_dep = dependencies[rndix]
    analyze_dep(sent, sent_dep)
    
analyze_random_sent(test_data, parser, dependencies)<jupyter_output>sentence: ['<ROOT>', '2', '.', 'that', 'mr.', 'lantos', 'supported', 'the', 'rights', 'of', 'the', 'witnesses', 'to', 'take', 'the', 'fifth', 'amendment', '.']
--------------------
DEPENDENCIES
--------------------
	2  -->  .
	lantos  -->  mr.
	supported  -->  lantos
	supported  -->  that
	supported  -->  2
	rights  -->  the
	witnesses  -->  the
	witnesses  -->  of
	rights  -->  witnesses
	take  -->  to
	amendment  -->  fifth
	amendment  -->  the
	take  -->  amendment
	rights  -->  take
	supported  -->  rights
	supported  -->  .
	<ROOT>  -->  supported<jupyter_text>What does the data look like? How is the data parsed?A sample from `data/dev.conll````    1	Influential	_	ADJ	JJ	_	2	amod	_	_    2	members	_	NOUN	NNS	_	10	nsubj	_	_    3	of	_	ADP	IN	_	6	case	_	_    4	the	_	DET	DT	_	6	det	_	_    ...    33	sale	_	NOUN	NN	_	28	nmod	_	_    34	of	_	ADP	IN	_	36	case	_	_    35	sick	_	ADJ	JJ	_	36	amod	_	_    36	thrifts	_	NOUN	NNS	_	33	nmod	_	_    37	.	_	PUNCT	.	_	10	punct	_	_    1	The	_	DET	DT	_	2	det	_	_``` Parsing```python    >>>line.strip().split('\t')    ...    >>>examples.append({'word': word, 'pos': pos, 'head': head, 'label': label})        >>>examples    [{'word': ['influential', 'members', 'of', 'the',...],       'pos': ['JJ', 'NNS', 'IN', 'DT',...],        'head': [2, 10, 6, 6,...],        'label': ['amod', 'nsubj', 'case', 'det',...]     },     {         ...     }```<jupyter_code>import utils
import utils.parser_utils

help(utils.parser_utils.read_conll)

# NOTE: use SHIFT-TAB to see docstrings<jupyter_output>Help on function read_conll in module utils.parser_utils:

read_conll(in_file, lowercase=False, max_example=None)
    Parse data.
    
    Parse raw data into python objects for further preprocessing.
    
    Args:
        in_file (str) : path to data
        lowercase ( bool)
        max_example ( int)
    
    Returns:
        (dict) : {'word': <word>, 'pos': <pos>, 'head': <head>, 'label': <label>}
{'fraction_non_alphanumeric': 0.09010034872962779, 'fraction_numerical': 0.008753825350508861, 'mean_word_length': 2.754575818303273, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 70, 'https://': 12, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 16, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '385866', 'license': 'MIT', 'n_tokens_mistral': 7880, 'n_tokens_neox': 7561, 'n_words': 3102, 'path': 'courses/cs224n/hw/a3/README.ipynb', 'repo_name': 'mandarup/lab140', 'size': '28102'}