<jupyter_start><jupyter_text>Homework 1: Word similarity tasks<jupyter_code>__author__ = "<NAME>"
__version__ = "CS224u, Stanford, Spring 2019"<jupyter_output><empty_output><jupyter_text>Contents1. [Overview](Overview)1. [Set-up](Set-up)1. [Dataset readers](Dataset-readers)1. [Dataset comparisons](Dataset-comparisons)  1. [Vocab overlap](Vocab-overlap)  1. [Pair overlap and score correlations](Pair-overlap-and-score-correlations)1. [Evaluation](Evaluation)  1. [Dataset evaluation](Dataset-evaluation)  1. [Dataset error analysis](Dataset-error-analysis)  1. [Full evaluation](Full-evaluation)1. [Homework questions](Homework-questions)  1. [PPMI as a baseline [1 point]](PPMI-as-a-baseline-[1-point])  1. [Gigaword with LSA at a few dimensions [1 point]](Gigaword-with-LSA-at-a-few-dimensions-[1-point])  1. [Gigaword with GloVe for a small number of iterations [1 point]](Gigaword-with-GloVe-for-a-small-number-of-iterations-[1-point])  1. [t-test reweighting [2 points]](t-test-reweighting-[2-points])  1. [Your original system [4 points]](Your-original-system-[4-points])1. [Bake-off [1 point]](Bake-off-[1-point])  OverviewWord similarity datasets have long been used to evaluate distributed representations. This notebook provides basic code for conducting such analyses with a number of datasets:| Dataset | Pairs | Task-type | Current best Spearman $\rho$ | Best $\rho$ paper |   ||---------|-------|-----------|------------------------------|-------------------|---|| [WordSim-353](http://www.cs.technion.ac.il/~gabr/resources/data/wordsim353/) | 353 | Relatedness | 82.8 | [Speer et al. 2017](https://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14972) || [MTurk-771](http://www2.mta.ac.il/~gideon/mturk771.html) | 771 | Relatedness | 81.0 | [Speer et al. 2017](https://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14972) || [The MEN Test Collection](http://clic.cimec.unitn.it/~elia.bruni/MEN) | 3,000 | Relatedness | 86.6 | [Speer et al. 2017](https://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14972)  | | [SimVerb-3500-dev](http://people.ds.cam.ac.uk/dsg40/simverb.html) | 500 | Similarity | 61.1 | [Mrki&scaron;&cacute; et al. 2016](https://arxiv.org/pdf/1603.00892.pdf) || [SimVerb-3500-test](http://people.ds.cam.ac.uk/dsg40/simverb.html) | 3,000 | Similarity | 62.4 | [Mrki&scaron;&cacute; et al. 2016](https://arxiv.org/pdf/1603.00892.pdf) |Each of the similarity datasets contains word pairs with an associated human-annotated similarity score. (We convert these to distances to align intuitively with our distance measure functions.) The evaluation code measures the distance between the word pairs in your chosen VSM (which should be a `pd.DataFrame`).The evaluation metric for each dataset is the [Spearman correlation coefficient $\rho$](https://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient) between the annotated scores and your distances, as is standard in the literature. We also macro-average these correlations across the datasets for an overall summary. (In using the macro-average, we are saying that we care about all the datasets equally, even though they vary in size.)This homework ([questions at the bottom of this notebook](Homework-questions)) asks you to write code that uses the count matrices in `data/vsmdata` to create and evaluate some baseline models as well as an original model $M$ that you design. This accounts for 9 of the 10 points for this assignment.For the associated bake-off, we will distribute two new word similarity or relatedness datasets and associated reader code, and you will evaluate $M$ (no additional training or tuning allowed!) on those new datasets. Systems that enter will receive the additional homework point, and systems that achieve the top score will receive an additional 0.5 points.  Set-up<jupyter_code>from collections import defaultdict
import csv
import itertools
import numpy as np
import os
import pandas as pd
from scipy.stats import spearmanr
import vsm
VSM_HOME = os.path.join('../data', 'vsmdata')

WORDSIM_HOME = os.path.join('../data', 'wordsim')<jupyter_output><empty_output><jupyter_text>Dataset readers<jupyter_code>def wordsim_dataset_reader(
        src_filename, 
        header=False, 
        delimiter=',', 
        score_col_index=2):
    """Basic reader that works for all similarity datasets. They are 
    all tabular-style releases where the first two columns give the 
    word and a later column (`score_col_index`) gives the score.

    Parameters
    ----------
    src_filename : str
        Full path to the source file.
    header : bool
        Whether `src_filename` has a header. Default: False
    delimiter : str
        Field delimiter in `src_filename`. Default: ','
    score_col_index : int
        Column containing the similarity scores Default: 2

    Yields
    ------
    (str, str, float)
       (w1, w2, score) where `score` is the negative of the similarity
       score in the file so that we are intuitively aligned with our
       distance-based code. To align with our VSMs, all the words are 
       downcased.

    """
    with open(src_filename, encoding='utf8') as f:
        reader = csv.reader(f, delimiter=delimiter)
        if header:
            next(reader)
        for row in reader:
            w1 = row[0].strip().lower()
            w2 = row[1].strip().lower()
            score = row[score_col_index]
            # Negative of scores to align intuitively with distance functions:
            score = -float(score)
            yield (w1, w2, score)

def wordsim353_reader():
    """WordSim-353: http://www.cs.technion.ac.il/~gabr/resources/data/wordsim353/"""
    src_filename = os.path.join(
        WORDSIM_HOME, 'wordsim353', 'combined.csv')
    return wordsim_dataset_reader(
        src_filename, header=True)

def mturk771_reader():
    """MTURK-771: http://www2.mta.ac.il/~gideon/mturk771.html"""
    src_filename = os.path.join(
        WORDSIM_HOME, 'MTURK-771.csv')
    return wordsim_dataset_reader(
        src_filename, header=False)

def simverb3500dev_reader():
    """SimVerb-3500: http://people.ds.cam.ac.uk/dsg40/simverb.html"""
    src_filename = os.path.join(
        WORDSIM_HOME, 'SimVerb-3500', 'SimVerb-500-dev.txt')
    return wordsim_dataset_reader(
        src_filename, delimiter="\t", header=True, score_col_index=3)

def simverb3500test_reader():
    """SimVerb-3500: http://people.ds.cam.ac.uk/dsg40/simverb.html"""
    src_filename = os.path.join(
        WORDSIM_HOME, 'SimVerb-3500', 'SimVerb-3000-test.txt')
    return wordsim_dataset_reader(
        src_filename, delimiter="\t", header=True, score_col_index=3)

def men_reader():
    """MEN: http://clic.cimec.unitn.it/~elia.bruni/MEN"""
    src_filename = os.path.join(
        WORDSIM_HOME, 'MEN', 'MEN_dataset_natural_form_full')
    return wordsim_dataset_reader(
        src_filename, header=False, delimiter=' ')<jupyter_output><empty_output><jupyter_text>This collection of readers will be useful for flexible evaluations:<jupyter_code>READERS = (wordsim353_reader, mturk771_reader, simverb3500dev_reader, 
           simverb3500test_reader, men_reader)<jupyter_output><empty_output><jupyter_text>Dataset comparisonsThis section does some basic analysis of the datasets. The goal is to obtain a deeper understanding of what problem we're solving – what strengths and weaknesses the datasets have and how they relate to each other. For a full-fledged project, we would want to continue work like this and report on it in the paper, to provide context for the results.<jupyter_code>def get_reader_name(reader):
    """Return a cleaned-up name for the similarity dataset 
    iterator `reader`
    """
    return reader.__name__.replace("_reader", "")<jupyter_output><empty_output><jupyter_text>Vocab overlapHow many vocabulary items are shared across the datasets?<jupyter_code>def get_reader_vocab(reader):
    """Return the set of words (str) in `reader`."""
    vocab = set()
    for w1, w2, _ in reader():
        vocab.add(w1)
        vocab.add(w2)
    return vocab
def get_reader_vocab_overlap(readers=READERS):
    """Get data on the vocab-level relationships between pairs of 
    readers. Returns a a pd.DataFrame containing this information.
    """
    data = []
    for r1, r2 in itertools.product(readers, repeat=2):       
        v1 = get_reader_vocab(r1)
        v2 = get_reader_vocab(r2)
        d = {
            'd1': get_reader_name(r1),
            'd2': get_reader_name(r2),
            'overlap': len(v1 & v2), 
            'union': len(v1 | v2),
            'd1_size': len(v1),
            'd2_size': len(v2)}
        data.append(d)
    return pd.DataFrame(data)
vocab_overlap = get_reader_vocab_overlap()
def vocab_overlap_crosstab(vocab_overlap):
    """Return an intuitively formatted `pd.DataFrame` giving 
    vocab-overlap counts for all the datasets represented in 
    `vocab_overlap`, the output of `get_reader_vocab_overlap`.
    """        
    xtab = pd.crosstab(
        vocab_overlap['d1'], 
        vocab_overlap['d2'], 
        values=vocab_overlap['overlap'], 
        aggfunc=np.mean)
    # Blank out the upper left to reduce visual clutter:
    for i in range(0, xtab.shape[0]):
        for j in range(i+1, xtab.shape[1]):
            xtab.iloc[i, j] = ''        
    return xtab        
vocab_overlap_crosstab(vocab_overlap)<jupyter_output><empty_output><jupyter_text>This looks reasonable. By design, the SimVerb dev and test sets have a lot of overlap. The other overlap numbers are pretty small, even adjusting for dataset size.  Pair overlap and score correlationsHow many word pairs are shared across datasets and, for shared pairs, what is the correlation between their scores? That is, do the datasets agree?<jupyter_code>def get_reader_pairs(reader):
    """Return the set of alphabetically-sorted word (str) tuples 
    in `reader`
    """
    return {tuple(sorted([w1, w2])): score for w1, w2, score in reader()}
def get_reader_pair_overlap(readers=READERS):
    """Return a `pd.DataFrame` giving the number of overlapping 
    word-pairs in pairs of readers, along with the Spearman 
    correlations.
    """    
    data = []
    for r1, r2 in itertools.product(READERS, repeat=2):
        if r1.__name__ != r2.__name__:
            d1 = get_reader_pairs(r1)
            d2 = get_reader_pairs(r2)
            overlap = []
            for p, s in d1.items():
                if p in d2:
                    overlap.append([s, d2[p]])
            if overlap:
                s1, s2 = zip(*overlap)
                rho = spearmanr(s1, s2)[0]
            else:
                rho = None        
            d = {
                'd1': get_reader_name(r1),
                'd2': get_reader_name(r2), 
                'pair_overlap': len(overlap),
                'rho': rho}
            data.append(d)
    df = pd.DataFrame(data)
    df = df.sort_values('pair_overlap', ascending=False)
    # Return only every other row to avoid repeats:
    return df[::2].reset_index(drop=True)
get_reader_pair_overlap()<jupyter_output><empty_output><jupyter_text>This looks reasonable: none of the datasets have a lot of overlapping pairs, so we don't have to worry too much about places where they give conflicting scores.  EvaluationThis section builds up the evaluation code that you'll use for the homework and bake-off. For illustrations, I'll read in a VSM created from `data/vsmdata/giga_window5-scaled.csv.gz`:<jupyter_code>giga5 = pd.read_csv(
    os.path.join(VSM_HOME, "giga_window5-scaled.csv.gz"), index_col=0)<jupyter_output><empty_output><jupyter_text>Dataset evaluation<jupyter_code>def word_similarity_evaluation(reader, df, distfunc=vsm.cosine):
    """Word-similarity evaluation framework.
    
    Parameters
    ----------
    reader : iterator
        A reader for a word-similarity dataset. Just has to yield
        tuples (word1, word2, score).    
    df : pd.DataFrame
        The VSM being evaluated.        
    distfunc : function mapping vector pairs to floats.
        The measure of distance between vectors. Can also be 
        `vsm.euclidean`, `vsm.matching`, `vsm.jaccard`, as well as 
        any other distance measure between 1d vectors.    
        
    Raises
    ------
    ValueError
        If `df.index` is not a subset of the words in `reader`.
    
    Returns
    -------
    float, data
        `float` is the Spearman rank correlation coefficient between 
        the dataset scores and the similarity values obtained from 
        `df` using  `distfunc`. This evaluation is sensitive only to 
        rankings, not to absolute values.  `data` is a `pd.DataFrame` 
        with columns['word1', 'word2', 'score', 'distance'].
        
    """
    data = []
    for w1, w2, score in reader():
        d = {'word1': w1, 'word2': w2, 'score': score}
        for w in [w1, w2]:
            if w not in df.index:
                raise ValueError(
                    "Word '{}' is in the similarity dataset {} but not in the "
                    "DataFrame, making this evaluation ill-defined. Please "
                    "switch to a DataFrame with an appropriate vocabulary.".
                    format(w, get_reader_name(reader))) 
        d['distance'] = distfunc(df.loc[w1], df.loc[w2])
        data.append(d)
    data = pd.DataFrame(data)
    rho, pvalue = spearmanr(data['score'].values, data['distance'].values)
    return rho, data
rho, eval_df = word_similarity_evaluation(men_reader, giga5)
rho
eval_df.head()<jupyter_output><empty_output><jupyter_text>Dataset error analysisFor error analysis, we can look at the words with the largest delta between the gold score and the distance value in our VSM. We do these comparisons based on ranks, just as with our primary metric (Spearman $\rho$), and we normalize both rankings so that they have a comparable number of levels.<jupyter_code>def word_similarity_error_analysis(eval_df):    
    eval_df['distance_rank'] = _normalized_ranking(eval_df['distance'])
    eval_df['score_rank'] = _normalized_ranking(eval_df['score'])
    eval_df['error'] =  abs(eval_df['distance_rank'] - eval_df['score_rank'])
    return eval_df.sort_values('error')
    
    
def _normalized_ranking(series):
    ranks = series.rank(method='dense')
    return ranks / ranks.sum()<jupyter_output><empty_output><jupyter_text>Best predictions:<jupyter_code>word_similarity_error_analysis(eval_df).head()<jupyter_output><empty_output><jupyter_text>Worst predictions:<jupyter_code>word_similarity_error_analysis(eval_df).tail()<jupyter_output><empty_output><jupyter_text>Full evaluation A full evaluation is just a loop over all the readers on which one want to evaluate, with a macro-average at the end:<jupyter_code>def full_word_similarity_evaluation(df, readers=READERS, distfunc=vsm.cosine):
    """Evaluate a VSM against all datasets in `readers`.
    
    Parameters
    ----------
    df : pd.DataFrame
    readers : tuple 
        The similarity dataset readers on which to evaluate.
    distfunc : function mapping vector pairs to floats.
        The measure of distance between vectors. Can also be 
        `vsm.euclidean`, `vsm.matching`, `vsm.jaccard`, as well as 
        any other distance measure between 1d vectors. 
    
    Returns
    -------
    pd.Series
        Mapping dataset names to Spearman r values.
        
    """        
    scores = {}     
    for reader in readers:
        score, data_df = word_similarity_evaluation(reader, df, distfunc=distfunc)
        scores[get_reader_name(reader)] = score
    series = pd.Series(scores, name='Spearman r')
    series['Macro-average'] = series.mean()
    return series
full_word_similarity_evaluation(giga5, distfunc=vsm.cosine)<jupyter_output><empty_output><jupyter_text>Homework questionsPlease embed your homework responses in this notebook, and do not delete any cells from the notebook. (You are free to add as many cells as you like as part of your responses.)  PPMI as a baseline [1 point]The insight behind PPMI is a recurring theme in word representation learning, so it is a natural baseline for our task. For this question, submit code to do the following:1. Read the two count matrices created with a window of 20 and a flat scaling function into `pd.DataFrame`s, as is done in the VSM notebooks. The files are `data/vsmdata/giga_window20-flat.csv.gz` and `data/vsmdata/imdb_window20-flat.csv.gz`, and the VSM notebooks provide examples of the needed code.1. Reweight these count matries with PPMI.1. Evaluate these reweighted matrices using `full_word_similarity_evaluation`.<jupyter_code>giga20_flat = pd.read_csv(os.path.join(VSM_HOME, 'giga_window20-flat.csv.gz'), index_col=0)
imdb20_flat = pd.read_csv(os.path.join(VSM_HOME, 'imdb_window20-flat.csv.gz'), index_col=0)
print(giga20_flat.shape, imdb20_flat.shape)
%%time
def ppmi(df):
    X = df.values
    def oe(X):
        colsum = X.sum(axis=0)
        rowsum = X.sum(axis=1)[:, np.newaxis]
        N = X.sum()
        observed = rowsum * colsum / N
        return X / observed

    oe_ = oe(X)
    oe_[oe_ <= 0] = 1
    return pd.DataFrame(np.log(oe_), columns=df.columns, index=df.index)

giga20_ppmi = ppmi(giga20_flat)
imdb20_ppmi = ppmi(imdb20_flat)
full_word_similarity_evaluation(giga20_ppmi, distfunc=vsm.cosine)
full_word_similarity_evaluation(imdb20_ppmi, distfunc=vsm.cosine)<jupyter_output><empty_output><jupyter_text>Gigaword with LSA at a few dimensions [1 point]We might expect PPMI and LSA to form a solid pipeline that combines the strengths of PPMI with those of dimensionality reduction. However, LSA has a hyper-parameter $k$ – the dimensionality of the final representations – that will impact performance. For this problem, submit code to do the following:1. Apply LSA with $k \in  \{100, 500, 1000\}$ to the PPMI reweighted version of `data/vsmdata/giga_window20-flat.csv.gz` that you created in the previous question.2. Print out each $k$ and its associated score. (For concise, clear code, you can use `results.loc['Macro-average']` where `results` is a `pd.DataFrame` returned by `full_word_similarity_evaluation`.)<jupyter_code>k_vals = [100, 500, 1000]

for i,k in enumerate(k_vals):
    latent = vsm.lsa(giga20_ppmi, k=k)
    results = full_word_similarity_evaluation(latent)
    print(k, '\t',results.loc['Macro-average'])<jupyter_output>100 	 0.41865756501944895
500 	 0.4264778077690529
1000 	 0.42375690439522157<jupyter_text>Gigaword with GloVe for a small number of iterations [1 point]Ideally, we would run GloVe for a very large number of iterations on a GPU machine to compare it against its close cousin PMI. However, we don't want this homework to cost you a lot of money or monopolize a lot of your available computing resources, so let's instead just probe GloVe a little bit to see if it has promise for our task. For this problem, submit code to do the following:1. Run GloVe for 10, 100, and 200 iterations on `data/vsmdata/giga_window20-flat.csv.gz`, using the `mittens` implementation of `GloVe`.   * For all the other parameters to `mittens.GloVe` besides `max_iter`, use the package's defaults.  * Because of the way that implementation is designed, these will have to be separate runs, but they should be relatively quick. 1. Print out each value of `max_iter` and its associated score according to `full_word_similarity_evaluation`. The trend should give you a sense for whether it is worth running GloVe for more iterations.  * Note: your trained GloVe matrix `X` needs to be wrapped in a `pd.DataFrame` to work with `full_word_similarity_evaluation`. `pd.DataFrame(X, index=giga20.index)` will do the trick.  * Note: if `glv` is your GloVe model, then running `glv.sess.close()` after each model is trained will silence warnings from TensorFlow about interactive sessions being active.<jupyter_code>from mittens import GloVe

iterations = [10, 100, 200]
for it in iterations:
    glv = GloVe(max_iter=it)
    X = glv.fit(giga20_ppmi.values)
    df = pd.DataFrame(X, index=giga20_ppmi.index)
    results = full_word_similarity_evaluation(df)
    print(f'iterations = {it}')
    print(results)
    print()
    glv.sess.close()<jupyter_output>WARNING:tensorflow:From /mnt/c/Users/user/Documents/Academic/Stanford CS224u/cs224u-spring-2019/nlu/lib/python3.7/site-packages/mittens/tf_mittens.py:54: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.

WARNING:tensorflow:From /mnt/c/Users/user/Documents/Academic/Stanford CS224u/cs224u-spring-2019/nlu/lib/python3.7/site-packages/mittens/tf_mittens.py:55: The name tf.InteractiveSession is deprecated. Please use tf.compat.v1.InteractiveSession instead.

WARNING:tensorflow:From /mnt/c/Users/user/Documents/Academic/Stanford CS224u/cs224u-spring-2019/nlu/lib/python3.7/site-packages/mittens/tf_mittens.py:206: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

WARNING:tensorflow:From /mnt/c/Users/user/Documents/Academic/Stanford CS224u/cs224u-spring-2019/nlu/lib/python3.7/site-packages/mittens/tf_mittens.py:161: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorfl[...]<jupyter_text>t-test reweighting [2 points] The t-test statistic can be thought of as a reweighting scheme. For a count matrix $X$, row index $i$, and column index $j$:$$\textbf{ttest}(X, i, j) = \frac{    P(X, i, j) - \big(P(X, i, *)P(X, *, j)\big)}{\sqrt{(P(X, i, *)P(X, *, j))}}$$where $P(X, i, j)$ is $X_{ij}$ divided by the total values in $X$, $P(X, i, *)$ is the sum of the values in row $i$ of $X$ divided by the total values in $X$, and $P(X, *, j)$ is the sum of the values in column $j$ of $X$ divided by the total values in $X$.For this problem, implement this reweighting scheme. You can use `test_ttest_implementation` below to check that your implementation is correct. You do not need to use this for any evaluations, though we hope you will be curious enough to do so!<jupyter_code>def test_ttest_implementation(func):
    """`func` should be an implementation of t-test reweighting as 
    defined above.
    """
    X = pd.DataFrame(np.array([
        [  4.,   4.,   2.,   0.],
        [  4.,  61.,   8.,  18.],
        [  2.,   8.,  10.,   0.],
        [  0.,  18.,   0.,   5.]]))    
    actual = np.array([
        [ 0.33056, -0.07689,  0.04321, -0.10532],
        [-0.07689,  0.03839, -0.10874,  0.07574],
        [ 0.04321, -0.10874,  0.36111, -0.14894],
        [-0.10532,  0.07574, -0.14894,  0.05767]])    
    predicted = func(X)
    assert np.array_equal(predicted.round(5), actual)
def ttest_rew8(df):
    X = df.values
    N = X.sum()
    P = X / N
    Pi_ = X.sum(axis=1)[:, np.newaxis] / N
    P_j = X.sum(axis=0) / N
    ttest = (P - Pi_ * P_j) / (Pi_ * P_j)**.5
    return ttest

test_ttest_implementation(ttest_rew8)<jupyter_output><empty_output><jupyter_text>Your original system [4 points]This question asks you to design your own model. You can of course include steps made above (ideally, the above questions informed your system design!), but your model should not be literally identical to any of the above models. Other ideas: retrofitting, autoencoders, GloVe, subword modeling, ... Your code needs to be able to 1. operate on all of the count matrices in `data/vsmdata` (except `gigawordnyt-advmod-matrix.csv.gz`, which doesn't have the right vocab for this task and is included just for fun); __other pretrained vectors cannot be introduced__; and 1. be self-contained, so that we can work with your model directly in your homework submission notebook. (If your model depends on external data or other resources, please upload to Canvas a zip archive containing those resources and your submission notebook.)  Bake-off [1 point]For the bake-off, we will release two additional datasets right after class on April 15. The announcement will go out on Piazza. We will also release reader code for these datasets that you can paste into this notebook. You will evaluate your custom model $M$ (from the previous question) on these new datasets using `full_word_similarity_evaluation`. Rules:1. Only one evaluation is permitted.1. No additional system tuning is permitted once the bake-off has started.To enter the bake-off, upload this notebook on Canvas:https://canvas.stanford.edu/courses/99711/assignments/187240The cells below this one constitute your bake-off entry.People who enter will receive the additional homework point, and people whose systems achieve the top score will receive an additional 0.5 points. We will test the top-performing systems ourselves, and only systems for which we can reproduce the reported results will win the extra 0.5 points.The bake-off will close at 4:30 pm on April 17. Late entries will be accepted, but they cannot earn the extra 0.5 points. Similarly, you cannot win the bake-off unless your homework is submitted on time.<jupyter_code># Enter your bake-off assessment code into this cell. 
# Please do not remove this comment.


# On an otherwise blank line in this cell, please enter
# your "Macro-average" value as reported by the code above. 
# Please enter only a number between 0 and 1 inclusive.
# Please do not remove this comment.<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10601548540798095, 'fraction_numerical': 0.02767520349414334, 'mean_word_length': 3.5552541146681134, 'pattern_counts': {'":': 0, '<': 73, '<?xml version=': 0, '>': 72, 'https://': 7, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '52905', 'license': 'Apache-2.0', 'n_tokens_mistral': 8396, 'n_tokens_neox': 7777, 'n_words': 2548, 'path': 'hw1_wordsim.ipynb', 'repo_name': 'samehamin/cs224u', 'size': '25196'}