<jupyter_start><jupyter_text>Homework and bake-off: pragmatic color descriptions<jupyter_code>__author__ = "<NAME>"
__version__ = "CS224u, Stanford, Spring 2020"<jupyter_output><empty_output><jupyter_text>Contents1. [Overview](Overview)1. [Set-up](Set-up)1. [All two-word examples as a dev corpus](All-two-word-examples-as-a-dev-corpus)1. [Dev dataset](Dev-dataset)1. [Random train–test split for development](Random-train–test-split-for-development)1. [Question 1: Improve the tokenizer [1 point]](Question-1:-Improve-the-tokenizer-[1-point])1. [Use the tokenizer](Use-the-tokenizer)1. [Question 2: Improve the color representations [1 point]](Question-2:-Improve-the-color-representations-[1-point])1. [Use the color representer](Use-the-color-representer)1. [Initial model](Initial-model)1. [Question 3: GloVe embeddings [1 points]](Question-3:-GloVe-embeddings-[1-points])1. [Try the GloVe representations](Try-the-GloVe-representations)1. [Question 4: Color context [3 points]](Question-4:-Color-context-[3-points])1. [Your original system [3 points]](Your-original-system-[3-points])1. [Bakeoff [1 point]](Bakeoff-[1-point])  OverviewThis homework and associated bake-off are oriented toward building an effective system for generating color descriptions that are pragmatic in the sense that they would help a reader/listener figure out which color was being referred to in a shared context consisting of a target color (whose identity is known only to the describer/speaker) and a set of distractors.The notebook [colors_overview.ipynb](colors_overview.ipynb) should be studied before work on this homework begins. That notebook provides backgroud on the task, the dataset, and the modeling code that you will be using and adapting.The homework questions are more open-ended than previous ones have been. Rather than asking you to implement pre-defined functionality, they ask you to try to improve baseline components of the full system in ways that you find to be effective. As usual, this culiminates in a prompt asking you to develop a novel system for entry into the bake-off. In this case, though, the work you do for the homework will likely be directly incorporated into that system.  Set-up See [colors_overview.ipynb](colors_overview.ipynb) for set-up in instructions and other background details.<jupyter_code>from colors import ColorsCorpusReader
import os
from sklearn.model_selection import train_test_split
from torch_listener_with_attention import (
    AttentionalColorizedNeuralListener, create_example_dataset)
import utils
from utils import START_SYMBOL, END_SYMBOL, UNK_SYMBOL
import numpy as np
utils.fix_random_seeds()
COLORS_SRC_FILENAME = os.path.join(
    "data", "colors", "filteredCorpus.csv")<jupyter_output><empty_output><jupyter_text>Dev datasets<jupyter_code>def load_from_pickle():
    import pickle 
    with open('dev_vocab.pickle', 'rb') as handle:
        dev_vocab = pickle.load(handle)
    with open('dev_seqs_test.pickle', 'rb') as handle:
        dev_seqs_test = pickle.load(handle)
    with open('dev_seqs_train.pickle', 'rb') as handle:
        dev_seqs_train = pickle.load(handle)
    with open('dev_cols_test.pickle', 'rb') as handle:
        dev_cols_test = pickle.load(handle)
    with open('dev_cols_train.pickle', 'rb') as handle:
        dev_cols_train = pickle.load(handle)
    with open('embedding.pickle', 'rb') as handle:
        embedding = pickle.load(handle)
    return dev_vocab, dev_seqs_test, dev_seqs_train, dev_cols_test, dev_cols_train, embedding
dev_vocab, dev_seqs_test, dev_seqs_train, dev_cols_test, dev_cols_train, embedding = load_from_pickle()<jupyter_output><empty_output><jupyter_text>Question 4: Color context [3 points]<jupyter_code>toy_color_seqs, toy_word_seqs, toy_vocab = create_example_dataset(
    group_size=50, vec_dim=2)
toy_color_seqs_train, toy_color_seqs_test, toy_word_seqs_train, toy_word_seqs_test = \
    train_test_split(toy_color_seqs, toy_word_seqs)
toy_mod = AttentionalColorizedNeuralListener(
    toy_vocab, 
    embed_dim=100, 
    embedding=embedding,
    hidden_dim=100, 
    max_iter=100, 
    batch_size=128)
_ = toy_mod.fit(toy_color_seqs_train, toy_word_seqs_train)
preds = toy_mod.predict(toy_color_seqs_test, toy_word_seqs_test)
correct = sum([1 if x == 2 else 0 for x in preds])
print(correct, "/", len(preds), correct/len(preds))<jupyter_output>27 / 38 0.7105263157894737<jupyter_text>If that worked, then you can now try this model on SCC problems!<jupyter_code>dev_color_mod = AttentionalColorizedNeuralListener(
    dev_vocab, 
    #embedding=dev_glove_embedding, 
    embed_dim=100,
    embedding=embedding,
    hidden_dim=100, 
    max_iter=500,
    batch_size=32,
    dropout_prob=0.7,
    eta=0.001,
    lr_rate=0.96,
    warm_start=True,
    device='cuda')
# Uncomment line if you want to continue training the previous model
# literal_listener.load_model("literal_listener.pt")

_ = dev_color_mod.fit(dev_cols_train, dev_seqs_train)
#import torch
#torch.cuda.empty_cache()
test_preds = dev_color_mod.predict(dev_cols_test, dev_seqs_test)
#dev_color_mod.predict(dev_cols_test, dev_seqs_test, probabilities=True)
train_preds = dev_color_mod.predict(dev_cols_train, dev_seqs_train)
#dev_color_mod.predict(dev_cols_test, dev_seqs_test, probabilities=True)
correct = sum([1 if x == 2 else 0 for x in test_preds])
print("test", correct, "/", len(test_preds), correct/len(test_preds))
correct = sum([1 if x == 2 else 0 for x in train_preds])
print("train", correct, "/", len(train_preds), correct/len(train_preds))
dev_color_mod.save_model("literal_listener_with_attention.pt")
def save_to_pickle():
    import pickle 

    with open('dev_vocab.pickle', 'wb') as handle:
        pickle.dump(dev_vocab, handle, protocol=pickle.HIGHEST_PROTOCOL)
    with open('dev_seqs_test.pickle', 'wb') as handle:
        pickle.dump(dev_seqs_test, handle, protocol=pickle.HIGHEST_PROTOCOL)
    with open('dev_seqs_train.pickle', 'wb') as handle:
        pickle.dump(dev_seqs_train, handle, protocol=pickle.HIGHEST_PROTOCOL)
    with open('dev_cols_test.pickle', 'wb') as handle:
        pickle.dump(dev_cols_test, handle, protocol=pickle.HIGHEST_PROTOCOL)
    with open('dev_cols_train.pickle', 'wb') as handle:
        pickle.dump(dev_cols_train, handle, protocol=pickle.HIGHEST_PROTOCOL)
    with open('embedding.pickle', 'wb') as handle:
        pickle.dump(embedding, handle, protocol=pickle.HIGHEST_PROTOCOL)
#save_to_pickle()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1022744855330342, 'fraction_numerical': 0.01686523286399505, 'mean_word_length': 5.670794633642931, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '84345', 'license': 'Apache-2.0', 'n_tokens_mistral': 2268, 'n_tokens_neox': 2168, 'n_words': 528, 'path': 'listeral_listener_with_attention.ipynb', 'repo_name': 'Christopher-Leung/cs224u', 'size': '6474'}