<jupyter_start><jupyter_text>Salary prediction, episode II: make it actually work (4 points)Your main task is to use some of the tricks you've learned on the network and analyze if you can improve __validation MAE__. Try __at least 3 options__ from the list below for a passing grade. Write a short report about what you have tried. More ideas = more bonus points. __Please be serious:__ " plot learning curves in MAE/epoch, compare models based on optimal performance, test one change at a time. You know the drill :)You can use either __pytorch__ or __tensorflow__ or any other framework (e.g. pure __keras__). Feel free to adapt the seminar code for your needs. For tensorflow version, consider `seminar_tf2.ipynb` as a starting point.<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>About the challengeFor starters, let's download and unpack the data from [here](https://www.dropbox.com/s/5msc5ix7ndyba10/Train_rev1.csv.tar.gz?dl=0). You can also get it from [yadisk url](https://yadi.sk/d/vVEOWPFY3NruT7) the competition [page](https://www.kaggle.com/c/job-salary-prediction/data) (pick `Train_rev1.*`).<jupyter_code>!curl -L https://www.dropbox.com/s/5msc5ix7ndyba10/Train_rev1.csv.tar.gz?dl=1 -o Train_rev1.csv.tar.gz
!tar -xvzf ./Train_rev1.csv.tar.gz
data = pd.read_csv("./Train_rev1.csv", index_col=None)
data.shape<jupyter_output><empty_output><jupyter_text>One problem with salary prediction is that it's oddly distributed: there are many people who are paid standard salaries and a few that get tons o money. The distribution is fat-tailed on the right side, which is inconvenient for MSE minimization.There are several techniques to combat this: using a different loss function, predicting log-target instead of raw target or even replacing targets with their percentiles among all salaries in the training set. We gonna use logarithm for now._You can read more [in the official description](https://www.kaggle.com/c/job-salary-predictiondescription)._<jupyter_code>data['Log1pSalary'] = np.log1p(data['SalaryNormalized']).astype('float32')

plt.figure(figsize=[8, 4])
plt.subplot(1, 2, 1)
plt.hist(data["SalaryNormalized"], bins=20);
plt.title("NormSalary")

plt.subplot(1, 2, 2)
plt.hist(data['Log1pSalary'], bins=20);
plt.title("LogSalary")<jupyter_output><empty_output><jupyter_text>Our task is to predict one number, __Log1pSalary__.To do so, our model can access a number of features:* Free text: __`Title`__ and  __`FullDescription`__* Categorical: __`Category`__, __`Company`__, __`LocationNormalized`__, __`ContractType`__, and __`ContractTime`__.<jupyter_code>text_columns = ["Title", "FullDescription"]
categorical_columns = ["Category", "Company", "LocationNormalized", "ContractType", "ContractTime"]
target_column = "Log1pSalary"

data[categorical_columns] = data[categorical_columns].fillna('NaN') # cast missing values to string "NaN"

data.sample(3)<jupyter_output><empty_output><jupyter_text>Preprocessing text dataJust like last week, applying NLP to a problem begins from tokenization: splitting raw text into sequences of tokens (words, punctuation, etc).__Your task__ is to lowercase and tokenize all texts under `Title` and `FullDescription` columns. Store the tokenized data as a __space-separated__ string of tokens for performance reasons.It's okay to use nltk tokenizers. Assertions were designed for WordPunctTokenizer, slight deviations are okay.<jupyter_code>print("Raw text:")
print(data["FullDescription"][2::100000])
import nltk
tokenizer = nltk.tokenize.WordPunctTokenizer()

# see task above
for column in text_columns:
  data[column] = data[column].apply(lambda x: " ".join(tokenizer.tokenize(str(x).lower())))<jupyter_output><empty_output><jupyter_text>Now we can assume that our text is a space-separated list of tokens:<jupyter_code>print("Tokenized:")
print(data["FullDescription"][2::100000])
assert data["FullDescription"][2][:50] == 'mathematical modeller / simulation analyst / opera'
assert data["Title"][54321] == 'international digital account manager ( german )'<jupyter_output><empty_output><jupyter_text>Not all words are equally useful. Some of them are typos or rare words that are only present a few times. Let's count how many times is each word present in the data so that we can build a "white list" of known words.<jupyter_code># Count how many times does each token occur in both "Title" and "FullDescription" in total
# build a dictionary { token -> it's count }
from collections import Counter

token_counts = Counter()

for column in text_columns:
  for line in data[column].values:
    token_counts.update(line.split(" "))

# hint: you may or may not want to use collections.Counter
print("Total unique tokens :", len(token_counts))
print('\n'.join(map(str, token_counts.most_common(n=5))))
print('...')
print('\n'.join(map(str, token_counts.most_common()[-3:])))

assert token_counts.most_common(1)[0][1] in  range(2600000, 2700000)
assert len(token_counts) in range(200000, 210000)
print('Correct!')
# Let's see how many words are there for each count
plt.hist(list(token_counts.values()), range=[0, 10**4], bins=50, log=True)
plt.xlabel("Word counts");<jupyter_output><empty_output><jupyter_text>Now filter tokens a list of all tokens that occur at least 10 times.<jupyter_code>min_count = 10

# tokens from token_counts keys that had at least min_count occurrences throughout the dataset
tokens = sorted(t for t, c in token_counts.items() if c >= min_count)
# Add a special tokens for unknown and empty words
UNK, PAD = "UNK", "PAD"
tokens = [UNK, PAD] + sorted(tokens)
print("Vocabulary size:", len(tokens))

assert type(tokens) == list
assert len(tokens) in range(32000, 35000)
assert 'me' in tokens
assert UNK in tokens
print("Correct!")<jupyter_output><empty_output><jupyter_text>Build an inverse token index: a dictionary from token(string) to it's index in `tokens` (int)<jupyter_code>token_to_id = {t: i for i,t in enumerate(tokens)}
assert isinstance(token_to_id, dict)
assert len(token_to_id) == len(tokens)
for tok in tokens:
    assert tokens[token_to_id[tok]] == tok

print("Correct!")<jupyter_output><empty_output><jupyter_text>And finally, let's use the vocabulary you've built to map text lines into neural network-digestible matrices.<jupyter_code>UNK_IX, PAD_IX = map(token_to_id.get, [UNK, PAD])

def as_matrix(sequences, max_len=None):
    """ Convert a list of tokens into a matrix with padding """
    if isinstance(sequences[0], str):
        sequences = list(map(str.split, sequences))
        
    max_len = min(max(map(len, sequences)), max_len or float('inf'))
    
    matrix = np.full((len(sequences), max_len), np.int32(PAD_IX))
    for i,seq in enumerate(sequences):
        row_ix = [token_to_id.get(word, UNK_IX) for word in seq[:max_len]]
        matrix[i, :len(row_ix)] = row_ix
    
    return matrix
print("Lines:")
print('\n'.join(data["Title"][::100000].values), end='\n\n')
print("Matrix:")
print(as_matrix(data["Title"][::100000]))<jupyter_output><empty_output><jupyter_text>Now let's  encode the categirical data we have.As usual, we shall use one-hot encoding for simplicity. Kudos if you implement more advanced encodings: tf-idf, pseudo-time-series, etc.<jupyter_code>from sklearn.feature_extraction import DictVectorizer

# we only consider top-1k most frequent companies to minimize memory usage
top_companies, top_counts = zip(*Counter(data['Company']).most_common(1000))
recognized_companies = set(top_companies)
data["Company"] = data["Company"].apply(lambda comp: comp if comp in recognized_companies else "Other")

categorical_vectorizer = DictVectorizer(dtype=np.float32, sparse=False)
categorical_vectorizer.fit(data[categorical_columns].apply(dict, axis=1))<jupyter_output><empty_output><jupyter_text>The deep learning partOnce we've learned to tokenize the data, let's design a machine learning experiment.As before, we won't focus too much on validation, opting for a simple train-test split.__To be completely rigorous,__ we've comitted a small crime here: we used the whole data for tokenization and vocabulary building. A more strict way would be to do that part on training set only. You may want to do that and measure the magnitude of changes.<jupyter_code>MODEL_VERSION = "V6"
EMBEDDING_SIZE = 100
TITLE_MAX_LEN = 10
DESCRIPTION_MAX_LEN = 10
from sklearn.model_selection import train_test_split

data_train, data_val = train_test_split(data, test_size=0.2, random_state=42)
data_train.index = range(len(data_train))
data_val.index = range(len(data_val))

print("Train size = ", len(data_train))
print("Validation size = ", len(data_val))
def make_batch(data, max_len_title=None, max_len_description=None, word_dropout=0):
    """
    Creates a keras-friendly dict from the batch data.
    :param word_dropout: replaces token index with UNK_IX with this probability
    :returns: a dict with {'title' : int64[batch, title_max_len]
    """
    batch = {}
    batch["Title"] = as_matrix(data["Title"].values, max_len_title)
    batch["FullDescription"] = as_matrix(data["FullDescription"].values, max_len_description)
    batch['Categorical'] = categorical_vectorizer.transform(data[categorical_columns].apply(dict, axis=1))
    
    if word_dropout != 0:
        batch["FullDescription"] = apply_word_dropout(batch["FullDescription"], 1. - word_dropout)
    
    if target_column in data.columns:
        batch[target_column] = data[target_column].values
    
    return batch

def apply_word_dropout(matrix, keep_prop, replace_with=UNK_IX, pad_ix=PAD_IX,):
    dropout_mask = np.random.choice(2, np.shape(matrix), p=[keep_prop, 1 - keep_prop])
    dropout_mask &= matrix != pad_ix
    return np.choose(dropout_mask, [matrix, np.full_like(matrix, replace_with)])
make_batch(data_train[:3], max_len_title=TITLE_MAX_LEN, max_len_description=DESCRIPTION_MAX_LEN)<jupyter_output><empty_output><jupyter_text>Get pre-trained word embeddings<jupyter_code>!wget http://nlp.stanford.edu/data/glove.6B.zip
!unzip -q glove.6B.zip
import os

path_to_glove_file = f"/content/glove.6B.{EMBEDDING_DIM}d.txt"

embeddings_index = {}
with open(path_to_glove_file) as f:
    for line in f:
        word, coefs = line.split(maxsplit=1)
        coefs = np.fromstring(coefs, "f", sep=" ")
        embeddings_index[word] = coefs

print("Found %s word vectors." % len(embeddings_index))
num_tokens = len(tokens) + 2
hits = 0
misses = 0

# Prepare embedding matrix
embedding_matrix = np.zeros((num_tokens, EMBEDDING_DIM))
for word, i in token_to_id.items():
    embedding_vector = embeddings_index.get(word)
    if embedding_vector is not None:
        # Words not found in embedding index will be all-zeros.
        # This includes the representation for "padding" and "OOV"
        embedding_matrix[i] = embedding_vector
        hits += 1
    else:
        misses += 1
print("Converted %d words (%d misses)" % (hits, misses))<jupyter_output><empty_output><jupyter_text>ArchitectureOur basic model consists of three branches:* Title encoder* Description encoder* Categorical features encoderWe will then feed all 3 branches into one common network that predicts salary. This clearly doesn't fit into keras' __Sequential__ interface. To build such a network, one will have to use __[Keras Functional API](https://keras.io/models/model/)__.<jupyter_code>import keras
import keras.layers as L
def build_model(n_tokens=len(tokens), n_cat_features=len(categorical_vectorizer.vocabulary_), hid_size=64):
    """ Build a model that maps three data sources to a single linear output: predicted log1p(salary) """
    
    l_title = L.Input(shape=[None], name="Title")
    l_descr = L.Input(shape=[None], name="FullDescription")
    l_categ = L.Input(shape=[n_cat_features], name="Categorical")

    emb_title = L.Embedding(output_dim=EMBEDDING_DIM, 
                            input_dim=num_tokens,
                            input_length=TITLE_MAX_LEN,
                            embeddings_initializer=keras.initializers.Constant(embedding_matrix),
                            trainable=False
                            )(l_title)
    c_title = L.Conv1D(1, (2, ), padding='same', activation='relu')(emb_title)
    p_title = L.GlobalMaxPooling1D()(c_title)

    emb_descr = L.Embedding(output_dim=EMBEDDING_DIM, 
                            input_dim=num_tokens,
                            input_length=DESCRIPTION_MAX_LEN,
                            embeddings_initializer=keras.initializers.Constant(embedding_matrix),
                            trainable=False
                            )(l_descr)
    c_descr = L.Conv1D(1, (2, ), padding='same', activation='relu')(emb_descr)
    p_descr = L.GlobalMaxPooling1D()(c_descr)

    f_categ = L.Dense(64, activation='relu')(l_categ)
    
    x = L.concatenate([p_title, p_descr, f_categ])
    x = L.Dense(64, activation='relu')(x)
    x = L.Dense(64, activation='relu')(x)

    output_layer = L.Dense(1, activation='linear')(x)
    # end of your code
    
    model = keras.models.Model(inputs=[l_title, l_descr, l_categ], outputs=[output_layer])
    model.compile('adam', 'mean_squared_error', metrics=['mean_absolute_error'])
    return model
    
model = build_model()
model.summary()

dummy_pred = model.predict(make_batch(data_train[:100]))
dummy_loss = model.train_on_batch(make_batch(data_train[:100]), data_train['Log1pSalary'][:100])[0]
assert dummy_pred.shape == (100, 1)
assert len(np.unique(dummy_pred)) > 20, "model returns suspiciously few unique outputs. Check your initialization"
assert np.ndim(dummy_loss) == 0 and 0. <= dummy_loss <= 250., "make sure you minimize MSE"<jupyter_output><empty_output><jupyter_text>Training and evaluationAs usual, we gonna feed our monster with random minibatches of data. As we train, we want to monitor not only loss function, which is computed in log-space, but also the actual error measured in dollars.<jupyter_code>def iterate_minibatches(data, batch_size=256, shuffle=True, cycle=False, **kwargs):
    """ iterates minibatches of data in random order """
    while True:
        indices = np.arange(len(data))
        if shuffle:
            indices = np.random.permutation(indices)

        for start in range(0, len(indices), batch_size):
            batch = make_batch(data.iloc[indices[start : start + batch_size]], **kwargs)
            target = batch.pop(target_column)
            yield batch, target
        
        if not cycle: break<jupyter_output><empty_output><jupyter_text>Model trainingWe can now fit our model the usual minibatch way. The interesting part is that we train on an infinite stream of minibatches, produced by `iterate_minibatches` function.<jupyter_code>#Callbacks
import datetime

# Load the TensorBoard notebook extension
%load_ext tensorboard

logdir = os.path.join("logs", datetime.datetime.now().strftime(f"{MODEL_VERSION}: %d.%m.%Y - %H:%M:%S"))
tensorboard_callback = keras.callbacks.TensorBoard(logdir, histogram_freq=1)

callbacks = [
             keras.callbacks.TensorBoard(logdir, histogram_freq=1),
             keras.callbacks.EarlyStopping(monitor="val_loss", mode="min", patience = 5),
             keras.callbacks.ModelCheckpoint(f"{MODEL_VERSION}_best_model.h5", monitor="val_loss", mode="min", save_best_only=True)
            ]
%tensorboard --logdir logs --port 6007
batch_size = 256
epochs = 100            # definitely too small
steps_per_epoch = 100  # for full pass over data: (len(data_train) - 1) // batch_size + 1

model = build_model()

history = model.fit(iterate_minibatches(data_train, batch_size, cycle=True, word_dropout=0.05), 
                    epochs=epochs, steps_per_epoch=steps_per_epoch,
                    
                    verbose = 0,

                    validation_data=iterate_minibatches(data_val, batch_size, cycle=True),
                    validation_steps=data_val.shape[0] // batch_size,

                    callbacks=callbacks
                    )
model.load_weights(f"{MODEL_VERSION}_best_model.h5")
def print_metrics(model, data, batch_size=batch_size, name="", **kw):
    squared_error = abs_error = num_samples = 0.0
    for batch_x, batch_y in iterate_minibatches(data, batch_size=batch_size, shuffle=False, **kw):
        batch_pred = model.predict(batch_x)[:, 0]
        squared_error += np.sum(np.square(batch_pred - batch_y))
        abs_error += np.sum(np.abs(batch_pred - batch_y))
        num_samples += len(batch_y)
    print("%s results:" % (name or ""))
    print("Mean square error: %.5f" % (squared_error / num_samples))
    print("Mean absolute error: %.5f" % (abs_error / num_samples))
    return squared_error, abs_error
    
print_metrics(model, data_train, name='Train')
print_metrics(model, data_val, name='Val');
def plot_metrics(history):
  fig, axs = plt.subplots(2, 2, figsize=(15,10))
  axs[0, 0].plot(history.history['loss'][1:])
  axs[0, 0].set_title('loss')
  axs[0, 1].plot(history.history['mean_absolute_error'][1:])
  axs[0, 1].set_title('mean_absolute_error')
  axs[1, 0].plot(history.history['val_loss'][1:])
  axs[1, 0].set_title('val_loss')
  axs[1, 1].plot(history.history['val_mean_absolute_error'][1:])
  axs[1, 1].set_title('val_mean_absolute_error')

plot_metrics(history)<jupyter_output><empty_output><jupyter_text>More data analitics<jupyter_code>plt.figure(figsize=[8, 4])
for i, column in enumerate(text_columns):
  sentence_length = [ len(sentence.split(" ")) for sentence in data_train[column] ]
  plt.subplot(1, 2, i+1)
  plt.hist(sentence_length, bins=20)
  plt.title(column)
  print(f"{column}: ",
        f"average = {np.average(sentence_length)} ",
        f"median = {np.median(sentence_length)} ",
        f"90% quant = {np.percentile(sentence_length, 90)} ",
        )<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11588248536695182, 'fraction_numerical': 0.015814948221521838, 'mean_word_length': 4.728239845261122, 'pattern_counts': {'":': 0, '<': 71, '<?xml version=': 0, '>': 72, 'https://': 6, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '518588', 'license': 'MIT', 'n_tokens_mistral': 5982, 'n_tokens_neox': 5674, 'n_words': 1505, 'path': 'week02_classification/homework_part2.ipynb', 'repo_name': 'Dimfeel/nlp_course', 'size': '17768'}