<jupyter_start><jupyter_text>Inspecting what the LSTM Tweet generator has learned We trained the LSTM model (A 256 unit LSTM straight into a softmax output layer) on $2^{18}$ (262,144) Tweets. That might not seem like many, but remember each tweet needs 32 training examples (with a windowsize of 64 and a max length of 160 characters) so that's about 8.3 million training examples. Running on an AWS p3.2xlarge instance (on a Tesla V100 GPU) it took about 20 minutes per epoch.The training took part in two stages (not planned, this was just how it turned out) - first it was trained only on the first half of that data for about 30 epochs. After it looked like the model was starting to over-fit (but without looking like it had got the hang of generating tweets) so I doubled the size of the training set, and ran it for another 10-12 epochs or so.I say 'looking like' - throughout the training process, I had the model generate a few tweets based on some seed text (randomly pulled in from a tweet) at various settings of the temperature parameter, and eyeballed the resulting gibberish to get a feel for how far through training the model was.I later realised that this wasn't a particularly good check on the model's ability to generate coherent English language tweets - it seems as though the model is particularly sensitive to what is in the seed text and how much of it it gets. So generating based on 10 or 20 seed characters gets very different results to generating based on 64 seed characters. In an ideal world, we do want the model to be able to freely generate coherent tweets based on a minimal seed (ideally, just an emoji). However, in this notebook we'll take a more detailed look at what's going on.Let's start by loading up the Twitter dataset that we trained the model on, and the global variables that we used to set the size of the sets.<jupyter_code>import numpy as np
import pandas as pd
import data_load_utils as util
from math import ceil
tweets = util.filter_tweets_min_count(util.read_tweet_data('data/emojis_homemade.csv'), min_count=1000)
tweets['text'] = util.filter_text_for_handles(tweets['text'])

MAX_TWEET_LENGTH = 160
WINDOW_SIZE = 64
STEP = 3

samples_per_tweet = int(ceil((MAX_TWEET_LENGTH - WINDOW_SIZE) / STEP)) # 32
tweets_per_batch = 64
samples_per_batch = samples_per_tweet * tweets_per_batch # 2048

chars_univ, chars_univ_idx = util.get_universal_chars_list()
# Here we're going to look at examples from the Dev set but we might want to run analyses on the Train set 

TRAIN_SIZE = 2**18 # 262,144 training examples
DEV_SIZE = 2**12   # 4096 Dev examples

n_train_batches = TRAIN_SIZE / tweets_per_batch
n_dev_batches = DEV_SIZE / tweets_per_batch

tweets_train = tweets.iloc[0:TRAIN_SIZE] 
tweets_dev = tweets.iloc[TRAIN_SIZE:TRAIN_SIZE+DEV_SIZE] 
import keras
from keras import layers
from keras.models import Sequential
from keras import callbacks
model = keras.models.load_model("models/tweet_gen_model-train250k-0.830.hdf5") # 256 LSTM units, ~30 epochs training<jupyter_output><empty_output><jupyter_text>Examples of generating tweets<jupyter_code>def sample (preds, temperature = 1.0):
    preds = np.asarray(preds).astype('float64')
    preds = np.log(preds) / temperature
    exp_preds = np.exp(preds)
    preds = exp_preds / np.sum(exp_preds)
    probas = np.random.multinomial(1, preds, 1)
    return np.argmax(probas)
import tensorflow as tf
import random
import sys
def generate_tweet(seed_text, window_size=WINDOW_SIZE,
		   length=MAX_TWEET_LENGTH, temperatures=[0.3, 0.5, 0.8, 1.0],
		   random_seed = None):

    if random_seed:
        np.random.seed(random_seed) # np.random.seed needed by tf 
        tf.set_random_seed(random_seed)
        
    n_seed_chars = window_size
    #first, pad out the seed_text with whitespace     
    seed_text = util.pad_text(seed_text, window_size)

    print ('--- Generating with seed: "' + seed_text + '" [' + str(len(seed_text)) + ']')

    # try a range of sampling temperatures
    for temperature in temperatures:
        generated_text = seed_text
        print ('--------- temperature:', temperature)
        sys.stdout.write(generated_text)

        for i in range (length - n_seed_chars):
            # one-hot encode the characters generated so far
            sampled = np.zeros((1, WINDOW_SIZE, len(chars_univ)))
            for t, char in enumerate (generated_text):
                sampled[0, t, chars_univ_idx[char]] = 1

            # sample the next character
            preds = model.predict(sampled, verbose=0)[0]
            next_index = sample(preds, temperature)
            next_char = chars_univ[next_index]

            generated_text += next_char
            generated_text = generated_text[1:]

            sys.stdout.write(next_char)

        print ("\n")    

def get_seed_text(tweets, seed_length=64, random_seed = None):
    if random_seed: np.random.seed(random_seed)
        
    seed_tweet = tweets.iloc[np.random.randint(0, len(tweets))]
    seed_text = seed_tweet['text'][0:seed_length]

    print (seed_text + "\n" + seed_tweet['text'])
    
    return seed_text
seed_text = get_seed_text(tweets, seed_length=0)
generate_tweet("", temperatures=[0.4, 0.8, 1.0, 1.2, 1.4])<jupyter_output>--- Generating with seed: "                                                                " [64]
--------- temperature: 0.4<jupyter_text>edit: gibberish when seed_text < 64 chars now fixed. It's still a bit stream-of-consciousness but genuinely I think it's because the model doesn't have much to work from here.<jupyter_code>seed_text = get_seed_text(tweets, random_seed=2)
generate_tweet(seed_text, temperatures=[0.4, 0.6, 0.8, 1.0, 1.2, 1.5], random_seed = 2)
seed_text = get_seed_text(tweets, random_seed=3)
generate_tweet(seed_text, temperatures=[0.4, 0.6, 0.8, 1.0, 1.2, 1.5], random_seed = 3)<jupyter_output>Will stick them on the shopping cart and figure out how to use t
Will stick them on the shopping cart and figure out how to use them when I need to 
--- Generating with seed: "Will stick them on the shopping cart and figure out how to use t" [64]
--------- temperature: 0.4
Will stick them on the shopping cart and figure out how to use the same pe<jupyter_text>It's interesting that the model's figured out how to use hashtags - the format ('' character followed by CamelNotationCaps). Firstly, how they syntactically fit into a tweet, either closing it out a tweet or following a sentence, as punctuation (e.g., Sathwye JifksyjK0 etc) or as a WordReplacement (e.g., Shivagsuntar etc). It's also figured out how URLs work in tweets.It's perhaps unsurprising that hashtags were one of the first things the model got the hang of during training (note to self - flesh this point out with an example of the model earlier during training). They're used consistently across multiple different tweets, often more regularly than meaningful english language words, and given how the data was collected (basically, whenever I remembered to leave the tweet harvesting script running) there's likely to be bands of uniformity across the data. (note to self - could plot a graphic of hashtag use over time throughout the dataset)<jupyter_code>seed_text = get_seed_text(tweets, random_seed=4)
generate_tweet(seed_text, temperatures=[0.4, 0.6, 0.8, 1.0, 1.2, 1.5], random_seed = 4)
seed_text = get_seed_text(tweets, random_seed=5)
generate_tweet(seed_text, temperatures=[0.4, 0.6, 0.8, 1.0, 1.2, 1.5], random_seed = 5)
seed_text = get_seed_text(tweets, random_seed=6)
generate_tweet(seed_text, temperatures=[0.4, 0.6, 0.8, 1.0, 1.2, 1.5], random_seed = 6)
seed_text = get_seed_text(tweets, random_seed=7)
generate_tweet(seed_text, length=300, temperatures=[0.4, 0.6, 0.8, 1.0, 1.2, 1.5], random_seed = 7)<jupyter_output>RT Here's my plan:1 - Vote by mail for &amp; 2 - Volunteer to GO
RT Here's my plan:1 - Vote by mail for &amp; 2 - Volunteer to GOTV betwee
--- Generating with seed: "RT Here's my plan:1 - Vote by mail for &amp; 2 - Volunteer to GO" [64]
--------- temperature: 0.4
RT Here's my plan:1 - Vote by mail for &amp; 2 - Volunteer to GOT to #S<jupyter_text>I like `Volunteer to GOD is way`. But the model doesn't seem to have got that the theme of the seed was political (`Vote by mail for &amp`) and has come up with a load of unrelated hashtags (a spot of Googling reveals that `LightinSunta'` was invented by the model, although [`MTVEMABiggestFansBTS` exists, it's a K-pop thing](https://twitter.com/hashtag/mtvemabiggestfansbts). Sarkar is an [Indian Tamil-language action drama film](https://en.wikipedia.org/wiki/Sarkar_(2018_film).<jupyter_code>seed_text = get_seed_text(tweets, random_seed=8)
generate_tweet(seed_text, temperatures=[0.4, 0.6, 0.8, 1.0, 1.2, 1.5], random_seed = 8)
seed_text = get_seed_text(tweets, random_seed=9)
generate_tweet(seed_text, length=300, temperatures=[0.4, 0.6, 0.8, 1.0, 1.2, 1.5], random_seed = 9)<jupyter_output>RT Life comes at you fast. One day you own a machine gun...the n
RT Life comes at you fast. One day you own a machine gun...the next you're on the floor, handcuffed, with your face betw
--- Generating with seed: "RT Life comes at you fast. One day you own a machine gun...the n" [64]
--------- temperature: 0.4
RT Life comes at you fast. One day you own a machine gun...the new<jupyter_text>The model really likes K-Pop `EXO EXO EXO_TEMP EXO_DontMessUpMyTempo`<jupyter_code>seed_text = get_seed_text(tweets, random_seed=10)
generate_tweet(seed_text, temperatures=[0.4, 0.6, 0.8, 1.0, 1.2, 1.5], random_seed = 10)
seed_text = get_seed_text(tweets, random_seed=11)
generate_tweet(seed_text, temperatures=[0.4, 0.6, 0.8, 1.0, 1.2, 1.5], random_seed = 11)
seed_text = get_seed_text(tweets, random_seed=12)
generate_tweet(seed_text, temperatures=[0.4, 0.6, 0.8, 1.0, 1.2, 1.5], random_seed = 12)
seed_text = get_seed_text(tweets, random_seed=13)
generate_tweet(seed_text, temperatures=[0.4, 0.6, 0.8, 1.0, 1.2, 1.5], random_seed = 13)
seed_text = get_seed_text(tweets, random_seed=24)
generate_tweet(seed_text, temperatures=[0.4, 0.6, 0.8, 1.0, 1.2, 1.5], random_seed = 24)
seed_text = get_seed_text(tweets, random_seed=28)
generate_tweet(seed_text, temperatures=[0.4, 0.6, 0.8, 1.0, 1.2, 1.5], random_seed = 28)
seed_text = get_seed_text(tweets, random_seed=30)
generate_tweet(seed_text, temperatures=[0.4, 0.6, 0.8, 1.0, 1.2, 1.5], random_seed = 30)
seed_text = "Someone took the dumbbells I left outside the house, #Strong #Go"
print (len(seed_text))
generate_tweet(seed_text, temperatures=[0.4, 0.6, 0.8, 1.0, 1.2, 1.5], random_seed = 1)<jupyter_output>64
--- Generating with seed: "Someone took the dumbbells I left outside the house, #Strong #Go"
--------- temperature: 0.4
Someone took the dumbbells I left outside the house, #Strong #Go #Authand #Good #Fillow #TheFirst # # #  # #   #Jimin #MTVEMABiggestFansBTS #BTS #MTVEMABiggestF

--------- temperature: 0.6
Someone took the dumbbells I left outside the house, #Strong #Go #DGFTinteress #Chue #Castary # # # # #                                                    :: yo

--------- temperature: 0.8
Someone took the dumbbells I left outside the house, #Strong #God #Nexx #valuman # # # #           @MyPJ 11923 3 3 2Parandom - 1008-187  Spinniestheh https://t.

--------- temperature: 1.0
Someone took the dumbbells I left outside the house, #Strong #GoDasket #Fansanay Watco - I watch 10.2bCos out if your skate with sister looking amazing  honest 

--------- temperature: 1.2
Someone took the dumbbells I left outside the house, #Strong #Go Day?Itely Vija FHY HE LOOK AND MEAS HR DONIXIL MION BABA[...]
{'fraction_non_alphanumeric': 0.1008308810801454, 'fraction_numerical': 0.03695689804396746, 'mean_word_length': 3.948608137044968, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 20, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '423427', 'license': 'MIT', 'n_tokens_mistral': 4112, 'n_tokens_neox': 3746, 'n_words': 1366, 'path': 'Inspecting the LSTM Tweet Generator.ipynb', 'repo_name': 'nicksexton/emojitranslate', 'size': '11554'}