<commit_before>---
layout:     post
title:      "Alternative Intelligence"
date:       2017-05-31 14:57:00
summary:    "Donald Trump simulator using RNNs"
categories: things stuff
---

A technical overview of my Donald Trump Twitter simulator, which can be found at [https://kevin-albert.github.io/bingbingbongbongbingbingbing](https://kevin-albert.github.io/bingbingbongbongbingbingbing).  

## Introduction

An ongoing topic of attention in news and politics, <NAME>'s twitter account is also an ever-growing source of data for natural language analysis. By applying a widely-used character-level deep RNN model ([Generating Sequences With Recurrent Neural Networks. Graves, 2014](https://arxiv.org/pdf/1308.0850.pdf), [The Unreasonable Effectiveness of Recurrent Neural Networks. Karpathy, 2015](https://karpathy.github.io/2015/05/21/rnn-effectiveness/)), we are able to build a simple and strikingly realistic Donald Trump simulation.

## Training Data

The training data is simply the corpus of <NAME>'s tweets from 2009 to 2017, with topics ranging from his book "The Art of The Deal", his love of <NAME> during 2012 election, his enemies <NAME> and <NAME>, the height of Trump Tower, and of course the power his Twitter account &ndash; adding up to approximately 3.3MB of raw text which was generously provided by the [Trump Twitter Archive](http://trumptwitterarchive.com/). The data is all concatenated into a single file, with each tweet preceded by the symbol `^` (which the network learns to interpret as the start of a new thought). During training, the entire file is scanned and the internal state of the recurrent layers is reset approximately every 10000 characters.

## Architecture:

* **Layers**: 166 (Input) -> 256 (LSTM) -> 256 (LSTM) -> 256 (LSTM) -> 166 (Output)
* **Optimization**: Nesterov Accelerated Momentum
* **BPTT Sequence Length**: 100
* **Learning Rate**: 0.0005
* **Learning Rate Decay Factor**: 0.975
* **Regularization**: Add gaussian noise to output with standard deviation 0.01

## Training
<commit_msg>Update links<commit_after>---
layout:     post
title:      "Alternative Intelligence"
date:       2017-05-31 14:57:00
summary:    "Donald Trump simulator using RNNs"
categories: things stuff
---

A technical overview of my Donald Trump Twitter simulator, which can be found at [https://kevin-albert.github.io/bingbingbongbongbingbingbing](https://kevin-albert.github.io/bingbingbongbongbingbingbing).  

## Introduction

An ongoing topic of attention in news and politics, <NAME>'s twitter account is also an ever-growing source of data for natural language analysis. By applying a widely-used character-level deep RNN model ([Graves, 2014](https://arxiv.org/pdf/1308.0850.pdf), [Karpathy, 2015](https://karpathy.github.io/2015/05/21/rnn-effectiveness/)), we are able to build a simple and strikingly realistic Donald Trump simulation.

## Training Data

The training data is simply the corpus of <NAME>'s tweets from 2009 to 2017, with topics ranging from his book "The Art of The Deal", his love of <NAME> during 2012 election, his enemies <NAME> and <NAME>, the height of Trump Tower, and of course the power his Twitter account &ndash; adding up to approximately 3.3MB of raw text which was generously provided by the [Trump Twitter Archive](http://trumptwitterarchive.com/). The data is all concatenated into a single file, with each tweet preceded by the symbol `^` (which the network learns to interpret as the start of a new thought). During training, the entire file is scanned and the internal state of the recurrent layers is reset approximately every 10000 characters.

## Architecture:

* **Layers**: 166 (Input) -> 256 (LSTM) -> 256 (LSTM) -> 256 (LSTM) -> 166 (Output)
* **Optimization**: Nesterov Accelerated Momentum
* **BPTT Sequence Length**: 100
* **Learning Rate**: 0.0005
* **Learning Rate Decay Factor**: 0.975
* **Regularization**: Add gaussian noise to output with standard deviation 0.01

## Training

{'fraction_non_alphanumeric': 0.09523809523809523, 'fraction_numerical': 0.04383975812547241, 'mean_word_length': 4.647226173541963, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 21, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1196608', 'n_tokens_mistral': 1255, 'n_tokens_neox': 1039, 'n_words': 496}