<jupyter_start><jupyter_text>**7 December 2020** Neural networks Admin* Spring TA applications close **today**    * https://cis-student-hiring.coecis.cornell.edu/    * Applications are brief and the experience is rewarding ... and you get paid* Problem set 11 (the last one!)    * Due Tuesday, 12/8, at 11:59pm* Read Antoniak et al. for Wednesday    * Antoniak et al. use social media data to study birth narratives. Maria will lead a discussion of their research and of the ethical and practical issues involved.        * See also the optional reading, on the ethics of using data from fan communities.    * Fourth and final reading response due by the evening of Tuesday, 12/15    * If responding in the final week, task will be to reflect on the range of methods and problems covered in the Wednesday readings over the course of the semester. This is somewhat longer and more difficult than a standard reading response.* Friday section will be open project consultation and group work.    * Attendance optional    * Contemplating trying gather.town. Details to come via Campuswire.  Neural networks and deep learning* We've used NLP tools at many points this semester, but this isn't an NLP class* That said, neural methods have transformed many areas of NLP over the last decade    * And deep learning -- a subset of neural methods -- has been very widely applied in machine learning and AI* Our tasks today: define "neural network," relate neural nets to other learning systems, take a look at how a neural network works, and show how to implement a very simple neural classifier in Python What is a neural network?* A neural network is a computing system comprising artificial neurons* Neurons were originally (1940s) intended to model organic brain behavior    * But now, the name is really just a bit of jargon. No one thinks its important whether or not computational neurons have anything to do with biological neurons.* Individual neurons are mathematical functions that take a vector of input values and produce a single output value.    * We've seen lots of these kinds of functions over the semester, not all of them related to actually existing neural networks    * What matters are the details of the functions and the ways they relate to one another in a network* In a neural network, the neurons are connected to one another in one or more layers, so that the output of one neuron is the input of another (or many others)     Logistic regression* Logistic regression **is not a neural network** in the modern sense, but it captures much of the spirit of a basic neural network and a lot of the math is related, so let's revisit it* Fit training data to a linear model: $z = W_0 + W_1 x_1 + W_2 x_2 + ...$    * Values of $x$ are observed properties of an object (counts of individual words, say)    * The $W$s are weights. We multiply the weight associated with each word (for example) by the number of times that word occurs in a document.        * These types of element-wise multiplications between two vectors are called **dot products**    * Add up the weight * count products and we produce an output value, $z$    * Note that values of $z$ can range from -infinity to +infinity* Transform the linear value into a score between 0 and 1 using the sigmoid function: $$\sigma(t) = \frac{1}{1 + e^{-z}}$$* Sigmoid function looks like this:    * When we train a logistic regression classifier, we're trying to learn the set of weights that produce the most accurate classifications* We learn the weights by:    * Initializing to random values (or equal values, or some arrangement that reflects our best guess about the correct weights)    * Calculating **cross-entropy loss**, that is, how far away are our predicted outputs from the known-true (gold) values.        * Our goal is to minimize this loss function by adjusting the weights in our model        * See Jurafsky and Martin, ch. 5, for the math, but the short version is that we take (roughly) the negative log of the sum of the differences between the predicted labels (as probabilities ranging from 0 to 1) and the true labels (which are either 0 or 1)        * Trivia point: logistic regression is a more advanced version of the **perceptron** (which uses a binary loss function rather than a probabilistic one). The perceptron was invented at Cornell (by <NAME> in 1958).    * Adjusting our weights using **gradient descent**        * Again, the math isn't important to us, but ... we find the gradient (slope) of the loss function by partial differentiation with respect to each of the weights. In short, we find how the loss function chages in response to small changes in each weight, then move the weight in the direction that minimizes the loss. Repeat until the loss function stops changing (much) and hope we've found the global minimum (that is, the globally best weights).* If you've been around neural networks and machine learning, these terms will sound familiar: loss function, gradient descent. Now you know what they mean. From logistic regression to feed-forward networks* The problem with logistic regression (which is a great classifier for many problems!) is that it can only learn linear relationships between inputs and outputs. If our problem is nonlinear, logistic regression might not work well on it.* The simplest way to understand the relationship between logistic regression and a basic neural network is that a neural network is made up of multiple logistic-like functions, each of which can learn a different part of the correct solution (where "solution" = function that best fits the training data)* Here's a schematic representation (from Jurafsky and Martin) of a feed-forward network with a single hidden layer (the middle one, with labels $h_i$):* There are three layers here: input, hidden, and output.    * The input layer is the data you feed into the system.    * The hidden layer is where the weights are adjusted to maximize classification accuracy. This is what *learns*.    * The output layer translates numerical values calculated in the hidden layer into class probabilities (that is, into specific classification decisions).* The math in this case is the same as in the logistic case, except that:    * We have matrices of weights across the neurons, rather than a single vector of weights for a single neuron    * We have a vector of outputs from the hidden layer, rather a single, scalar output    * Gradient descent is harder, because there are more paths to differentiate        * This is the most consequential difference in practical terms, because it really slows down training        * The standard approach is **backpropagation**. For details, See Jurafsky and Martin, ch. 7. It's like partial differentiation, but performed piece-wise backward through the all the possible paths from outputs to inputs via the hidden layer(s).          From shallow to deep* Even a neural network with a single hidden layer (of possibly infinite width; that is, made up of arbitrarily many neurons) can be shown to be able to represent a function of arbirary complexity    * Note in passing: this is a remarkable result. It means that neural networks are immensely flexible in the relationships between inputs and outputs that they can model.    * But this fact doesn't imply that it's *easy* to learn a correct or high-performing representation of an arbitrary function in a neural network* In practice, it can be more efficient to build networks that are narrower but *deeper*; that have more layers* Deep learning also largely removes the need for (ceratin kinds of) feature engineering, since the layers learn maximally effective transformations of the data    * But the right kinds of data still need to be present in the first place!    * If you only give your network word counts, it won't magically engineer paratextual features.* You may have heard of **convolutional** neural networks and **recurrent** neural networks. These are networks in which there is not a strict one-to-one connection between all the neurons in each layer.    * Convolutional networks are widely used in image recognition    * Recurrent networks (in which parts of layers are connected both forward and backward) are often used in NLP applications* All of this is **bloody slow** and involves a lot of matrix math. Two main factors have driven the deep learning revolution over the last decade:    * Web-scale data, which provides enough instances to learn fine distinctions in complex decision boundaries        * A method that can model arbitrarily complex functions isn't much good if you don't have enough data to explore the function space    * GPUs (graphics cards), which are essentially super-fast matrix calculators        * These make computing with all that data tractable (more or less)  Basic neural network classification in `sklearn`The only neural classifier built into `sklearn` is the multi-layer perceptron (which isn't really a perceptron at all, but the name stuck). We'll demonstrate it here; it's easy and works as a drop-in replacement for any other classifier.For more adavnced work with neural networks, you'd want to explore frameworks like [Keras](https://keras.io/) and [PyTorch](https://pytorch.org/), which are more flexible and support computations on GPUs.<jupyter_code># load embedding representation of reviews data
import numpy as np
import os
import pickle
from   sklearn.linear_model import LogisticRegression
from   sklearn.model_selection import cross_val_score
from   sklearn.neural_network import MLPClassifier
from   sklearn.preprocessing import StandardScaler

with open(os.path.join('supplements', 'X_embed.pickle'), 'rb') as f:
    X = pickle.load(f) # embedding-based
with open(os.path.join('supplements', 'X_tfidf.pickle'), 'rb') as f:
    X_tfidf = pickle.load(f) # token-based
with open(os.path.join('supplements', 'y.pickle'), 'rb') as f:
    y = pickle.load(f) # labels
# scale
X = StandardScaler().fit_transform(X)
print(X.shape)
%%time
# logit score
logit_scores = cross_val_score(LogisticRegression(max_iter=1000), X, y, cv=5)
print("Logit accuracy:", np.mean(logit_scores))
%%time
# MLP score, no optimization
mlpc = MLPClassifier()
mlp_scores = cross_val_score(mlpc, X, y, cv=5, n_jobs=-1)
print("MLP accuracy:", np.mean(mlp_scores))<jupyter_output>MLP accuracy: 0.6618566741819093
CPU times: user 57 ms, sys: 112 ms, total: 169 ms
Wall time: 50.4 s<jupyter_text>Not great! It's slower and performs worse than logistic regression. Recall, for comparison, that our token-based logistic regression score (from the problem set) was around 0.66.Let's try some tuning (tuning neural networks is super important) ...<jupyter_code>%%time
# Grid search: wide vs. deep, and compare solvers
from sklearn.model_selection import GridSearchCV
import warnings

params = {
    'hidden_layer_sizes': [(300,), (100,), (10,), (2,), (100,10), (30,10), (10,2)],
    'solver':['adam', 'lbfgs'],
}
clf = GridSearchCV(mlpc, params, n_jobs=-1)

with warnings.catch_warnings() as w:
    warnings.simplefilter("ignore")
    clf.fit(X[:2000], y[:2000]) # Note subset of the data!
# Which parameters are best?
clf.best_params_
# What's the cv score of the best classifier?
clf.best_score_
%%time
# Score after tuning
mlp_tuned_scores = cross_val_score(
    MLPClassifier(max_iter=500, **clf.best_params_), 
    X, 
    y, 
    cv=5,
    n_jobs=-1,
    verbose=1
)
print("MLP accuracy (tuned):", np.mean(mlp_tuned_scores))
%%time
# Compare the untuned, token-based version
mlp_tfidf_scores = cross_val_score(
    mlpc, 
    StandardScaler().fit_transform(X_tfidf.toarray()), 
    y, 
    cv=5, 
    n_jobs=-1,
    verbose=1
)
print("MLP accuracy (using tokens):", np.mean(mlp_tfidf_scores))<jupyter_output>[Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.
{'fraction_non_alphanumeric': 0.06364790520524757, 'fraction_numerical': 0.010156580617858655, 'mean_word_length': 4.395433789954338, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 8, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '150017', 'license': 'MIT', 'n_tokens_mistral': 3125, 'n_tokens_neox': 2890, 'n_words': 1681, 'path': 'lectures/lec-12-07-neural-methods.ipynb', 'repo_name': 'wilkens-teaching/info3350-f20', 'size': '11825'}