<jupyter_start><jupyter_text>7. TensorFlow We are now going to go through a quick **TensorFlow** example. You may be wondering: "What is TensorFlow?" So here is a quick synopsis from the `https://www.tensorflow.org/` website:> TensorFlow™ is an open source software library for numerical computation using data flow graphs. Nodes in the graph represent mathematical operations, while the graph edges represent the multidimensional data arrays (tensors) communicated between them. The flexible architecture allows you to deploy computation to one or more CPUs or GPUs in a desktop, server, or mobile device with a single API. TensorFlow was originally developed by researchers and engineers working on the Google Brain Team within Google's Machine Intelligence research organization for the purposes of conducting machine learning and deep neural networks research, but the system is general enough to be applicable in a wide variety of other domains as well.So TensorFlow is one of the main API's that is used in the creation of neural nets, and other machine learning algorithms. It is state of the art as of 2018, so let's get started.We can begin with our imports.<jupyter_code>import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Now let's create our random training data. This is the same process that we used earlier in our training and predictions walk throughs. We have **three** gaussian clouds, 500 of each class centered at **(0,2)**, **(2,2**, and **(-2,2)**. We will make a scatter plot of the data in case your forgot what it looked like, as well as creating indicator variable (i.e. one hot encoding) for the targets.<jupyter_code># create random training data again
Nclass = 500
D = 2 # dimensionality of input
M = 3 # hidden layer size
K = 3 # number of classes

X1 = np.random.randn(Nclass, D) + np.array([0, -2])
X2 = np.random.randn(Nclass, D) + np.array([2, 2])
X3 = np.random.randn(Nclass, D) + np.array([-2, 2])
X = np.vstack([X1, X2, X3]).astype(np.float32)

Y = np.array([0]*Nclass + [1]*Nclass + [2]*Nclass)

# let's see what it looks like
fig = plt.figure(figsize=(14,10))
plt.scatter(X[:,0], X[:,1], c=Y, s=100, alpha=0.5)
plt.show()

N = len(Y)
# turn Y into an indicator matrix for training
T = np.zeros((N, K))
for i in range(N):
    T[i, Y[i]] = 1<jupyter_output><empty_output><jupyter_text>Tensorflow works with its own kind's of variables so we need to have a function for initing the weights. This will return a `tf` variable. It is going to be initialized the same was as in `numpy`, because everything in `TensorFlow` has a `numpy` analog. So it is going to be a a random normal of size `shape` with a standard deviation of `0.01`.<jupyter_code>def init_weights(shape):
    return tf.Variable(tf.random_normal(shape, stddev=0.01))<jupyter_output><empty_output><jupyter_text>We will also want to define the forward direction, just as we did before with `numpy`. It is going to take in `X, W1, b1, W2, b2` just as we did before. It will then use `tensorflow` functions: a tensorflow `sigmoid` and a tensorflow `matrix multiplication`. We then return **just the activation** (not the softmax). This is one main difference between numpy and tensorflow, is that when we calculate the **cost** we will want want the the **logits**, aka the activations, as the input- *not* the output of the softmax.<jupyter_code>def forward(X, W1, b1, W2, b2):
    Z = tf.nn.sigmoid(tf.matmul(X, W1) + b1)
    return tf.matmul(Z, W2) + b2<jupyter_output><empty_output><jupyter_text>Next up, we will create tensorflow **placeholders**. This represents your **X and Y data**. What happens in tensorflow is that it creates a graph, so it knows how to calculate everything, but nothing has a value yet. So the `tfx` below is just a placedholder for the data. We will say it is of type `float32`, and the shape of it is `None` by `D`. This way the second dimension will be `D` because that is the dimensionality of the data, but we can pass in any size `N`. We will do the same thing with `tfY`, a place holder for `Y`.<jupyter_code>tfX = tf.placeholder(tf.float32, [None, D])
tfY = tf.placeholder(tf.float32, [None, K])<jupyter_output><empty_output><jupyter_text>Now we will create our weights, similar to how we did in `numpy`.<jupyter_code>W1 = init_weights([D, M])         # dimensions (D x M)
b1 = init_weights([M])            # dimensions (M x 1)
W2 = init_weights([M , K])        # dimensions (M x K)
b2 = init_weights([K])            # dimensions (K x 1)<jupyter_output><empty_output><jupyter_text>Now we can get the output variable, remember this has no value yet.<jupyter_code>py_x = forward(tfX, W1, b1, W2, b2)<jupyter_output><empty_output><jupyter_text>**Note:** Here is another thing different about tensorflow. We define the `cost function`, which is the mean and then the function `soft_max_cross_entropy_with_logits`. This takes in our predictions, `py_x`, and the targets, `tfY`.<jupyter_code>cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(
    labels=tfY,
    logits=py_x
    )
)<jupyter_output><empty_output><jupyter_text>So Tensorflow is going to calculate the gradients and do gradient descent automatically, so we don't have to specify the derivative in tensorflow. We are going to create what is a called a train function, and that comes from `tf.train`. We pass in a `learning_rate`<jupyter_code>train_op = tf.train.GradientDescentOptimizer(0.05).minimize(cost)<jupyter_output><empty_output><jupyter_text>Next we will create a `predict` that is just the `argmax` of `py_x` on axis = 1.<jupyter_code>predict_op = tf.argmax(py_x, 1)<jupyter_output><empty_output><jupyter_text>In tensorflow we need to create things called sessions. For now just treat this as something that needs to be done.<jupyter_code>sess = tf.Session()
init = tf.global_variables_initializer()
sess.run(init)<jupyter_output><empty_output><jupyter_text>Now we will run our neural network by performing gradient descent. We will call `sess.run()` twice. The first time to perform backpropagation training, and the second time to make our predictions. This is going to take in whats called a `feed_dict`. This is a dictionary where the keys are tensorflow placeholders, and the values are the actual values you want to pass in (our X inputs and the T targets).<jupyter_code>for i in range(1000):
    sess.run(train_op, feed_dict={tfX: X, tfY: T})             
    pred = sess.run(predict_op, feed_dict={tfX: X, tfY: T})   # returns our predictions
    if i % 100 == 0:
        print("Accuracy:", np.mean(Y == pred))<jupyter_output>Accuracy: 0.9626666666666667
Accuracy: 0.962
Accuracy: 0.9613333333333334
Accuracy: 0.9606666666666667
Accuracy: 0.962
Accuracy: 0.962
Accuracy: 0.9613333333333334
Accuracy: 0.9613333333333334
Accuracy: 0.962
Accuracy: 0.962<jupyter_text>Scikit Learn Artificial Neural NetworkIn the real world we won't be be implementing our Neural Networks from scratch; it simply doesn't make sense to do that when we have awesome libraries already there for us to utilize. `Scikit Learn` has a great implementation of this! This can be done in only 3 lines of code! A line to create the model, a line to train the model, and a line to make predictions!<jupyter_code>import pandas as pd
from sklearn.neural_network import MLPClassifier
from sklearn.utils import shuffle

def get_data():
    df = pd.read_csv('data/ecommerce_data.csv')
    data = df.as_matrix()
    np.random.shuffle(data)
    X = data[:,:-1]
    Y = data[:,-1].astype(np.int32)

    # one-hot encode the categorical data
    N, D = X.shape
    X2 = np.zeros((N, D+3))
    X2[:,0:(D-1)] = X[:,0:(D-1)] # non-categorical

    # one-hot
    for n in range(N):
      t = int(X[n,D-1])
      X2[n,t+D-1] = 1
    X = X2

    # split train and test
    Xtrain = X[:-100]
    Ytrain = Y[:-100]
    Xtest = X[-100:]
    Ytest = Y[-100:]

    # normalize columns 1 and 2
    for i in (1, 2):
        m = Xtrain[:,i].mean()
        s = Xtrain[:,i].std()
        Xtrain[:,i] = (Xtrain[:,i] - m) / s
        Xtest[:,i] = (Xtest[:,i] - m) / s

    return Xtrain, Ytrain, Xtest, Ytest

Xtrain, Ytrain, Xtest, Ytest = get_data()
# create the neural network
model = MLPClassifier(hidden_layer_sizes=(20, 20), max_iter=2000)

# train the neural network
model.fit(Xtrain, Ytrain)

# print the train and test accuracy
train_accuracy = model.score(Xtrain, Ytrain)
test_accuracy = model.score(Xtest, Ytest)
print("train accuracy:", train_accuracy, "test accuracy:", test_accuracy)<jupyter_output>train accuracy: 0.9975 test accuracy: 0.93
{'fraction_non_alphanumeric': 0.1018410626893498, 'fraction_numerical': 0.02913073875553484, 'mean_word_length': 4.466878980891719, 'pattern_counts': {'":': 0, '<': 51, '<?xml version=': 0, '>': 52, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 5, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '339286', 'license': 'MIT', 'n_tokens_mistral': 2894, 'n_tokens_neox': 2718, 'n_words': 1100, 'path': 'Deep_Learning/01-Neural_Networks-07-TensorFlow.ipynb', 'repo_name': 'NathanielDake/NathanielDake.github.io', 'size': '8582'}