<jupyter_start><jupyter_text><NAME>Neural Networks in Tensorflow 2.xClassification basicsOptimization<jupyter_code># %tensorflow_version 2.x - switch only for Google Colab notebook
%tensorflow_version 2.x

import tensorflow as tf
print(tf.__version__)<jupyter_output>TensorFlow 2.x selected.
2.1.0<jupyter_text>Dataset representation* each **sample** is described by $n$ **features** (vector with $n$ numbers)* $m$ samples are represented by matrix with size $m \times n$  Linear regression - review  Example - Boston houses dataset (506x13)<jupyter_code>from   sklearn import datasets

X, y = datasets.load_boston(return_X_y=True)

X = tf.convert_to_tensor(X, dtype=float)
y = tf.convert_to_tensor(y, dtype=float)

m, n = X.shape

X.shape, y.shape<jupyter_output><empty_output><jupyter_text>Linear modelSimple linear model with $n+1$ parameters (weights):* $w_1, w_2, \ldots, w_n$ - called by _kernel_, _weights_,* $b$ - usually called by _bias_ or _intercept_.It performs the following matrix-vector operation:$$ y = Xw+b \in \mathbb{R}^m $$<jupyter_code># TODO: initialize model parameters W, b with zeros

W = tf.Variable(tf.zeros((n, 1)), dtype=tf.float32)
b = tf.Variable(0.0, dtype=tf.float32)
W.shape, b.shape<jupyter_output><empty_output><jupyter_text>* Implement the forward propagation $$ \hat y = Xw + b \in \mathbb{R}^{m} $$* Implement the _mean squarred error_ (MSE)<jupyter_code># TODO

def model(X): # Compute logits
  return tf.add(tf.matmul(X, W), b)
    
def loss(y_true, y_predict): # MSE loss
  return tf.reduce_mean(tf.pow(y_true - y_predict, 2))
loss( y, model(X) ).numpy()<jupyter_output><empty_output><jupyter_text>Optimization in Tensorflowsee https://www.tensorflow.org/api_docs/python/tf/keras/optimizers **Gradient descent algorithm**<jupyter_code># Choose optimizer
optimizer = tf.keras.optimizers.SGD(learning_rate=0.000001, momentum=0.0, nesterov=False)  # simple Gradient-descent algorithm
# optimizer = tf.keras.optimizers.RMSprop(learning_rate=0.01)
# optimizer = tf.keras.optimizers.Adam()<jupyter_output><empty_output><jupyter_text>Single step of optimizerSee* `minimize`* `optimizer.apply_gradients`<jupyter_code>J = lambda : loss(y, model(X))
print( "Loss before optimiztion step: ", J().numpy() )

optimizer.minimize(J, [W, b])

print( "Loss after optimiztion step: ", J().numpy() )<jupyter_output>Loss before optimiztion step:  592.1469
Loss after optimiztion step:  174.04004<jupyter_text>Utilizing gradients<jupyter_code>with tf.GradientTape() as tape:
    J = loss(y, model(X))
dW, db = tape.gradient(J, [W, b])
dW.shape, db.shape
grads_and_vars = [(dW, W), (db, b)]
optimizer.apply_gradients(grads_and_vars)<jupyter_output><empty_output><jupyter_text>Some test<jupyter_code># Check Gradient descent

with tf.GradientTape() as tape:
    J = loss(y, model(X))
dW, db = tape.gradient(J, [W, b])

nextW = W.numpy() - optimizer.get_config()['learning_rate'] * dW.numpy()
nextb = b.numpy() - optimizer.get_config()['learning_rate'] * db.numpy()

grads_and_vars = [(dW, W), (db, b)]
optimizer.apply_gradients(grads_and_vars)
print(W.numpy() - nextW)
print(b.numpy() - nextb)<jupyter_output>[[0.]
 [0.]
 [0.]
 [0.]
 [0.]
 [0.]
 [0.]
 [0.]
 [0.]
 [0.]
 [0.]
 [0.]
 [0.]]
0.0<jupyter_text>Classification - cross entropy loss  Binary classificationLets consider **binary classification**Assume that $ y, \hat y\in\mathbb{R}^{m\times 1} $ are true labels and predictions, respectively, i.e.$$ y^{(i)} \in \{0,1\}, \quad \hat y^{(i)} \in [0,1]\qquad (i=1,2,\ldots,m). $$Here we show that `tf.losses.log_loss( ytrue, ypred )` computes $$ \mathtt{log\_{}loss}( \boldsymbol y, \hat{\boldsymbol y} ) = - \frac{1}{m} \sum_{i=1}^{m} \left[ y^{(i)} \log( {\hat y}^{(i)} ) +  (1-y^{(i)}) \log( 1-{\hat y}^{(i)} ) \right] $$  `numpy`<jupyter_code>m,n = 300, 10
import numpy as np
np.random.seed(1234)

np_X, np_Y = datasets.make_classification(n_samples=m, n_features=n, random_state=1234)

np_X = np_X.astype(np.float32)
np_Y = np_Y.astype(np.float32)

np_Y = np.reshape( np_Y, (m,1) )

np_W = np.random.randn(n,1).astype(np.float32)
np_b = np.float32(0.0)

def sigmoid(x):
  return 1.0/(1.0+np.exp(-x))

np_logits = np_X.dot(np_W) + np_b
np_pred = sigmoid( np_logits )     # shape = (300,1)
np_loss = - np.mean( np_Y * np.log(np_pred) + (1.0 - np_Y)*np.log(1.0-np_pred) )

np_loss<jupyter_output><empty_output><jupyter_text>`tf.losses.log_loss`<jupyter_code>tf_X = tf.convert_to_tensor( np_X, dtype=tf.float32 )
tf_Y = tf.convert_to_tensor( np_Y, dtype=tf.float32 )

tf_W = tf.Variable( np_W.astype('float32') )
tf_b = tf.Variable( np.float32( np_b ) )

tf_logits = tf.add(tf.matmul(tf_X,tf_W), tf_b)
tf_pred = tf.nn.sigmoid( tf_logits )

tf_loss = tf.compat.v1.losses.log_loss( tf_Y, tf_pred )                     # log_loss

tf_loss<jupyter_output><empty_output><jupyter_text>`tf.nn.sigmoid_cross_entropy_with_logits`<jupyter_code># tf_logits = tf.add(tf.matmul(tf_X,tf_W), tf_b)

tf_loss2 = tf.reduce_mean( tf.nn.sigmoid_cross_entropy_with_logits(labels=tf_Y, logits=tf_logits) )

tf_loss2<jupyter_output><empty_output><jupyter_text>Multiclass classificationNiech $X,y$ będą danymi uczącymi ($m$ próbek pochodzących z $K$ klas). Wówczas wartość funkcji kosztu dla predykcji $\hat{y}$ dana jest wzorem (ang. _cross entropy_)$$\begin{aligned}L(y,\hat{y}) = - \frac{1}{m} \sum_{i \in m} \sum_{k \in K} y_{k}^{(i)} \log\hat{y}_{k}^{(i)}\end{aligned}$$<jupyter_code>from sklearn import datasets

digits = datasets.load_digits()
np_X = digits.data

np_X.shape

np_Y = tf.keras.utils.to_categorical( digits.target )

m,n = np_X.shape
K = np_Y.shape[1]
m,n,K<jupyter_output><empty_output><jupyter_text>`numpy`<jupyter_code>np_W = np.random.randn(n,K)
np_b = 0.0

def softmax(x):
  x -= np.max(x,axis=1,keepdims=True)
  x  = np.exp(x)
  x /= np.sum(x,axis=1,keepdims=True)
  return x

np_pred = softmax( np_X.dot(np_W) + np_b )     # shape = (1797,10)

np_loss1 = -np.sum( np_Y * np.log( np_pred ) ) / m
np_loss2 = -np.mean( np.log( np_pred[ np_Y==1 ] ) )

np_loss1, np_loss2<jupyter_output><empty_output><jupyter_text>`tf.nn.softmax_cross_entropy_with_logits`<jupyter_code>tf_X = tf.convert_to_tensor( np_X, dtype=tf.float32 )
tf_Y = tf.convert_to_tensor( np_Y )

tf_W = tf.Variable( np_W.astype('float32') )
tf_b = tf.Variable( np.float32( np_b ) )

tf_logits = tf.add(tf.matmul(tf_X,tf_W), tf_b)

tf_loss = tf.reduce_mean( tf.nn.softmax_cross_entropy_with_logits( labels=tf_Y, logits=tf_logits ) )

tf_loss<jupyter_output><empty_output><jupyter_text>Toy dataset - FACESW Labeled Faces in the Wild, a database of face photographs designed for studying the problem of unconstrained face recognition. The data set contains more than 13,000 images of faces collected from the web. Each face has been labeled with the name of the person pictured. 1680 of the people pictured have two or more distinct photos in the data set. The only constraint on these faces is that they were detected by the Viola-Jones face detector. More details can be found in the [technical report](http://vis-www.cs.umass.edu/lfw/)**Dataset subset (min_faces_per_person=70, resize=0.4)**- 1288 images of size 50x37<jupyter_code>import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt
from sklearn import datasets
lfw_people = datasets.fetch_lfw_people(min_faces_per_person=70, resize=0.4)
plt.imshow( lfw_people.images[6] , cmap='gray')
plt.show()
print( lfw_people.target[6] , lfw_people.target_names[lfw_people.target[6]] )
n_samples, height, width = lfw_people.images.shape

Xdata = np.reshape( lfw_people.data, (n_samples, -1) )
ydata = lfw_people.target

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(Xdata, ydata, test_size=0.25, random_state = 980)

n_classes = 7
n_features = Xdata.shape[1]
target_names = lfw_people.target_names

print('')
print('n_samples  = ', n_samples)
print('n_features = ', n_features)
print('n_classes  = ', n_classes)
print('Xdata.shape = ', Xdata.shape)
print('X_train.shape = ', X_train.shape)
# print('y_train_one_hot.shape = ', y_train_one_hot.shape)
print('X_test.shape = ', X_test.shape)
# print('y_test_one_hot.shape = ', y_test_one_hot.shape)
X_train = X_train / X_train.max()<jupyter_output><empty_output><jupyter_text>MLP  Forward propagation<jupyter_code>tf.random.set_seed(1234)

W1 = tf.Variable(tf.random.normal((n_features, 100)), dtype=tf.float32)
b1 = tf.Variable(tf.random.normal((100,)), dtype=tf.float32)

W2 = tf.Variable(tf.random.normal((100, 50)), dtype=tf.float32)
b2 = tf.Variable(tf.random.normal((50,)), dtype=tf.float32)

W3 = tf.Variable(tf.random.normal((50, 7)), dtype=tf.float32)
b3 = tf.Variable(tf.random.normal((7,)), dtype=tf.float32)


def model(X):
  Z1 = tf.matmul(X, W1) + b1
  A1 = tf.nn.tanh(Z1)

  Z2 = tf.matmul(A1, W2) + b2
  A2 = tf.nn.tanh(Z2)

  Z3 = tf.matmul(A2, W3) + b3
  A3 = tf.nn.tanh(Z3)

  return Z3<jupyter_output><empty_output><jupyter_text>Loss<jupyter_code>y_train_one_hot = tf.one_hot(y_train, depth=n_classes)

def loss(y_pred, Y):
  loss = tf.nn.softmax_cross_entropy_with_logits(Y, y_pred)
  return tf.reduce_mean(loss)
loss(model(X_train), y_train_one_hot)<jupyter_output><empty_output><jupyter_text>Backward propagation<jupyter_code>with tf.GradientTape() as tape:
  L = loss(model(X_train), y_train_one_hot)

dW1, db1, dW2, db2, dW3, db3 = tape.gradient(L, [W1, b1, W2, b2, W3, b3])<jupyter_output><empty_output><jupyter_text>Single step of GD<jupyter_code>optimizer = tf.keras.optimizers.RMSprop()

optimizer.minimize(lambda : loss(model(X_train), y_train_one_hot), [W1, b1, W2, b2, W3, b3])
loss(model(X_train), y_train_one_hot)<jupyter_output><empty_output><jupyter_text>Final training loop<jupyter_code>for epoch in range(5000):
  optimizer.minimize(lambda : loss(model(X_train), y_train_one_hot), [W1, b1, W2, b2, W3, b3])
  if epoch % 100 == 0:
    print(f'Epoch: {epoch}, loss:{loss(model(X_train), y_train_one_hot)}')
y_pred = tf.nn.softmax(model(X_train))
y_pred = tf.argmax(y_pred, axis=1)
np.mean(y_pred == y_train)
y_pred_test = tf.nn.softmax(model(X_test))
y_pred_test = tf.argmax(y_pred_test, axis=1)

np.mean(y_pred_test == y_test)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15380825697113015, 'fraction_numerical': 0.027096265641935166, 'mean_word_length': 5.690837178642057, 'pattern_counts': {'":': 0, '<': 79, '<?xml version=': 0, '>': 79, 'https://': 1, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '166292', 'license': 'MIT', 'n_tokens_mistral': 4347, 'n_tokens_neox': 4177, 'n_words': 811, 'path': 'NN-DL/TensorflowWorkshop/2.Linear_regression_and_Classification_basics_exercise.ipynb', 'repo_name': 'TheFebrin/Machine-Learning', 'size': '10154'}