<jupyter_start><jupyter_text>1. TensorFlow BriefTensorFlow is called 'TensorFlow' because it handles the flow (node/mathematical operation) of Tensors, which are data structures that you can think of as multi-dimensional arrays. Tensors are represented as n-dimensional arrays of base dataypes such as a string or integer -- they provide a way to generalize vectors and matrices to higher dimensions.The ```rank``` of a Tensor provides the number of dimensions (n-dimensions) -- you can also think of this as the Tensor's order or degree.The ```shape``` of a Tensor defines its number of dimensions and the size of each dimension. //How many values in side each dimension.Let's first look at 0-d Tensors, of which a scalar is an example:  1.1 Scalar - 0 Dimension, Constant<jupyter_code>sport = tf.constant("Tennis", tf.string)
constant_2 = tf.constant(2, dtype=tf.float64)
number = tf.math.sqrt(constant_2)
print("'Sport' is a {}-d Tensor".format(tf.rank(sport).numpy()))
print("'number' is a {}-d Tensor".format(tf.rank(number).numpy()))<jupyter_output>'number' is a 0-d Tensor<jupyter_text>1.2 Vector,  List - 1 Dimention Tensors<jupyter_code>sports = tf.constant (["Tennis", "Basketball"], tf.string)
numbers = tf.constant ([3.141592, 1.4142213, 2.71821], tf.float64)
print("'Sports' is a {}-d Tensor with shape: {}".format(tf.rank(sports).numpy(), tf.shape(sports)))
print("`Numbers` is a {}-d Tensor with shape: {}".format(tf.rank(numbers).numpy(), tf.shape(numbers)))<jupyter_output>`Numbers` is a 1-d Tensor with shape: [3]<jupyter_text>1.3 Metrix - 2-D Dimentions Tensor<jupyter_code>matrix = tf.constant([[1.0, 2.0, 3.0, 4.0], [5, 6, 7, 8]], tf.float32)
matrix
assert isinstance(matrix, tf.Tensor)
assert tf.rank(matrix).numpy()==2<jupyter_output><empty_output><jupyter_text>1.4 Multiple Dimentions Tensor<jupyter_code># tf.zeros(shape, dtype=tf.dtypes.float32, name=None)<jupyter_output><empty_output><jupyter_text>1.4.1 Shape provide the number of elements in each Tensor Dimention<jupyter_code>a=tf.zeros(shape=4, dtype=tf.dtypes.float32)<jupyter_output><empty_output><jupyter_text>1.4.2 4-D Tensor: 10 Images with 256x256 pixel with GRB 3 colors chanel.<jupyter_code>images = tf.zeros([10, 256, 256, 3], tf.int32)
assert isinstance(images, tf.Tensor)
assert tf.rank(images).numpy() == 4
images
import pandas as pd
Image(url= "https://miro.medium.com/max/2000/1*8mlH1a4MEOxr6wDjCZE8Ug.png")
assert tf.shape(images).numpy().tolist() == [10, 256, 256, 3]
print("'images' is a {}-d Tensor with shape: {}".format(tf.rank(images).numpy(), tf.shape(images)))<jupyter_output>'images' is a 4-d Tensor with shape: [ 10 256 256   3]<jupyter_text>1.4.3 Use Slicing to access subtensors within a hgher-rank Tesor<jupyter_code>matrix
row_vector = matrix[1]
row_vector
column_vector = matrix[:, 2]
column_vector 
scalar = matrix[1,2]
scalar
print("'row_vector': {}".format(row_vector.numpy()))
print("'column_vector': {}".format(column_vector.numpy()))
print("'scalar': {}".format(scalar.numpy()))<jupyter_output>'scalar': 7.0<jupyter_text>2. Computation on Tensors<jupyter_code>a = tf.constant(15)
b = tf.constant(16)
c1 = tf.add(a,b)
c2 = a + b
c1
c2
matrix1 = tf.constant([[1,2], [3,4]], tf.int16)
matrix2 = tf.constant([['a','b'], ['c','d']], tf.string)
matrix1
matrix2
def func(a,b):
    c = tf.add(a,b)
    d = tf.subtract(b,1)
    e = tf.multiply(c, d)
    return e
func(4,5)<jupyter_output><empty_output><jupyter_text>3. Neural Networks in TensorFlow  3.1 Build a Dense Layer<jupyter_code>class DenseLayer(tf.keras.layers.Layer): 
    def __init__(self, num_output_nodes):    #Constructor
        super(DenseLayer, self).__init__()
        self.num_output_nodes = num_output_nodes
    def build(self, input_shape):  # Build Weight and bias
        d = int (input_shape[-1])
        self.w = self.add_weight ("weight", shape=[d, self.num_output_nodes])
        self.b = self.add_weight ("bias", shape=[1, self.num_output_nodes])
    def call (self, x):    # Compute output y
        z = tf.matmul(x, self.W) + self.b
        y = tf.tanh(z)
        return y
tf.random.set_seed(1)
myLayer = DenseLayer(3)
myLayer.build((1,2))   # 1-> W, 2->bias
x_input = tf.constant([1,2,3], tf.int16, shape=(1,3))  # x_input is as a shape of 1-row, 2-col, with values are [3, 4]
y = myLayer.call(x_input) 

x_input
print(y.numpy())
#from mitdeeplearning import lab1
#lab1.test_custom_dense_layer_output(y)<jupyter_output><empty_output><jupyter_text>3.2 Build NN by Sequential Model - Keras<jupyter_code>### Define a Neural network using the sequential API ### 
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense
# Construct Sequencial NN:
# X (4) -> 1st Dense Layer (3 output units, with activation sigmoid) -> 2nd Dense Layer (5 output units, with activation as Rectified Linear Unit)
# Define the number of outputs
num_output_nodes = 3
# https://keras.io/api/models/sequential/
# https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dense?version=stable
# First Define the model
# Create a Sequential model 
seqModel = tf.keras.Sequential()
# Create First Dense_Layer, with the output units =3, activation function as 'sigmoid‘， weight ='glorot_uniform'/'random_normal
dense_layer = tf.keras.layers.Dense(num_output_nodes, input_shape = (4,), activation = 'sigmoid', kernel_initializer = 'glorot_uniform', bias_initializer ='zeros')
# Add the frist dense layer to the sequential model
seqModel.add(dense_layer)
# Add another Dense layer with 4 outuput unites // No need to put X vector
seqModel.add(tf.keras.layers.Dense(5, activation ='relu'))
seqModel.output_shape
# Initialize Input, shold be consistent with the first dense layer. //input_shape = (4,)
x_input = tf.constant([[1,2,3,4]])
seqModel_output = seqModel(x_input).numpy()
seqModel_output
print(seqModel_output)
## Add a Dense Layer as the first layer: Define Number of output Nodes, X input share is 16 Elements, activation funtion as sigmoid. /'relu'.
# seqModel.add(tf.keras.layers.Dense(num_output_nodes, input_shape=(16,), activation = 'sigmoid'))
## Add second Dense layer with output of 4 Units.
# seqModel.add(tf.keras.layers.Dense(4))
# The final output units is 4. 
# seqModel.output_shape<jupyter_output><empty_output><jupyter_text>3.3 Define Sequencial Model by Subclassing of Model Class<jupyter_code># https://www.tensorflow.org/api_docs/python/tf/keras/Model?version=stable
from tensorflow.keras import Model
from tensorflow.keras.layers import Dense
class subclassModel(tf.keras.Model):
    # In __init__, define the Model's layers
    def __init__(self, num_output_nodes):
        super(subclassModel, self).__init__()
        self.dense_layer = Dense(num_output_nodes, activation = 'relu', kernel_initializer = 'random_normal', bias_initializer ='zeros' )
    
    # Define forward pss in the call function
    def call(self, inputs):
            return self.dense_layer(inputs)
num_output_nodes = 5
seqModel2= subclassModel(num_output_nodes)
x_input = tf.constant([[1,2,3,4,5], [6,7,8,9,10]], shape = (2,5))
print(seqModel2.call(x_input))<jupyter_output>tf.Tensor(
[[0.         0.         0.         0.16681956 0.        ]
 [0.         0.         0.         0.20123696 0.11567369]], shape=(2, 5), dtype=float32)<jupyter_text>3.4 Define Sequential Model by Subclassing with argument 'isidentity'<jupyter_code>from tensorflow.keras import Model
from tensorflow.keras.layers import Dense
class IdentityModel(tf.keras.Model):
    def __init__(self, num_output_nodes):
        super(IdentityModel, self).__init__()
        self.dense_layer = tf.keras.layers.Dense(num_output_nodes, activation='tanh')
        
    # the behavior where the network outputs the input, unchanged, under control of the isidentity argument (Identity Metrix).
    def call(self, inputs, isidentity=False):
        x = self.dense_layer(inputs)
        if isidentity:
            return inputs
        return x
    
num_output_nodes = 3
seqModel3 = IdentityModel(num_output_nodes)
x_input = tf.constant([[1,0,0],[0,1,0],[0,0,1]], shape = (3,3))
print(seqModel3.call(x_input, isidentity=True))
print(seqModel3.call(x_input, isidentity=False))<jupyter_output>tf.Tensor(
[[-0.43272695 -0.4129905   0.52313375]
 [-0.5649544  -0.040906   -0.7084498 ]
 [ 0.65080065  0.6764062   0.65374887]], shape=(3, 3), dtype=float32)<jupyter_text>4. Automatic Differentiation in TensorFlow [Automatic differentiation](https://en.wikipedia.org/wiki/Automatic_differentiation)is one of the most important parts of TensorFlow and is the backbone of training with[Backpropagation](https://en.wikipedia.org/wiki/Backpropagation)Use TesorFlow GradientTape ['tf.GradientTape'](https://www.tensorflow.org/api_docs/python/tf/GradientTape?version=stable)to trace operations for computing gradients. When a forward pass is made through the network, all forward-pass operations get recorded to a "tape"; then, to compute the gradient, the tape is played backwards. By default, the tape is discarded after it is played backwards; this means that a particular `tf.GradientTape` can onlycompute one gradient, and subsequent calls throw a runtime error. However, we can compute multiple gradients over the same computation by creating a ```persistent``` gradient tape.  Compute gradients using GradientTape and access them for computation. Define the simple function $ y = x^2$ and compute the gradient:<jupyter_code># Gradient Computation with GradientTape
# y = x^2
# x = 3.0
x = tf.Variable(3.0)
# Initiate the Gradient Tape
with tf.GradientTape() as tape:
    # Define the function
    y = x * x 
# Access the gradient -- derivative of y with respect to x
dy_dx = tape.gradient(y,x)
assert dy_dx.numpy() == 6.0
dy_dx.numpy()<jupyter_output><empty_output><jupyter_text>4.1 Automatic Differenciation and SGD (Stochastic Gradient Descent) In training neural networks, use differentiation and stochastic gradient descent (SGD) to optimize a loss function. Use GradientTape to compute and access derivatives.Use automatic differentiation and SGD to find the minimum of  𝐿=(𝑥−𝑥𝑓)2 .   𝑥𝑓  is a variable for a desired value which is should be optimize for;  𝐿  represents a loss, need to be minimized. ( 𝑥𝑚𝑖𝑛=𝑥𝑓 )<jupyter_code># Loss Function mnimizatin with Automatic diffrentication and SGD
# Initialize a random value for x. //mean=0.0, standard diviation (stddev=1). 68% data points gathering between [-1,1], mean==0.
x = tf.Variable([tf.random.normal([1])])
print("Initializing x={}".format(x.numpy()))
Image(url= "https://www.varsitytutors.com/assets/vt-hotmath-legacy/hotmath_help/topics/normal-distribution-of-data/normal-distribution-1.gif")
learning_rate = 1e-2 # Learning Rate for SGD. 1e-2 = 1 x 10^-2 =0.01
history = []   # Initialize history with empaty array
# Define the target value
x_f = 8  
# Iterations = ; Compute Loss in each iteration 
# Compute the derivative of loss (y) with respect to x, and perform the SGD update.
# Use GradientTape to record each iteration's derivative
for i in range (500):
    with tf.GradientTape() as tape:
        loss = (x-x_f)**2
        
        # gradient = dloss_dx : compute the derivative of loss with the respect of x
        gradient = tape.gradient(loss,x)  # "forward pass": record the current loss on the tape

        # SDG update new x 
        new_x = x - learning_rate * gradient 
        
        # Assign new_x to x
        x.assign(new_x)
        
        history.append(x.numpy()[0])
        
# Plot the evolution of x as we optimize towards x_f
plt.plot(history, 'r-')   #Predicted value
plt.plot([0, 500],[x_f, x_f], 'b-')   # 0,x_f; 500,x_f;
plt.legend(('Predicted', 'True'))
plt.xlabel('Iteration') 
plt.ylabel('x value')
# after 200 iternation, the prdiction value is getting close to the true value as 4.<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12353241449719245, 'fraction_numerical': 0.03428620044240259, 'mean_word_length': 4.635186960690317, 'pattern_counts': {'":': 1, '<': 51, '<?xml version=': 0, '>': 55, 'https://': 8, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '485394', 'license': 'MIT', 'n_tokens_mistral': 4233, 'n_tokens_neox': 3963, 'n_words': 1140, 'path': 'Music_Generation_Engine.ipynb', 'repo_name': 'Themoonflow/MusicGenerationRNNs', 'size': '11754'}