<jupyter_start><jupyter_text>Hello World: Introduction to TensorFlowIn this session, we will be writing our first ever TensorFlow code. If you are already familiar with Python and NumPy, you will find the syntax is pretty similar and quite easy to understand. However, even if you are not, don't worry, this tutorial is written for people who are lacking previous Python/NumPy experience.  Importing TensorFlowTo use TensorFlow, you must import it in your code first. If you know C/C++, this is equivalent to something like `include `. Here, note below that we are importing TensorFlow with an alias (nickname) called `tf` because typing `tensorflow` in our code all the time is kind of inconvenient (as opposed to just typing the shorthand `tf`). You'll realize what I mean soon.<jupyter_code>import tensorflow as tf<jupyter_output><empty_output><jupyter_text>TensorsIf you were able to run the code above, i.e. `import tensorflow`, you are a half way there. Yes I'm serious. TensorFlow is actually quite easy to pick up and the syntax is straightforward. Rather, the biggest frustration point is always installing and configuring TensorFlow. So the fact that you were able to run the code above without an error already indicate you are half way to success for this session.Now, the first thing we are going to learn is how to create a variable so that we can do some math later. In TensorFlow, variables are basically called "Tensors," and yes, these are the same as tensors you've learned from a math course---the ones that generalize matrices.Let's first create a bunch of tensors, just to get started:<jupyter_code>a = tf.constant(7)
b = tf.constant(3.6)
c = tf.constant([1, 2])
d = tf.constant([[1, 2, 3], [4, 5, 6]])
e = tf.constant([[[1, 2, 3], [4, 5, 6]], [[7, 8, 9], [10, 11, 12]]])

print(a)
print(b)
print(c)
print(d)
print(e)<jupyter_output><empty_output><jupyter_text>How was it? Easy, huh? So yes, creating tensors are just as simple as `tf.constant()` and throw in a number/numbers within the parenthesis. Here, without square brackets `[]`, the number will be recognized as a scalar, or a zero-dimensional (0-D) tensor by TensorFlow. If you have square brackets, then the numbers will be recognized as a vector (1-D tensor), a matrix (2-D tensor), or some high-dimensional tensor, depending on how many layers of square brackets you nested.Please also note that tensors have `shape` (see what's been printed above). The shape of a tensor is basically the number of elements in each axis. In case of a scalar (0-D tensor), it's just a single number, and the shape is basically empty `()`. For a vector (1-D tensor), the shape looks like `(n,)` where `n` is the dimension of the vector (number of elements in the vector). For a matrix (2-D tensor), the shape looks like `(m,n)`, where `m` is the number of rows and `n` is the number of columns of the matrix. A 3-D tensor can be thought of as 2-D tensors (matrices) stacked up in the third dimension. For example, if there are three 2x2 matrices stacked up on each other, they form a 3-D tensor of the shape `(2, 2, 3)`---the first (2,2) implies the shape of those individual matrices, and the last 3 implies how many of them are in the stack.Finally, tensors also have `dtype`, indicating the variable type of the tensor. For example, `int32` indicates that the tensor is an **int**eger type and `float32` says that the tensor is a floating point number. The number 32 at the end is a suffix indicating the number of bits used for representing these tensors. 32-bit data types are the most common, but in case you need a high precision, you can use 64-bit data types. More detailed discussion about the data types is in fact beyond the scope of this course. For now, it should be sufficient to know that there are two major data types, namely `int32` and `float32`, and that they are not compatible to each other (we'll see this in the 'Operations' section below).  Tensor Indexing/SlicingTensorFlow allows you to access elements of a tensor via indexing and slicing. If you know Python already, TensorFlow follows the standard syntax of Python, so a lot of things below will make sense immediately. Even if you're not, the syntax is pretty straightforward, so nothing to worry about.Let's first take a look at 1-D tensors:<jupyter_code>a = tf.constant([0,1,2,3,4,5,6,7,8,9])
print(a)<jupyter_output><empty_output><jupyter_text>First of all, elements of tensors can be accessed via an index put within a pair of square brackets. For example, the 7-th element of tensor `a` can be accessed with `a[7]`. Note that the indices start at 0 in python. Also, an index can be a negative integer. In that case, the elements will be counted backward, from the last element in the array. That is, `a[-1]` is the last element in the array, `a[-2]` is the second from the last, so on and so forth.<jupyter_code>print(a[0])   # zero-th element
print(a[3])   # thrid element
print(a[-1])  # last element<jupyter_output><empty_output><jupyter_text>Sometimes, you may need to access a range of elements, not just a single element. For example, let's say you would like to take all elements between the third and the seventh element. In TensorFlow (like in regular Python), this is called slicing, and can be done with the colon `:` operator. The basic usage is quite simple. Within the square bracket, put a range of indices (instead of a single index), defined as `start:end:step`. Note that `end` is exclusive. See below, for example:<jupyter_code>print(a[0:3:1])     # sub-array containing 0-th, 1-st, and 2-nd elements of `a`. Note 3 is exclusive.
print(a[0:3:2])     # now the step size is 2, so it will print 0-th and 2-nd elements.
print(a[-3:-1:1])   # start and end indices can be negative.
print(a[-1:-3:-1])  # step size can also be negative.<jupyter_output><empty_output><jupyter_text>Also, `start`, `end`, `step` can be omitted depending on the context.<jupyter_code>print(a[:3:1])   # start index is omitted. it is assumed to be 0.
print(a[:3])     # step size is omitted. it is assumed to be 1.
print(a[7:])     # end index and step size are omitted. if end index is omitted, it will print through the last element.
print(a[::2])    # start and end indices are missing.
print(a[::-1])   # start and end indices are missing. step is negative. this prints out `a` in a reversed order.
print(a[:])      # start, end, and step are all missing. this is the same as `a`.<jupyter_output><empty_output><jupyter_text>Tensors of higher rank can be indexed/sliced in a similar way. It's just that you need indices for different axis separately. See the example below.<jupyter_code>b = tf.constant([[1,2,3,4],[5,6,7,8],[9,10,11,12]])
print(b, '\n')
print(b[1, 2])    # element at the 1-st row and 2-nd column
print(b[0, :])    # 0-th row and all columns
print(b[:, 1])    # all rows in 1-st column
print(b[0:2, ::-1])  # 0-th and 1-st row, columns reversed<jupyter_output><empty_output><jupyter_text>Manipulating ShapesWhen you write your own machine learning code later, a surprisingly large amount of your time will be spent on manipulating the shapes of tensors. For example, it is very common to 'flatten' a 2-D tensor to a 1-D tensor by concatenating the rows into a single row. `tf.reshape()` provides a fast reshaping operation, as demonstrated in the examples below:<jupyter_code>x = tf.constant([[0,1,2,3,4,5,6,7,8,9], [10,11,12,13,14,15,16,17,18,19], [20,21,22,23,24,25,26,27,28,29]])
print(x)
print(tf.reshape(x, (1,30)))    # flatten it into a single row
print(tf.reshape(x, (5,6)))     # reshape it to a 5x6 matrix
print(tf.reshape(x, (5,-1)))    # if the size of an axis is negative, tensorflow automatically calculates the right size. 
print(tf.reshape(x, (5,3,2)))   # reshape it to a 3-D tensor<jupyter_output><empty_output><jupyter_text>In the above, note that `tf.reshape()` just rearranges the tensor shape without reordering the elements. For example, reshaping a `(3,10)` tensor to a `(10,3)` tensor is not equivalent to the matrix transpose:<jupyter_code>print(tf.reshape(x, (10,3)))
print(tf.transpose(x))<jupyter_output><empty_output><jupyter_text>Moreover, during tensor algebra, tensors with dummy dimensions (axis of size 1) frequently show up. `tf.squeeze()` allows you to get rid of such dummy dimensions. See the example below.<jupyter_code>x = tf.constant([[[1,2,3,4,5]]])   # (1,1,5) tensor. the first two axes are dummy dimensions.
print(x)
print(tf.squeeze(x))<jupyter_output><empty_output><jupyter_text>Conversely, sometimes, you may require to add a dummy dimension to match the shape of tensors. This can be done by `tf.expand_dims()`.<jupyter_code>x = tf.constant([1,2,3,4,5])        # tensor of shape (5,)
print(x)
print(tf.expand_dims(x, axis=0))    # inserts a new axis at 0-th dimension. becomes (1,5)
print(tf.expand_dims(x, axis=1))    # inserts a new axis at 1-st dimension. becomes (5,1)
print(tf.expand_dims(x, axis=-1))   # inserts a new axis at the last dimension. equivalent to the previous line.<jupyter_output><empty_output><jupyter_text>Operations (Tensor Ops)Tensors can be added, subtracted, multiplied, divided, exponentiated, etc., through a variety of mathematical operations defined in TensorFlow. For example, the code below shows the basic arithmetic operations among scalar tensors:<jupyter_code>addition = a + b
subtraction = a - b
multiplication = a * b
division = a / b

print(addition)
print(subtraction)
print(multiplication)
print(division)<jupyter_output><empty_output><jupyter_text>Oops, an error! If you read the error message carefully, it says "... input ... was expected to be a int32 tensor but is a float tensor ..." Indeed, like I said above, two tensors with different `dtype`s are not compatible to each other. When we first defined the tensor `a` and the tensor `b`, we didn't explicitly say anything about their data types, but tensorflow just inferred a data type from the number we assigned. Tensor `a`, which was initiallized with the value 7, was automatically recognized as an integer data type, whereas, Tensor `b`, initialized with the value 3.6 was automatically recognized as a floating point data type. So when you deal with Tensors, you need to be very careful about their data types.By the way, if you really want to play with tensors with two different data types, there is actually a way, which is to explicitly convert the data type. The conversion, also called *casting*, can be conducted by calling `tf.cast()` function. The example below should be self-explanatory.<jupyter_code>a = tf.cast(a, dtype=tf.float32)
addition = a + b
subtraction = a - b
multiplication = a * b
division = a / b

print(addition)
print(subtraction)
print(multiplication)
print(division)<jupyter_output><empty_output><jupyter_text>BroadcastingOne of the frequent situations in tensor algebra is adding the same vector quantity to a list of vectors (a 2-D tensor), multiplying a different scaling factor on each dimension, and such. For example, imagine you have an array of points, whose xyz-coordinates are stored as a (N, 3) tensor, where N is the number of points and 3 represents the xyz. Say, you want to move these points in the direction `[3, 1, 2]`. In this case, a brute force approach would be to repeat adding `[3, 1, 2]` to each of the rows in the (N,3) coordinate tensor using for loop or something, which apparently requires a lot of computational time. Alternatively, you may duplicate `[3, 1, 2]` by N times, make it a (N, 3) tensor, so that it can be added to the coordinate tensor directly. However, this can be really inconvenient and make your code unnecessarily complicated, not to mention the waste of memory due to the duplication.For this situation, TensorFlow supports what is called *broadcasting*. Simply speaking, when you try to combine two tensors with different sizes via adding, multiplying (element-wise), subtracting, etc., TensorFlow automatically recognizes axes that are compatible/incompatible and stretches the shape of one tensor to fit another. For example, consider the following example of adding a (10,3) tensor and a (1,3) tensor:<jupyter_code>a = tf.random.uniform(shape=(5,3))    # a tensor of a shape (5,3) filled with random numbers sampled from a uniform distribution in range [0,1]
b = tf.constant([1,2,3], dtype=tf.float32)  # (3,) tensor

print(a)
print(b)
print(a+b)     # this should be invalid in a rigorous mathematical sense. however, tensorflow broadcasts `b` to fit `a` so that the addition can still be calculated<jupyter_output><empty_output><jupyter_text>While broadcasting can come quite handy in many different situations, you have to be very careful that broadcasting is not a magic wand that makes things happen all the time. Below are some common mistakes that beginners make:<jupyter_code>a = tf.random.uniform(shape=(5,3))    # again, a (5,3) tensor
b = tf.constant([1,2,3,4], dtype=tf.float32)  # (4,) tensor this time. note this is not compatible with `a` in any axis.

print(a+b)  # this will produce an error
a = tf.random.uniform(shape=(5,3))    # again, a (5,3) tensor
b = tf.random.uniform(shape=(2,3))    # (2,3) tensor. 3 is compatible, but not 2. 
print(a*b)  # this will produce an error
a = tf.random.uniform(shape=(5,3))    # again, a (5,3) tensor
b = tf.random.uniform(shape=(1,3,1))  # (1,3,1) tensor. 3 is compatible, the rank of the tensor is larger than 'a' 
print(a*b)  # this will produce an error<jupyter_output><empty_output><jupyter_text>The above mistakes may sound silly when they are explicitly stated like above, but those are quite common in practice, as the convenience of the broadcasting operation makes people forget about tensor dimensions.  Some more Tensor OpsOkay, now that we know how to define tensors and perform the basic arithmetic operations, let's see below some other mathematical operations defined in TensorFlow. While it would be nice to know all the functions available in TensorFlow, we will not cover the exhaustive list of functions here. Rather, we will see only a few subsets of those operations below. For the full list of details, please see the [official documentation](https://www.tensorflow.org/api_docs/python/tf/math).<jupyter_code>print(tf.add([1,2,3], [4,5,6]))                 # element-wise addition. equivalent to '+' operator
print(tf.abs([[-1, 2], [-3, 4]]))               # element-wise absolute values
print(tf.cos([3.14, 1, 0.25]))                  # element-wise cosine
print(tf.multiply([1, 2], [3, 4]))              # element-wise multiplication. equivalent to '*' operator
print(tf.matmul([[1, 2], [3, 4]], [[2], [3]]))  # matrix multiplication. note the difference with tf.multiply. equivalent to '@' operator
print(tf.reduce_sum([[1,2], [3,4]])             # summation of all elements.
print(tf.argmax([[1,2],[3,4]]))                 # find the index of the largest element<jupyter_output><empty_output><jupyter_text>There are also some linear algebra operations defined in TensorFlow.<jupyter_code>X = tf.constant([[3, 1],[2,4]], dtype=tf.float32)
print(tf.linalg.det(X))    # determinant of X
print(tf.linalg.trace(X))  # trace of X
print(tf.linalg.inv(X))    # matrix inverse of X
print(tf.linalg.eig(X))    # eigenvalues and eigenvectors of X<jupyter_output><empty_output><jupyter_text>See? Not so complicated, right? At least, you can now use TensorFlow as a linear algebra library, where you can calculate algebraic equations among tensors, which is quite cool already.But there is something beyond. As we will see later, deep neural networks involve massive tensor operations. TensorFlow is in fact optimized for those large tensor operations. To talk about this, let's consider an example of computing multiplication of two large matrices, say (10000, 5000) and (5000, 7000):<jupyter_code>import time   # a library imported to measure computational time

A = tf.random.uniform([10000, 5000])   # create a random matrix of a size 10000 x 5000. each value ranges between [0, 1]
B = tf.random.uniform([5000, 7000])    #

tic = time.time()
C = tf.matmul(A, B)       # could have written as `A @ B`
toc = time.time()

print(C)
print(f'{toc-tic} seconds elapsed.')<jupyter_output><empty_output><jupyter_text>It's a pretty heavy operation. If you think about it, multiplying a row of A with a column of B requires 5000 times of scalar-scalar multiplications, and we need to repeat that for 10000 x 7000 times in total, leading to the computational complexity of $\mathcal{O}(MNP)$, where $M$, $N$, $P$ are the dimensions of matrices being multiplied---$M \times N$ matrix multiplied by $N \times P$ matrix (if you don't know the big-O notation, please watch [this](https://www.youtube.com/watch?v=__vX2sjlpXU)).Now, TensorFlow uses hardware accelerators such as GPUs. In a nutshell, GPUs are built for handling complex graphics rendering tasks while you are playing video games, watching movies, etc. But in fact, those graphics rendering operations turn out to be a bunch of matrix/vector operations (geometric transforms). So, for efficient rendering, GPUs are highly optimized for matrix/vector operations and they are usually multiple orders of magintude faster than CPUs, making it also suitable for scientific computing. TensorFlow can get really powerful by making it so easy for you to run complex tensor operations on GPUs.In fact, when you define a Tensor, they live in either of two universes---CPU or GPU. You can check that by calling `.device` after the Tensor name. For example:<jupyter_code>print(A.device)
print(B.device)
print(C.device)<jupyter_output><empty_output><jupyter_text>Depending on which configuration you are using, you will see either CPU or GPU. If you have a [TensorFlow compatible GPU card](https://www.tensorflow.org/install/gpu), the default should be GPU (if you see CPU on the list, that means your TensorFlow installation is problematic).Although TensorFlow automatically picks a hardware accelerator implicitly, there are occasions where you may need to specify the device explicitly. For example, if you have multiple GPU cards, then you can make operations parallel by distributing the load to different GPU cards. For those cases, `tf.device()` sets a scope for the device:<jupyter_code>print("On CPU:")
with tf.device("CPU:0"):
    A = tf.random.uniform([10000, 5000])
    B = tf.random.uniform([5000, 7000])
    print("A:", A.device)
    print("B:", B.device)
    tic = time.time()
    C = tf.matmul(A, B)
    toc = time.time()
    print(f'{toc-tic} seconds elapsed on CPU.')

# Only if GPU is available
if tf.config.experimental.list_physical_devices("GPU"):
    print("On GPU:")
    with tf.device("GPU:0"):    # If you have multiple GPU cards, you can change numbers to GPU:1, GPU:2, etc.
        A = tf.random.uniform([10000, 5000])
        B = tf.random.uniform([5000, 7000])
        print("A:", A.device)
        print("B:", B.device)
        tic = time.time()
        C = tf.matmul(A, B)
        toc = time.time()
        print(f'{toc-tic} seconds elapsed on GPU.')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09625696561875723, 'fraction_numerical': 0.022973399221953527, 'mean_word_length': 4.5704245973645685, 'pattern_counts': {'":': 0, '<': 81, '<?xml version=': 0, '>': 81, 'https://': 3, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '213090', 'license': 'MIT', 'n_tokens_mistral': 5804, 'n_tokens_neox': 5425, 'n_words': 2605, 'path': 'notebooks/L01_Hello_World/hello_world.ipynb', 'repo_name': 'stephenbaek/dlcourse', 'size': '19022'}