<jupyter_start><jupyter_text>Lab 02 - "Fundamentals of Python Programming"Chartered Financial Data Scientist (CFDS), Autumn Term 2020 The lab environment of the **"Chartered Financial Data Scientist (CFDS)"** course is powered by Jupyter Notebooks (https://jupyter.org), which allows one to perform a great deal of data analysis and statistical validation. In this second lab, we want to have a look at the basic data types, containers, decision structures, loops and functions of the Python programming language.  The second lab (**"Halloween Edition"**) builds in parts on the excellent Python tutorial of the Stanford University CS 231n lecture series developed by <NAME>, <NAME> and <NAME>. The original tutorial is available under the following URL: http://cs231n.github.io/python-numpy-tutorial/.  As always, pls. don't hesitate to ask all your questions either during the lab, post them in our NextThought lab discussion forum (https://financial-data-science.nextthought.io), or send us an email (using our fds.ai email addresses).  Lab Objectives: After today's lab you should be able to:    > 1. Understand the basic **data types** of the Python e.g. integer, boolean, string.> 2. Understand the basic **data containers** of Python e.g. lists and dictionaries.> 3. Have a first idea of **numpy arrays** and **tensors**, a concept that we will use a lot in machine learning.> 4. Know Python's **decision structures** to guide the workflow of a program.> 5. Understand how to **loop** over data containers in order to access and manipulate individual values.> 6. Implement small **functions** that allow for the execution of several Python statements.  Let's start with a motivational video:<jupyter_code>from IPython.display import YouTubeVideo
# Powered by TensorFlow: helping paleographers transcribe medieval text using machine learning
YouTubeVideo('v-FgOACRgfs', width=1000, height=600)<jupyter_output><empty_output><jupyter_text>1. Python Versions There are two different versions of Python, 2.x and 3.x, the latter of which has some backward-compatibility issues. However, as of January 1, 2020, Python 2.x is no longer supported so you should not use it. Thus, for this class, all code will use Python 3.x (were x refers to an arbitrary version number). You may want to check your Python version at the command line by running `python --version`.  2. Fundamental Python Data Types In general, a data type defines the format, sets the upper and lower bounds of the data so that a program could use it appropriately. However, **Python data types** are just more than that. In Python, we don’t need to declare a variable with explicitly mentioning the data type. This feature is famously known as **dynamic typing**. Python determines the type of a literal (an element of code that has a fixed value) directly from the syntax at **runtime**. For example, the quotes mark the declaration of a string value, square brackets represent a list and curly brackets for a dictionary. Also, the non-decimal numbers will get assigned to Integer type whereas the ones with a decimal point will be a float. There are four basic data types in the Python programming language:> * **Integer's** - represent positive or negative whole numbers with no decimal point.> * **Float's** - represent positive or negative real numbers and are written with a decimal point.> * **String's** - represent sequences of Unicode characters.> * **Boolean's** - represent constant objects that are either 'False' and 'True'.   (Source: https://www.tes.com/teaching-resource/python-version-3-data-types-11949410.)   2.1 Numerical Data Type "Integer" Numbers are one of the most prominent Python data types. Numbers in Python are often called just **'integers'** or **'ints'** and are positive or negative whole numbers with no decimal point. In Python 3, there is effectively no limit to how big an integer value can be.Of course, it is constrained by the **amount of memory** your system has, as are all things. But beyond that, an integer can be as long as you need it to be:<jupyter_code>x = 3
x<jupyter_output><empty_output><jupyter_text>Print the variable type:<jupyter_code>type(x)<jupyter_output><empty_output><jupyter_text>Note that `x` still has the same value after these operations. This is due to the fact that we did not assign a new value to this variable. We can change its value by combining the two notations that we used above:<jupyter_code>print(x, 'original value')
x = x + 1
print(x, 'after adding 1')
x = x * 2
print(x, 'after multiplying with 2')
x = x - 5
print(x, 'after subtracting 5')
x = x ** 2
print(x, 'after exponentation with 2')<jupyter_output><empty_output><jupyter_text>Here are some shortcuts for basic operations combined with value assignment:<jupyter_code>x += 1
print(x)  # prints "4"
x *= 2
print(x)  # prints "8"<jupyter_output><empty_output><jupyter_text>2.2 Numerical Data Type "Float" The **float** type in Python represents real numbers and are written with a decimal point dividing the integer and fractional parts. As a result, float values are specified with a decimal point and are often called just **'floats'**.A float type number can have precision up to **15 decimal places**:<jupyter_code>y = 3.0
print(y)<jupyter_output><empty_output><jupyter_text>Print the variable type:<jupyter_code>type(y)<jupyter_output><empty_output><jupyter_text>Basic mathematical operations:<jupyter_code>print(y + 1)   # addition
print(y - 1)   # subtraction
print(y * 2)   # multiplication
print(y ** 2)  # exponentation<jupyter_output><empty_output><jupyter_text>Optionally, the character e or E followed by a positive or negative integer may be appended to specify scientific notation:<jupyter_code>z = 1e-7      # equals 0.0000001
print(z)
print(z + 1)
print(z * 2)
print(z ** 2)<jupyter_output><empty_output><jupyter_text>2.3 Non-Numerical Data Type "String" A sequence of one or more characters enclosed within either single quotes **‘** or double quotes **”** is considered as a **string** in Python. Any letter, a number or a symbol could be a part of the string. All the characters between the opening delimiter and matching closing delimiter are part of the string:<jupyter_code>hello = 'hello'   # string literals can use single quotes
world = "world, my name is peterli"   # or double quotes; it does not matter.
print(hello)<jupyter_output><empty_output><jupyter_text>Print the variable type:<jupyter_code>type(hello)<jupyter_output><empty_output><jupyter_text>Print the length of each string in terms of a number of characters:<jupyter_code>print(len(hello))
print(len(world))<jupyter_output><empty_output><jupyter_text>Like the most programming languages, Python allows accessing individual characters of a string. But it also supports negative indexes. Index of `‘-1’` represents the last character of the String. Similarly using `‘-2’`, we can access the penultimate character of the string and so on:<jupyter_code>halloween = 'spooky'
halloween[0]   # get the first string character
print(halloween[-1])    # get the last string character
print(halloween[-2])    # get the penultimate character<jupyter_output><empty_output><jupyter_text>Concatenate two strings, e.g., to form a sentence:<jupyter_code>hw = hello + '_' + world + ' 12' # string concatenation
print(hw)  # prints "hello world, my name is <NAME>"<jupyter_output><empty_output><jupyter_text>Concatenate two strings in C/C notation (also allowed in Python):<jupyter_code>hw2 = ' %s %s %d' % (hello, world, 12)  # sprintf style string formatting
print(hw2)  # prints "hello world, my name is <NAME>"<jupyter_output><empty_output><jupyter_text>Concatenate two strings in Python3 notation:<jupyter_code>hw3 = '{} {} 12'.format(hello, world)
print(hw3) # prints "hello world, my name is <NAME>"<jupyter_output><empty_output><jupyter_text>String objects have a bunch of useful methods; for example:<jupyter_code>s = "hello"                     # init string variable
print(s.capitalize())           # capitalize a string; prints "Hello"
print(s.upper())                # convert a string to uppercase; prints "HELLO"
print(s.rjust(7))               # right-justify a string, padding with spaces; prints "  hello"
print(s.center(7))              # center a string, padding with spaces; prints " hello "
print(s.replace('l', '(ell)'))  # replace all instances of one substring with another;
                                # prints "he(ell)(ell)o"
print('  world '.strip())       # strip leading and trailing whitespace; prints "world"
print('abracadabra'.split('a')) # split string whenever the letter 'a' occurs<jupyter_output><empty_output><jupyter_text>2.4 Non-Numerical Data Type "boolean" Python 3 provides a Boolean data type. Objects of type boolean type may have one of two values, "True" or "False":<jupyter_code>a = True
b = False
print(a)
print(b)<jupyter_output><empty_output><jupyter_text>Print the variable type:<jupyter_code>type(a)<jupyter_output><empty_output><jupyter_text>Booleans are often used in Python to test conditions or constraints. For example, a string in Python can be tested for truth value. The return type will be then a Boolean value (True or False). Let’s have a look at a few examples:<jupyter_code>s1 = '<NAME>'
result_upper = s1.isupper()        # test if string contains only upper case characters
print(result_upper)<jupyter_output><empty_output><jupyter_text>Let's have a look at a couple more examples:<jupyter_code>s2 = '<NAME>'
result_upper = s2.isupper()        # test if string contains only upper case characters
print(result_upper)
n1 = 10
result_greather_than = n1 > 100                 # test if 10 > 100
print(result_greather_than)
n2 = 99
result_in_between = n2 > 10 and n2 < 100        # test if 99 > 10 and 99 < 100
print(result_in_between)<jupyter_output><empty_output><jupyter_text>We can even logically combine the tested conditions above:<jupyter_code>print(a and b)               # Logical AND; prints "False"
print(a or b)                # Logical OR; prints "True"
print(a and result_upper)    # Logical AND; prints "True"
print(not a)                 # Logical NOT; prints "False"
print(a != b)                # Logical XOR; prints "True"<jupyter_output><empty_output><jupyter_text>As you will see in upcoming labs, expressions in Python are often evaluated in a boolean context, meaning they are interpreted to represent truth or falsehood. A value that is true in Boolean context is sometimes said to be “truthy,” and one that is false in Boolean context is said to be “falsy”.  Exercises: We recommend you to try the following exercises as part of the lab:**1. Write a set of (or single) Python command(s) that compare the first and last character of a string.**> Write a set of Python commands that compare the first and last character of a string. In case both characters are the same print 'True' otherwise print 'False'. Test your statements using the words 'spooky' and 'sweets'.<jupyter_code># ***************************************************
# INSERT YOUR SOLUTION CODE HERE
# ***************************************************<jupyter_output><empty_output><jupyter_text>**2. Write a set of (or single) Python command(s) that determine the properties of a string.**> Write a set of Python commands that determine the number of characters of a string and if the characters are all upper case. If the number of characters is between 5 and 12 characters and all upper case print 'True' otherwise print 'False'. Test your statements using the words 'BROOMSTICK', 'ghostly', and 'mYstEriOUs'.<jupyter_code># ***************************************************
# INSERT YOUR SOLUTION CODE HERE
# ***************************************************<jupyter_output><empty_output><jupyter_text>**3. Write a set of (or single) Python command(s) that prints a very scary sentence.**> Write a set of Python commands that prints the scariest sentence that you could imagine. The sentence should include at least 3 of the following words 'tarantula', 'moonlight', 'supernatural', 'fog', 'owl', 'nightmare', or 'poltergeist'.<jupyter_code># ***************************************************
# INSERT YOUR SOLUTION CODE HERE
# ***************************************************<jupyter_output><empty_output><jupyter_text>3. Fundamental Python Data Containers There are four collection data types in the Python programming language:> * **List** - is a collection which is ordered and changeable. Allows duplicate members.> * **Tuple** - is a collection which is ordered and unchangeable. Allows duplicate members.> * **Set** - is a collection which is unordered and unindexed. No duplicate members.> * **Dictionary** - is a collection which is unordered, changeable and indexed. No duplicate members.When choosing a collection type, it is useful to understand the properties of that type. Choosing the right type for a particular data set could mean retention of meaning, and, it could mean an increase in efficiency or security. During this lab, we will have a closer look into **lists** and **dictionaries**.  (Source: https://www.tes.com/teaching-resource/python-version-3-data-types-11949410.)   3.1. Data Container "List" A list is a collection of basic Python data types "elements" which is ordered and changeable.  In Python, lists are written with square brackets (equivalent to an array). Python lists allow duplicate elements, are resizeable and can contain elements of different data types. Lists can be used like this:<jupyter_code>spooky_list = [42, 5, 128, 5, 97, 208]    # create a list
print(spooky_list)                        # print list<jupyter_output><empty_output><jupyter_text>Print the variable type:<jupyter_code>type(spooky_list)<jupyter_output><empty_output><jupyter_text>Determine individual elements of a list:<jupyter_code>print(spooky_list[2])           # print third element of list created
print(spooky_list[-1])          # print the last list element<jupyter_output><empty_output><jupyter_text>**Note**: Python list indices start with 0. That means that the first list element has the index 0, the second element has the index 1, and so on...Determine the total number of list elements:<jupyter_code>print(len(spooky_list))         # print the number of elements contained in the list<jupyter_output><empty_output><jupyter_text>Replace a list element by assigning a new value at a specific index:<jupyter_code>spooky_list[2] = 'happy'        # lists can contain elements of different types
print(spooky_list)<jupyter_output><empty_output><jupyter_text>Append an element to the end of a list:<jupyter_code>spooky_list.append('coding')    # add a new element to the end of the list
print(spooky_list)<jupyter_output><empty_output><jupyter_text>Remove the last element of a list:<jupyter_code>this_last = spooky_list.pop()   # remove and return the last element of the list
print(this_last)                # prints the element removed 
print(spooky_list)              # prints the remaining elements<jupyter_output><empty_output><jupyter_text>Create a list of numbers:<jupyter_code>list_of_scary_numbers = list(range(5))     # range is a built-in function that creates a list of integers
print(list_of_scary_numbers)               # prints "[0, 1, 2, 3, 4]"<jupyter_output><empty_output><jupyter_text>Slice list using distinct indexing techniques:<jupyter_code>print(list_of_scary_numbers[2:4])    # get a slice from index 2 to 4 (exclusive); prints "[2, 3]"
print(list_of_scary_numbers[2:])     # get a slice from index 2 to the end; prints "[2, 3, 4]"
print(list_of_scary_numbers[:2])     # get a slice from the start to index 2 (exclusive); prints "[0, 1]"
print(list_of_scary_numbers[:])      # get a slice of the whole list; prints ["0, 1, 2, 3, 4]"
print(list_of_scary_numbers[:-1])    # slice indices can be negative; prints ["0, 1, 2, 3]"<jupyter_output><empty_output><jupyter_text>Replace range of list elements:<jupyter_code>list_of_scary_numbers[2:4] = [8, 9]  # assign a new sublist to a slice
print(list_of_scary_numbers)         # prints "[0, 1, 8, 9, 4]"<jupyter_output><empty_output><jupyter_text>3.2. Data Container "Dictionary" In Python, dictionaries are used to store (key, value) pairs.  A dictionary is a collection of basic Python data types "elements" which is unordered, changeable and indexed. In Python, dictionaries are written with curly brackets. Dictionaries can be used like this:<jupyter_code>this_screaming_dictionary = {'cat': 'animal', 'night': 'non-animal', 'vampire': 'non-animal', 'owl': 'animal'}  # create a new dictionary
print(this_screaming_dictionary)<jupyter_output><empty_output><jupyter_text>Retrieve and print the value corresponding to the key "cat":<jupyter_code>print(this_screaming_dictionary['cat'])          # get an entry from a dictionary; prints "spooky"<jupyter_output><empty_output><jupyter_text>Add a new dictionary element:<jupyter_code>this_screaming_dictionary['pumpkin'] = 'ugly'    # set an entry in a dictionary
print(this_screaming_dictionary)<jupyter_output><empty_output><jupyter_text>Retrieve and print the value corresponding to the added entry with key "pumpkin":<jupyter_code>print(this_screaming_dictionary['pumpkin'])          # get an entry from a dictionary; prints "ugly"<jupyter_output><empty_output><jupyter_text>Retrieve and print all dictionary keys:<jupyter_code>keys = this_screaming_dictionary.keys()          # obtain all dictionary keys
print(keys)<jupyter_output><empty_output><jupyter_text>Retrieve and print all dictionary values:<jupyter_code>values = this_screaming_dictionary.values()      # obtain all dictionary values
print(values)<jupyter_output><empty_output><jupyter_text>Try to retrieve a dictionary value that is not contained in the dictionary (this will result in an error):<jupyter_code>#print(this_screaming_dictionary['ghost'])                  # KeyError: 'ghost' not a key of the dictionary<jupyter_output><empty_output><jupyter_text>However, we can "catch" such errors using:<jupyter_code>print(this_screaming_dictionary.get('ghost', 'N/A'))       # get an element with a default; prints "N/A"
print(this_screaming_dictionary.get('pumpkin', 'N/A'))     # get an element with a default; prints "ugly")<jupyter_output><empty_output><jupyter_text>Remove an element from a dictionary:<jupyter_code>del this_screaming_dictionary['pumpkin']                   # remove an element from a dictionary
print(this_screaming_dictionary)<jupyter_output><empty_output><jupyter_text>Try to retrieve the removed dictionary element:<jupyter_code>print(this_screaming_dictionary.get('pumpkin', 'N/A'))    # "pumpkin" is no longer a key; prints "N/A"<jupyter_output><empty_output><jupyter_text>Exercises: We recommend you to try the following exercises as part of the lab:**1. Write a set of (or single) Python command(s) that determine the number of characters of a list element.**> Write a set of Python commands that determine the number of characters of the second element of a list. In case the element consists of more than 4 characters print 'True' otherwise print 'False'. Test your statements using the following lists `['angel', 'nightmare', 'poltergeist']` and `['darkness', 'king', 'fairy', 'owl']`.<jupyter_code># ***************************************************
# INSERT YOUR SOLUTION CODE HERE
# ***************************************************<jupyter_output><empty_output><jupyter_text>**2. Write a set of (or single) Python command(s) that compares the elements of a list.**> Write a set of Python commands that compares the first and last elements of a list. In case both elements consist of the same characters print 'True' otherwise print 'False'. Test your statements using the following list `['BROOMSTICK', 'ghostly', 'mYstEriOUs', 'BROOMSTICK']` and `['darkness', 'king', 'fairy', 'owl']`.<jupyter_code># ***************************************************
# INSERT YOUR SOLUTION CODE HERE
# ***************************************************<jupyter_output><empty_output><jupyter_text>**3. Write a set of (or single) Python command(s) that removes elements of a list.**> Write a set of Python commands to print a specified list after removing the 0th, 2th, 3th, and 5th element. Test your statements using the following list `['BROOMSTICK', 'Happy', 'mYstEriOUs', 'BROOMSTICK', 'Halloween', 'Poltergeist']`.<jupyter_code># ***************************************************
# INSERT YOUR SOLUTION CODE HERE
# ***************************************************<jupyter_output><empty_output><jupyter_text>4. Numpy Arrays and PyTorch TensorsLists are useful, but rather memory intensive. If you want to process large amounts of data - like we do in machine learning applications - you want to use your limited memory more efficiently. [NumPy](https://numpy.org/) is here to help you. NumPy is the fundamental package for scientific computing with Python. It implements the concept of arrays, which are lists on steroids. Let's see what they can do.Creating an array is as easy as creating a list:<jupyter_code>import numpy as np  # we have to import NumPy first

l = [0, 1, 2]   # this is a list
a = np.array(l) # this is an array
a<jupyter_output><empty_output><jupyter_text>You can create an array from a list, or you can use the NumPy equivalent of `range`:<jupyter_code>a = np.arange(0, 3, 1)
a<jupyter_output><empty_output><jupyter_text>The first big difference between lists and arrays is that all elements in an array **must** be of the same data type (list elements can have different data types, though). As a result, an array has a data type attribute:<jupyter_code>a.dtype<jupyter_output><empty_output><jupyter_text>In this case, the data type is 64-bit integer, which is a derivate of the integer we already met as a basic data type. Arrays are able to perform element-wise operations:<jupyter_code>print(a + 1)  # element-wise addition
print(a - 4)  # element-wise subtraction
print(a * 2)  # element-wise multiplication
print(a ** 2)  # element-wise exponentiation
#print(l+1)   # a list cannot do this<jupyter_output><empty_output><jupyter_text>Some operations may require your array to adopt a different datatype. Python will do that for you:<jupyter_code>print(a/2)
print((a/2).dtype)<jupyter_output><empty_output><jupyter_text>Indexing and slicing work the exact same way as for lists:<jupyter_code>print(a[0])
print(a[:2])
print(a[-1])<jupyter_output><empty_output><jupyter_text>You can also use arithmetic operations involving two arrays:<jupyter_code>print(a + np.array([1, 2, 3]))
#print(l + [1, 2, 3])  # in the case of lists, this will concatenate the two lists<jupyter_output><empty_output><jupyter_text>You can consider a one-dimensional array as a **vector**. But arrays are not limited to one dimension. You can create a two-dimensional array, which you can then consider as a **matrix**:<jupyter_code>m = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
m<jupyter_output><empty_output><jupyter_text>Everything we learned about arrays so far holds up for two-dimensional arrays:<jupyter_code>print(m[0])       # the first element of m is a one-dimensional array
print(m[:, 0])    # the first element of each row
print(m + 1)      # element-wise addition
print(m * a)      # multiply vector a with each row of matrix m<jupyter_output><empty_output><jupyter_text>Please note the notation in line 2: `m[:, 0]`This might look weird to you, but it's easy to explain what happens here: in a two-dimensional array, you have one index for each dimension. In `[:, 0]`, `:` refers to the first dimension (the rows) and `0` refers to the second dimension (the columns). Thus, it retrieves the element in the first column for each row. If you have additional (n) dimensions, the same logic applies, which gives you a powerful tool to easily access data in your arrays.Since `a` is a vector and `m` is a matrix, we can compute the dot product (or *inner product*) of the two:<jupyter_code>np.dot(m, a)<jupyter_output><empty_output><jupyter_text>More details on what you can do with arrays will follow in a later lab. Just two more things to mention:    Why are arrays useful? Because they are fast. Let me show you just how fast. Let's assume you have a long list and want to change every element in that list. This will take a while:<jupyter_code>l = range(10000000)
l2 = []
for x in l:
    l2.append(x+1)
l2<jupyter_output><empty_output><jupyter_text>If you try the same in NumPy, you can do it in fewer lines of code and with much faster runtime:<jupyter_code>a = np.arange(10000000)
a+1<jupyter_output><empty_output><jupyter_text>NumPy is so much fast since it vectorizes computations: it applies the operation on each element of an array at the same time. This is possible since NumPy uses C implementations in the background. You, as a user, benefit from the speed of C and the simplicity of Python at the same time. Finally, in the next lab courses we will be dealing with tensors. A **tensor**, mathematically, is just an n-dimensional array. In terms of programming, we will be using the Python module `pytorch`, which implements tensors (NumPy arrays can be used as tensors, too, but we prefer to use pytorch tensors for a reason that we will explain later to you). The good thing for you is that pytorch tensors work exactly like NumPy arrays:<jupyter_code>import torch

a = torch.tensor([1, 2, 3])

print(a + 1)  # element-wise addition
print(a * 2)  # element-wise multiplication
print(a[2])   # indexing
print(a[1:])  # slicing

b = torch.tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9]])

print(torch.mul(b, a))  # dot product<jupyter_output><empty_output><jupyter_text>Exercises: **1. Write a set of (or single) Python command(s) that adds a constant to every element of an array and a list.**> Write a set of Python commands to increase the value of each element of array `a` by 3. Do the same for list `l`.<jupyter_code># ***************************************************
# INSERT YOUR SOLUTION CODE HERE
# ***************************************************<jupyter_output><empty_output><jupyter_text>**2. Write a single Python command that creates a 2d-array that looks like the following matrix:**    1 0 0     0 1 0    0 0 1    > Hint: you can use `np.array(l)` where `l` is a list to create the matrix.<jupyter_code># ***************************************************
# INSERT YOUR SOLUTION CODE HERE
# ***************************************************<jupyter_output><empty_output><jupyter_text>**3. Write a single Python command that turns array `a` into [5, 5, 5, 5].**> Apply a mathematical operation to bring `a` into the desired form.<jupyter_code># ***************************************************
# INSERT YOUR SOLUTION CODE HERE
# ***************************************************<jupyter_output><empty_output><jupyter_text>5. Fundamental Programming Structures As part of this lab, we want to have a closer look at three basic programming structures of Python:> * **For-Loops** - used to iterate over a sequence of program statements.> * **Decision Structures** - used to anticipate conditions occurring while the execution of a program.> * **Functions** - used to define a block of code which only runs when it is called.  5.1 Python Loop Structures To keep a program doing some useful work we need some kind of repetition, looping back over the same block of code again and again. Below we will describe the different kinds of loops available in Python.  5.1.1. The "For"-Loop and Lists The for loop that is used to iterate over elements of a sequence, it is often used when you have a piece of code which you want to repeat a specifc "n" number of times. The nature of a for-loop in Python is very straightforward  **"for all elements in a list, do this".**  Let's say that you have a list; you can then loop over the list elements using the `for` keyword like this:<jupyter_code># list initialization
halloween_elements = ['cat', 'night', 'pumpkin']

# loop initialization and run
for anyname in halloween_elements: 
    print(anyname)<jupyter_output><empty_output><jupyter_text>**Note:** Python relies on the concept of indentation, using whitespace (or tabs), to define the scope in the code. Other programming languages such as Java or C often use brackets or curly-brackets for this purpose. Let's have a look at another example of a for-loop:<jupyter_code># init a list of numbers
numbers = [1, 10, 20, 30, 40, 50]

# init the result
result = 0

# loop initialization and run
for number in numbers:
    result = result + number
    # result += number

# print the result
print(result)<jupyter_output><empty_output><jupyter_text>To loop over a list of numbers, we can use Python's `range` function. The `range(lower_bound, upper_bound, step_size)` function generates a sequence of numbers, starting from the `lower_bound` to the `upper_bound`. The `lower_bound` and `step_size` parameters are optional. By default the lower bound is set to zero, the incremental step is set to one.<jupyter_code># loop over range elements
for i in range(1, 10):
    
    # print current value of i
    print(i)<jupyter_output><empty_output><jupyter_text>To break out from a loop, you can use the keyword `break`. Let's have a look at the following example:<jupyter_code># loop over range elements
for i in range(1, 10000000000):
    
    # case: current value of i equals 3?
    if i == 3:
        
        # break: stop the loop
        break 
        
    # print current value of i
    print(i)<jupyter_output><empty_output><jupyter_text>In contrast, the `continue` keyword is used to tell Python to skip the rest of the statements in the current loop block and to continue to the next iteration of the loop.<jupyter_code># loop over range elements
for i in range(1, 10):
    
    # case: current value of i equals 3?
    if i == 3:
        
        # continue: jump to next loop iteration 
        continue
    
    # print current value if i
    print(i)<jupyter_output><empty_output><jupyter_text>If you want access to the index of each element within the body of a loop, use the built-in enumerate function:<jupyter_code>halloween_elements = ['cat', 'night', 'pumpkin']
for idx, element in enumerate(halloween_elements):
    print('#%d: %s' % (idx + 1, element))<jupyter_output><empty_output><jupyter_text>When programming frequently, we want to transform one type of data into another. As a simple example, consider the following code that computes square numbers:<jupyter_code>nums = [0, 1, 2, 3, 4]
squares = []
for x in nums:
    squares.append(x ** 2)
print(squares)<jupyter_output><empty_output><jupyter_text>You can make this code simpler using a list comprehension:<jupyter_code>nums = [0, 1, 2, 3, 4]
squares = [x ** 2 for x in nums]
print(squares)<jupyter_output><empty_output><jupyter_text>List comprehensions can also contain conditions:<jupyter_code>nums = [0, 1, 2, 3, 4]
even_squares = [x ** 2 for x in nums if x % 2 == 0]
print(even_squares)<jupyter_output><empty_output><jupyter_text>5.1.2. The "For"-Loop and Dictionaries Similarly, it is easy to iterate over the keys in a dictionary:<jupyter_code>d = {'pumpkin': 0, 'cat': 4, 'spider': 8}
for animal in d:
    legs = d[animal]
    print('A %s has %d legs' % (animal, legs))<jupyter_output><empty_output><jupyter_text>If you want access to keys and their corresponding values, use the items method:<jupyter_code>d = {'pumpkin': 2, 'cat': 4, 'spider': 8}
for animal, legs in d.items():
    print('A %s has %d legs' % (animal, legs))<jupyter_output><empty_output><jupyter_text>Exercises: We recommend you to try the following exercises as part of the lab:**1. Write a Python loop that multiplies all elements of a list with 10.**> Write a Python loop that multiplies all elements of a list with `66`. The input list is given by `range(0, 10)` and its output should result in a list as denoted by: `[0, 66, 132, 198, 264, 330, 396, 462, 528, 594]`.<jupyter_code># ***************************************************
# INSERT YOUR SOLUTION CODE HERE
# ***************************************************<jupyter_output><empty_output><jupyter_text>**2. Write a Python loop that prints the numbers 1 to 10 backwards.**> Write a Python loop that prints the numbers 0 to 10 backwards. The output of the loop should print the following: `10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0`.<jupyter_code># ***************************************************
# INSERT YOUR SOLUTION CODE HERE
# ***************************************************<jupyter_output><empty_output><jupyter_text>5.2 Python Decision Structures Decision structures evaluate multiple expressions which produce **True** or **False** as an outcome. When solving a data science task, you often need to determine which action to take and which statements to execute if an outcome is **True** or **False** otherwise.  Let's briefly recap Python's use of logical or mathematical conditions:<jupyter_code># init sample variables
a = 4
b = 7
print(a == b)  # equals
print(a != b)  # not equals
print(a < b)   # less than
print(a <= b)  # less than or equal to 
print(a > b)   # greater than
print(a >= b)  # greater than or equal to<jupyter_output><empty_output><jupyter_text>The mathematical conditions outlined above can be used in several ways, most commonly in if-statements. An if-statement is written by using the `if` keyword. Let's have a look at an example:<jupyter_code># init sample variables
a = 4
b = 7

# test condition
if b > a:
    print("b is greater than a")<jupyter_output><empty_output><jupyter_text>In the example above we used two variables, `a` and `b`, which are used as part of the if-statement to test whether `b` is greater than `a`. As a is 4, and b is 7, we know that 7 is greater than 4, and so we print that "b is greater than a". We can easily enhance the if-statement above by adding additional conditions using the `elif` keyword. The `elif` keyword is pythons way of saying "if the previous conditions were not true, then try this condition":<jupyter_code>a = 4
b = 4

# test condition 1
if b > a:
  print("b is greater than a")

# test condition 2
elif a == b:
  print("a and b are equal")

elif a != b:
    print("test check and so on... ")<jupyter_output><empty_output><jupyter_text>Finally, we can use the `else` keyword to catch any case which isn't found by the preceding conditions:<jupyter_code>a = 8
b = 4

# test condition 1
if b > a:
  print("b is greater than a")

# test condition 2
elif a == b:
  print("a and b are equal")

# all other cases
else:
  print("a is greater than b")<jupyter_output><empty_output><jupyter_text>In the example above the value assigned to a variable `a` is greater than the value assigned to `b`, so the first `if` condition is not true. Also, the `elif` condition is not true, so we ultimately go to the `else` condition and print that "a is greater than b".  Exercises: We recommend you to try the following exercises as part of the lab:**1. Write a Python decision structure that prints all the numbers from 0 to 6 except 4 and 7.**> Write a Python decision structure that prints a number if it doesn't equal to 4 and 7. If the number equals 4 or 7 it should print 'forbidden number'.<jupyter_code># ***************************************************
# INSERT YOUR SOLUTION CODE HERE
# ***************************************************<jupyter_output><empty_output><jupyter_text>**2. Write a Python decision structure that evaluates if a number is a multiple of 5 and 7.**> Write a Python decision structure that evaluates if number is a multiple of 5 and 7. Hint: You may want to use Python's percentage `%` operator as part of your case evaluation.<jupyter_code># ***************************************************
# INSERT YOUR SOLUTION CODE HERE
# ***************************************************<jupyter_output><empty_output><jupyter_text>Dictionary comprehensions: These are similar to list comprehensions, but allow you to easily construct dictionaries. For example:<jupyter_code>nums = [0, 1, 2, 3, 4]
even_num_to_square = {x: x ** 2 for x in nums if x % 2 == 0}
print(even_num_to_square)<jupyter_output><empty_output><jupyter_text>5.3 Python Functions  A function is a block of organized, reusable code that is used to perform a single, related action. Functions provide better modularity for your application and allow for a high degree of reusable code. As you already saw, Python provides you with many built-in functions such as `print()`, etc. but you can also create your functions. These functions are called **user-defined functions**.A function is a block of code which only runs when it is called. You can pass data, known as parameters, into a function. A function can return data as a result. Python functions are defined using the `def` keyword.  (Source: https://swcarpentry.github.io/python-novice-inflammation/06-func/index.html.) Let's define our first function that takes a string as input parameter and prints it:<jupyter_code># defines a printme function
def print_me(characters):
    
   # this prints a passed string into this function
   print(characters)<jupyter_output><empty_output><jupyter_text>Now, we can call our newly defined function using the function name followed by the arguments that we aim to pass in parenthesis:<jupyter_code>print_me(characters="I'm first call to user defined function!")
print_me(characters="Again second call to the same function")<jupyter_output><empty_output><jupyter_text>Isn't that fantastic?  Now that we understood the syntax to create customized functions, we can create even more complex functions. Let's implement a function that determines if a given integer number is positive, negative or zero using a decision structure and prints the result accordingly:<jupyter_code># defines a sign evaluation function
def sign(x):
    
    # case: positive value
    if x > 0:
        
        # return the string 'positive'
        return 'positive'
    
    # case: negative value
    elif x < 0:
        
        # return the string 'negative'
        return 'negative'
    
    # else: other value
    else:
        
        # return the string 'zero'
        return 'zero'<jupyter_output><empty_output><jupyter_text>Now we call our function and print the result of sign evaluation for distinct values:<jupyter_code>print(sign(x=-1))
print(sign(x=0))
print(sign(x=1))<jupyter_output><empty_output><jupyter_text>We will often define functions to take optional keyword arguments. An optional argument is an argument that assumes a default value if a value is not provided in the function call for that argument. The following example provides an idea on default arguments, it prints the characters given in upper case if not specified differently, like this:<jupyter_code>def hello(characters, loud=1633):
    
    # case: default - loud print enabled
    if loud:
        print('HELLO, %s' % characters.upper())
        
    # case: non-loud print enabled
    else:
        print('Hello, %s!' % characters)
hello(characters='Helloween', loud=1000)
hello(characters='Helloween', loud=False)<jupyter_output><empty_output><jupyter_text>Exercises: We recommend you to try the following exercises as part of the lab:**1. Write a Python function to calculate the length of a string.**>Write a Python function named **"string_length"** to calculate the length of an arbitrary string. The function should take an arbitrary string as input and count the number of its characters. Test your function accordingly using various string values and print the results, e.g., input: 'Halloween', expected result: 9.<jupyter_code># ***************************************************
# INSERT YOUR SOLUTION CODE HERE
# ***************************************************<jupyter_output><empty_output><jupyter_text>**2. Write a Python program to get the largest number from a list.**>Write a Python function named **"max_num_in_list"** to get the largest number from a list. The function should take an arbitrary list of integer values as an input and should return the integer that corresponds to the highest value. Test your function accordingly using various string values and print the results, e.g., input: [1, 5, 8, 3], expected result: 8.<jupyter_code># ***************************************************
# INSERT YOUR SOLUTION CODE HERE
# ***************************************************<jupyter_output><empty_output><jupyter_text>**3. Write a Python program to count the number of characters (character frequency) in a string.**>Write a Python function named **"char_frequency"** to count the number of distinct characters occurring in it. The function should take an arbitrary string as an input and should return the count of occurrence each individual character. Test your function accordingly using various string values and print the results, e.g., input: 'Happy Halllllloweeeeeen!', expected result: {'a': 2, ' ': 1, 'e': 6, 'H': 2, 'l': 6, 'o': 1, 'n': 1, 'p': 2, '!': 1, 'w': 1, 'y': 1}.<jupyter_code># ***************************************************
# INSERT YOUR SOLUTION CODE HERE
# ***************************************************<jupyter_output><empty_output><jupyter_text>**Bonus: Write a Python function that takes a list of words and returns the one exhibiting the most characters.**>Write a Python function named **find_longest_word** that takes a list of words and returns the length of the longest word in the list. The function should take an arbitrary list of string values (words) as an input and should return the word that exhibits the most characters. Test your function accordingly using various lists of string values and print the results, e.g., input: ['Happy', 'Halloween', '2018'], expected result: 'Halloween'.<jupyter_code># ***************************************************
# INSERT YOUR SOLUTION CODE HERE
# ***************************************************<jupyter_output><empty_output><jupyter_text>[BONUS] 6. Basic Python Image Manipulations The **"Python Imaging Library (Pillow)"** by <NAME> and contributors (abbreviated as PIL, in newer versions known as Pillow), is a free library for the Python programming language that adds support for opening, manipulating, and saving many different image file formats (https://pillow.readthedocs.io/en/stable/). Using the PIL library in combination with the Numpy library allows for a great deal image analysis and manipulation. To make an arbitrary image interpretable for a computer images are usually represented by 3D tensors as shown below:  (Source: **"Learning Perceptual Organization with Minimal Human Supervision"**, <NAME>, UC Berkeley, 2019.)  Pls. note, that prior to using PIL you need to install the Python Image package. A brief installation instruction can be obtained from: https://pillow.readthedocs.io/en/5.3.x/installation.html. The package can installed via the following command:<jupyter_code>!pip3 install pillow<jupyter_output><empty_output><jupyter_text>Upon successful installation we can now import the `PIL` and `Scikit-Image` libraries:<jupyter_code># use this statement to show plots inside your notebook
%matplotlib inline

# import the pil, numpy and the matplotlib libraries
from PIL import Image
import numpy as np
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>The `PIL` library supports a wide variety of image file formats. To read files from disk, use the image read `imread` function of the `Scikit-Image` library. You don’t have to know the file format to open a file. The library automatically determines the format based on the contents of the file.  6.1. Image Loading and Plotting Let's open and plot the `halloween.jpg` image file:<jupyter_code># import the requests and bytesIO libraries
import requests
from io import BytesIO

# set URL of the image to be loaded 
url = "https://raw.githubusercontent.com/financial-data-science/CFDS-Notebooks/master/lab_02/halloween.jpg"

# send load request to URL
response = requests.get(url)

# parse and convert image data to numpy array
image = np.asarray(Image.open(BytesIO(response.content)))

# plot image in Jupyter notebook
plt.imshow(image, vmin=0, vmax=255)<jupyter_output><empty_output><jupyter_text>Let's also have a look at the exact shape of the image:<jupyter_code>h, w, c = image.shape

print(h)   # prints the image height
print(w)   # prints the image width
print(c)   # prints the number of image color channels = 3 for red, green, blue (rgb)<jupyter_output><empty_output><jupyter_text>6.2. Image Cropping Use PIL to extract the upper pumpkin of the image:<jupyter_code>upper_pumpkin = image[20:250, 20:300, :]
plt.imshow(upper_pumpkin, vmin=0, vmax=255)
upper_pumpkin.shape<jupyter_output><empty_output><jupyter_text>6.3. Image Channel Extraction Let's extract the distinct (RGB) colour channels of the image:<jupyter_code># extraction of the red colour channel
red_channel = np.zeros(upper_pumpkin.shape, dtype='uint8')
red_channel[:,:,0] = upper_pumpkin[:,:,0]

plt.imshow(red_channel)
# extraction of the green colour channel
green_channel = np.zeros(upper_pumpkin.shape, dtype='uint8')
green_channel[:,:,1] = upper_pumpkin[:,:,1]

plt.imshow(green_channel)
# extraction of a blue colour channel
blue_channel = np.zeros(upper_pumpkin.shape, dtype='uint8')
blue_channel[:,:,2] = upper_pumpkin[:,:,2]

plt.imshow(blue_channel)<jupyter_output><empty_output><jupyter_text>6.4. Image Manipulation Greyscale the image by setting the third image channel to mean of all colour values:<jupyter_code>gray = upper_pumpkin.mean(axis=2)
plot = plt.imshow(gray, cmap=plt.cm.gray, vmin=0, vmax=255)<jupyter_output><empty_output><jupyter_text>To learn more about PIL and its capabilities visit: https://pillow.readthedocs.io.  Lab Summary: In this second lab, the basic data types and containers of the Python programming language are presented. The code and exercises presented in this lab may serve as a starting point for more complex and tailored analytics.  You may want to execute the content of your lab outside of the Jupyter notebook environment, e.g. on a compute node or a server. The cell below converts the lab notebook into a standalone and executable python script. Pls. note, that to convert the notebook you need to install Python's **nbconvert** library and its extensions:<jupyter_code># installing the nbconvert library
!pip install nbconvert
!pip install jupyter_contrib_nbextensions<jupyter_output><empty_output><jupyter_text>Convert the Jupyter notebook into a plain Python script:<jupyter_code>!jupyter nbconvert --to script cfds_lab_02.ipynb<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1221646492227645, 'fraction_numerical': 0.012556169508288362, 'mean_word_length': 5.210709039925224, 'pattern_counts': {'":': 4, '<': 424, '<?xml version=': 0, '>': 463, 'https://': 10, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 3, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '102', 'dir': 'jupyter-structured-clean-dedup', 'id': '36135', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 13963, 'n_tokens_neox': 13143, 'n_words': 5669, 'path': 'lab_02/cfds_colab_02.ipynb', 'repo_name': 'aarimond/CFDS-Notebooks', 'size': '46567'}