<jupyter_start><jupyter_text>Contents (TODO) | [How to Read and Represent Data](../ica02/How_to_Read_and_Represent_Data.ipynb) >  Hello World!This notebook is designed to provide a quick and dirty introduction to Python programming. If you are already familiar with Python programming, you can safely skip many of the sections below and jump directly to assignment cells. This notebook is designed for complete newbies (=absolutely zero experience with Python) to help them quickly dive into the course materials for the rest of the course. This being said, it is NOT a purpose of this notebook to equip you with a comprehensive working knowledge and coding skills on Python. For more structured learning materials for Python, I strongly encourage you to purchase a book, take a (online) course, etc.Also, I hereby acknowledge that many of the codes and explanations I used in this notebook are originally from [Python Data Science Handbook](http://shop.oreilly.com/product/0636920034919.do) by <NAME>. The original codes and contents are available on their [GitHub](https://github.com/jakevdp/PythonDataScienceHandbook).  1. Jupyter NotebookThis notebook that you are reading right now is what is called a Jupyter notebook. It's actually a pretty cool way of doing note keeping on your experiments, as it allows you to put nicely formatted notes along with source codes executable on the fly. To get started, click the code below and hit `ctrl + enter` (or if you're using Mac, `cmd + enter`) and see what happens.<jupyter_code>print('Hello World!')<jupyter_output><empty_output><jupyter_text>Did you notice that the notebook printed out a message `Hello World!`? Basically, a Jupyter notebook is comprised of "small blocks of things" called *cells*. The one that you just ran is a cell containing a code and it's called a *code cell*. `ctrl + enter` (or `cmd + enter` for Mac) is usually how you run a code cell. Alternatively, you can hit `shift + enter`, which in this case, executes the code cell and move the focus to the next cell.There's another type of cell, which is a *text cell* or a *markdown cell*. The very paragraph that you are reading right now is actually a text cell. Text cells are for making notes to your codes essentially, but there can be a lot of different ways of using them. Text cells ues a text scripting language called 'Markdown'. Markdown is actually pretty simple to learn and for most of you, looking at [this cheat-sheet](https://www.markdownguide.org/cheat-sheet) should be just enough. Below is an example of a text cell. Double click it and change some texts there. After changes are made, simply hit `ctrl/shift + enter` as what you would do to run a code cell. ---In a text cell, you can render texts in **boldface** or *italicized*.You can also ~~strikethrough~~ words.> You can also quote someone like this.1. or you can create2. an ordered3. list- or even- an unordered- listYou can render sample codes like this:```cppinclude int main(void){    std::cout << "Hello World!" << std::endl;    return 0;}```or, inline, `myVar = 3`.You can also create a [hyperlink](http://www.stephenbaek.com),or include an image ![iowa](https://lh5.googleusercontent.com/p/AF1QipOg3dTb5dkfYhENpjJeplro3cLVaNLiQvhMHL8i=w213-h160-k-no)You can report some results in a table:| Table | Column || ----- | ------ || Row 1 |  30 % || Row 2 |  70 % |You can also build a laundry list like this:- [ ] Do homework- [ ] Buy some cookies- [x] Steve: Prepare a lecture material- [ ] Liz: Practice presentation--- To add a cell, you can simply hit `+` button on a toolbar, or `Insert > Insert Cell Below (Above)` in the main menu. By default, it creates a code cell, but you can always convert it to a text cell by clicking `Cell > Cell Type > Markdown`. If you use Google Colab, creating cells is even simpler. You would just need to put a mouse cursor to where you want to create a cell and click either `CODE` or `TEXT` depending on the type of a cell you want to create. Remarks on the course materials...In this course, I'll make lab notes, assignments, source codes, etc. available as Jupyter notebooks. The best way to learn from the notebooks is to run everything in there, cell by cell, and carefully observe what happens. I will also assign quiz, homework, and exercises using these notebooks, by leaving some blank code cells and such:<jupyter_code># Exercise Problem (Assignment)
# Fill in the blank to print your name.
print(                )<jupyter_output><empty_output><jupyter_text>Report typos, bugs, grammatical errors, etc.Lastly, I have to admit that I'm an extremely unorganized, careless person. I can guarantee you that you will find lots of typos, grammatical mistakes, bugs, etc. in the lecture materials. You may also have some suggestions for improving this course, things like 'Can you add lab sessions on parallel computing?' 'Example in lecture-03 is boring. Can you use different dataset?' or things like that. Please let me know about those by opening issues in the course GitHub. To open an issue, you can visit https://github.com/stephenbaek/bigdata/issues and click `New issue` button. Your suggestions are always welcome and you won't be penalized for pointing out problems or making suggestions. In fact, for those who raise an issue that I find particularly useful for improving the course, I'll give you a bonus added to your final grade.  2. Data Types in PythonTo be a good data scientist, you will be required to have a firm grasp on how data is stored and manipulated.This section outlines and contrasts how arrays of data are handled in the Python language.Users of Python are often drawn-in by its ease of use, one piece of which is dynamic typing. While a statically-typed language like C or Java requires each variable to be explicitly declared, a dynamically-typed language like Python and MATLAB skips this specification. For example, in C you might specify a particular operation as follows:```cpp/* C code */int result = 0;for(int i=0; i<100; i++){    result += i;}printf("%d", i);```while in Python, the equivalent operation could be written this way:<jupyter_code># Python code
result = 0
for i in range(100):
    result += i
print(result)<jupyter_output><empty_output><jupyter_text>Notice the main difference: in C, the **data types of each variable are explicitly declared, while in Python the types are dynamically inferred**. This means, for example, that we can assign any kind of data to any variable:<jupyter_code># Python code
x = 4
print(x)
x = "four"
print(x)<jupyter_output><empty_output><jupyter_text>Here we've switched the contents of x from an integer to a string. The same thing in C would lead (depending on compiler settings) to a compilation error or other unintented consequences:```cpp/* C code */int x = 4;x = "four";  // FAILS```This sort of flexibility is one piece that makes Python and other dynamically-typed languages convenient and easy to use. Understanding how this works is an important piece of learning to analyze data efficiently and effectively with Python. But what this type-flexibility also points to is the fact that Python variables are more than just their value; they also contain extra information about the type of the value. We'll explore this more in the sections that follow. Here are some examples of the core **data types** in Python and some common **operations**. **String** data type is any combination of charater/s and numbers, enclosed with `''` or `""`. Examples of String data values are: ```   'a', 'b', '2', '_', "a", "b", "0", "-"  'hello', "world"  "hello129"  "True", "False"```One of the valid arithmatic (binary) operation is addition (+) of two strings. When addition is applied to string data values, we call it **concatenation**Another valid arithmatic (binary) operation is multiplication (*) used with integer, which makes `n` copies of the string<jupyter_code>print( "hello" + "world" )

name = "StudentA"
print( "Welcome, " + name + ", to ISE:4172." )
print("hello" * 3 ) 

name = "StudentA"
print( "hello" + name * 3 )<jupyter_output>hellohellohello
helloStudentAStudentAStudentA<jupyter_text>**Integer** data type is the common integers, $\mathbb{Z}$. Examples of integer data values are: ```  1, 10, 400, 32  -1, -200, -302  0```All arithmetic operations, {+, -, *, /}, are valid between integers.Some other arithmetic operations on integers are - `**` : power- `//` : quotient value of the division- `%`  : remainder value of the division<jupyter_code>print( 1 + 2 ) 
print( 3 * 2 ) 
print( 2 - 1 )
print( 3 / 2 ) 

print( 2 ** 10 ) 
print( 32 // 3 ) 
print( 100 % 3 )<jupyter_output>3
6
1
1.5<jupyter_text>**Float** data types are any floating numbers, $\mathbb{R}$. Examples of float data values are:```  3.2, 5.123, 0.00002  -0.1, -0.042, -100.2  2.0, -2.0, 0.0```All arithmetic operations, {+, -, *, /}, are valid between floats.<jupyter_code>print( 1.2 + 4.2 ) 
print( 3.02 * 2.1 ) 
print( 2.03 - 0.01 ) 
print( 3.2 / 0.06 ) 

print( 2.4 ** 2 ) 
print( 32.1 // 3 )
print( 102.53 % 3 )<jupyter_output>5.4
6.3420000000000005
2.02
53.333333333333336
5.76
10.0
0.5300000000000011<jupyter_text>**Boolean** is a data type that has only two data values, `True` or `False`Some of the common (binary) operators for the boolean data type are: `and`, `or`, `` (greater than), `>=` (greater than or equal to), `==` (equals to), `!=` (not equals to).`not` is a uniary operator on boolean data type that negates the boolean value at the operand.<jupyter_code># examples of boolean operations

print( not True ) 
print( not False )
print()

print( True == False )
print( True != False ) 
print()

print( 3 < 18 ) 
print( 3 > 18 ) 
print( 3 == 3 )
print( 3.0 == 3 ) 
print()

print( True and False ) 
print( True and True ) 
print()

print( True or False ) 
print( False or False )<jupyter_output>False
True

False
True

True
False
True
True

False
True

True
False<jupyter_text>It is possible to convert from one data type to another data type. Conversion of data types is called **type casting**, which can be performed by using a specific conversion functions. Namely, `str()`, `int()`, `float()`, `double()`, `bool()`<jupyter_code>print( str(32.0) )
print( str(True) )
print()

print( int( 32.012) ) 
print( int( '32' ) )
print( int( True ) )  # True has an integer value of 1
print( int( False ) ) # False has an integer value of 0
# int(string data type) only works for string data values with integers enclosed with '' or "" only
# int('hello32') or int(32.0) would raise an error. 
print()

print( float( 32 ) ) 
print( float( '32.0' ) )
print( float( True ) ) 
print( float( False ) ) 
print()

print( bool( 1 ) ) 
print( bool( 0 ) ) 
print( bool( 3 ) ) 
print()

# bool(String data types) only converts to False when it is empty string '', ""
print( bool( 'yes' ) ) 
print( bool( 'no' ) ) 
print( bool( '' ) )<jupyter_output>32.0
True

32
32
1
0

32.0
32.0
1.0
0.0

True
False
True

True
True
False<jupyter_text>Before going any further, I want to provide you some basics of programming laguages in Python. **variable** is like a named/referenced "box" where you can store, retrieve, and update data value. Storage of variable is temporary, so that you can reuse the variable with different data values. **syntax** is a grammer in computer programming. There are many operations you can do with the programminmg languages, but you have to follow exact **syntax** (or grammer) for each of the operations to work correctly.  For example, **variable assignment** is an operation of assigning a data value to a variable. Its' syntax is `identifier = data_value`. **Identifier** is a name specifying one variable from the others. There are rules for valid identifier, which are: *   Identifiers can be a combination of letters in lowercase **(a to z)** or uppercase **(A to Z)** or digits **(0 to 9)** or an underscore `_`.  * i.e. `myClass`, `var_i`, and `print_this_to_screen` are all valid example.*   Identifier cannot start with a digit  * i.e. `1variable` is invalid, but `variable1` is valid* **keywords** cannot be used as identifiers  * **keywords** are reserved names by Python language, which serve a special purpose.   * i.e. `return`, `False`, `and`, `if`, `in`, etc.* special symbols like **!, @, , $, %**, etc are not allowed in identifier* identifier can be of any length* Python is case-sensitive. `Variable` and `variable` are two different identifiers.<jupyter_code># example of variable assignment and shows how we can reuse the variable

x = 12
y = 23
print( x ) 
x = x * y
print( x ) 
x = y
print( x ) 
y = 2
print( x ) 

# you can do multiple variable assignment
x, y = 10, 12
print( x ) 
print( y ) 
# list of keywords in Python

import keyword
print( keyword.kwlist )<jupyter_output>['False', 'None', 'True', 'and', 'as', 'assert', 'break', 'class', 'continue', 'def', 'del', 'elif', 'else', 'except', 'finally', 'for', 'from', 'global', 'if', 'import', 'in', 'is', 'lambda', 'nonlocal', 'not', 'or', 'pass', 'raise', 'return', 'try', 'while', 'with', 'yield']<jupyter_text>**order of precedence** of python operators is the determined order in which operations are processed. The order of precedence of python operators is following: * (highest precedence) parentheses `()`* exponentiation `**`* multiplication and division `*`, `/`* addition and subraction `+`, `-`i.e. `2 ** 1 + 1` is 3 and not 4. `3*1**3` is 3 and not 27. [check out for more complete list of the order of precendence in Python](https://docs.python.org/3/reference/expressions.html)<jupyter_code># Exercise Problem (Assignment)
# declare an integer variable named `a` with initial value of 7
# declare a floating number variable (float) named `b` with initial value of 7
# declare a string variable named `c` with initial value of 'seven'<jupyter_output><empty_output><jupyter_text>You can **comment** in the python by using `` for a line of comment, and by using a pair of triple single quatation marks (**```**) for the block of comments.<jupyter_code>'''
  This is block of comments
  by using triple single quatation marks
'''
# all the comments will be ignored by Python interpreter
# comment is just for humans, not for computer<jupyter_output><empty_output><jupyter_text>Python executes each statements sequentially from left-to-right, and top-to-bottom.There are some methods where we can control the order of executing statements, called **flow control**First method of **flow control** is **conditional statements**. The syntax for the conditional statements are following: ```pythonif (boolean data value):  statement1_1  statement1_2  ... else:   statement2_1  statement2_2  ...```Note the statements (statement1_1, statement1_2, statement2_1, statement2_2) are indented. Conditional statements operate by first checking boolean data value. If the boolean data value is `True`, then statements grouped by `if` are being executed, i.e. statement1_1 and statement1_2. If the boolean data value is `False`, then statements grouped by `else` are being executed, i.e. statement2_1 and statement2_2.conditional statements have a variant syntax-form. ```pythonif (boolean data value):  statement1  statement2  ... elif (boolean data value):  statement1  statement2  ... else:  statement1  statement2  ...```In the variant syntax-form for the conditional statements, there can be more than one condition to check. It checks first condition next to `if` keyword; if it is `True`, if-code-block is being executed. If the first condision is `False`, it checks the second condition next to `elif` keyword; if it is `True`, elif-code-block is being executed. If both conditions are `False`, else-code-block is being executed.You can have multiple `elif` statements.Another important fact about conditional statement is that you can have conditional statements inside of another conditional statements. For example, ```pythonnum = 23if ( num < 50 ):  print ( 'number is less than 50' )  if (num < 25):    print( 'number is less than 25' )  else:    print( 'number is bigger than 25 but less than 50' )else :  print( 'number is larger than 50' )   if (num < 75 ):    print( 'number is larger than 50 but less than 75' )   elif( num < 90 ):    print( 'number is larger than 75 but less than 75' )   else:     print( 'nu mber is larger than 90' )```<jupyter_code># example of using conditional statements
if ( True ): 
  print( 'True' ) 
else: 
  print( 'False' )
# example of using conditional statements
age = 18
if ( age < 18 ):
  print( 'you are underage' )
else: 
  print( 'you are not underage' )
# example of using conditional statements
# it is okay to have only if-block without else-block

name = 'student'
if ( name == 'student' ):
  print( 'welcome student' )
# example of using conditional statements
dayOfMonth = 21

if ( dayOfMonth < 10 ):
  print( 'It is the start of the month' )
elif ( dayOfMonth > 10 and dayOfMonth < 20 ):
  print( 'It is the middle of the month' ) 
elif ( dayOfMonth == 21 ):
  print( 'It is 21st of the month' ) 
else: 
  print( 'It is almost the end of the month' )
# example of using conditional statements
drink = 'juice'
age = 13

if ( drink == 'juice' ):
  print( 'Here is your juice' ) 
elif ( drink == 'alcohol'):
  if( age < 18 ): 
    print( "Can't serve alcohol to underage. Please select juice" )
else: 
  print( drink + ' is not on our menu' )
  print( "please select either 'juice' or 'alcohol'")

print( 'Thank you' )<jupyter_output>Can't serve alcohol to underage. Please select juice
Thank you<jupyter_text>Another method for flow control is a **loop**, or repetition. It controls the flow of codes by repeating loop-block-codes. We can control how many time we want to repeat.There are multiple ways to loop in Python. ```python  for loop: we can set how many times we want to repeat.  it will repeat 3 times with x having values of 0, 1, and 2 for each iterations (or repeat) x is just an identifier. You can make it to any identifier you want. for x in range( 3 ):  print( x )  for loop: we can specify the starting number and end number.  it will repeat 4 times with x having values of 2, 3, 4, and 5 for each iterationfor x in range( 2, 6 ):  print( x )  for loop: we can even specify how much we want to increment for each iteration (default 1) it will repeat 6 times with x having values of 2, 5, 8, 11, 14, and 17.for y in range (2, 20, 3):  print( y ) ``` Loop can be nested as much as we want.<jupyter_code>for x in range(9): 
  for y in range(9):
    print( str(x) + ' * ' + str(y) + ' = ' + str( (x*y) ) )
  print()<jupyter_output><empty_output><jupyter_text>We can do the same repetition with the `while` loop. The syntax of the while loop is: ```pythonwhile (boolean data value):  statement1  statement2  ...```Unlike for-loop, you have to specify the conditition for the loop when to stop. Also you have to make changes so that condition will be `False` at some point, otherwise you will have an **infinite loop**, which repeats infinately. examples of the `while loop` relevant to `for loop` examples:```python ex1: x = 0  you have to define an iterator with an initial valuewhile ( x < 3 ):  print( x )   x = x + 1  have to make changes to the iterator so that the condition will end at some point ex2:x = 2while (x < 6):  print( x )  x = x + 1   ex3: y = 2while( y < 20 ):  print( y )  y = y + 3 ex4: x = 0while( x < 9 ):  y = 0  while( y < 9 ):    print( str(x) + ' + ' + str(y) + ' = ' + str( (x*y) ) )    y = y + 1  x = x + 1``` There are special keywords you can use to have more control when operation with loops, namely `continue` and `break``continue` allows you to go to the next iteration rathen than executing the statements below `coninue`. For example:<jupyter_code># example of 'continue'

for i in range( 6 ):
  if (i==3):
    continue
  print(i)
```<jupyter_output>0
1
2
4
5<jupyter_text>`break` allows you to stop the loop immediately even the condition is stil true. For example:<jupyter_code># example of 'break'
for i in range(6):
  print( i ) 
  if (i==3):
    break<jupyter_output>0
1
2
3<jupyter_text>Now, it's time to introduce new data structure. So far, we could assign a single data value to a variable. `Arrays` allows us to save collection of data values. Out of many forms of arrays, fundamental arrays are introduced here: * **List** : a collection which is ordered and changeable. Duplicate members allowed.* **Tuple** : a collection which is ordered and unchageable. Duplicate members allowed.* **Dictionary** : a collection which is unordered, changeable, and indexed. Duplicate members not allowed. In Python, **list** is written with square brackets, `[]`. Elements/members of the list are seperated by comma, `,`. Elements of the list can be further referenced by index number. Index number of Python's list start with 0. Also, you can use negative indexing to reference from the end of the list. Range of indexes is also allowed in Python, where retrieve elements in specified range. There are some useful functions with the list: * len(`list`) : returns the number of the elements in the list* `list`.append(new_item) : add new_item to the end of the list* `list`.insert(index_num, new_item) : add new_item at index_num* `list`.remove(item) : remove item from the list* `list`.pop() : remove the last element of the listfind a complete list for functions of `list` [here](https://docs.python.org/3/tutorial/datastructures.html)<jupyter_code>exList = ["apple", "banana", "cherry" ]
print( exList ) 
print( exList[1] )    # element with index number 1 from exList 
print( exList[-1] )   # last element from exList
print()

exList[1] = "carrot"  # you can change element of the list
print( exList ) 
print()

exList = ["apple", "banana", "cherry", "orange", "kiwi", "melon", "mango"]
print( exList[2:5] )  # elements from index number 2 to index number 4
print( exList[2:] )   # elements from index number 2 to the end of the list
print( exList[:2] )   # elements from start of the list to the index number 1
print( exList[-2:] )  # last two elements in the list
print( exList[:-2] )  # elements from the start of the list to the last 3rd element in the list
print()

# use of list functions
exList = ["apple", "banana", "cherry"]
exList.append("orange")
print( exList ) 

exList = ["apple", "banana", "cherry"]
exList.insert(1, "orange")
print( exList ) 

exList = ["apple", "banana", "cherry"]
exList.remove("banana")
print( exList ) 

exList = ["apple", "banana", "cherry"]
exList.pop()
print( exList )<jupyter_output>['apple', 'banana', 'cherry']
banana
cherry

['apple', 'carrot', 'cherry']

['cherry', 'orange', 'kiwi']
['cherry', 'orange', 'kiwi', 'melon', 'mango']
['apple', 'banana']
['melon', 'mango']
['apple', 'banana', 'cherry', 'orange', 'kiwi']

['apple', 'banana', 'cherry', 'orange']
['apple', 'orange', 'banana', 'cherry']
['apple', 'cherry']
['apple', 'banana']<jupyter_text>Arrays are iterable. It means we can use loop to access each element of the arrays. For example:<jupyter_code>exList = ["apple", "banana", "cherry", "orange", "kiwi", "melon", "mango"]
for i in exList: 
  print( i )
print()

# using loop and indexing to iterate elements in the array
for i in range( len( exList ) ):
  print( exList[i] )<jupyter_output>apple
banana
cherry
orange
kiwi
melon
mango

apple
banana
cherry
orange
kiwi
melon
mango<jupyter_text>A **tuple** is a collection which is ordered and unchangeable, denoted with parantheses, `()` in Python. Indexing skills on list are valid with the tuples as well.<jupyter_code># example of tuple

exTuple = ("apple", "banana", "cherry")
print( exTuple ) 
print( exTuple[1] ) 
print()

# iterate elements of the tuple using loop
for i in exTuple: 
  print( i )<jupyter_output>('apple', 'banana', 'cherry')
banana

apple
banana
cherry<jupyter_text>A **dictionary** is a collection which is ordered, changeable and indexed, denoted with curly brackets, `{}` in Python. Dictionary has **keys** and **values**. You can access elements of the dictionary by referencing to its key name.<jupyter_code># example of dictionary
exDict = {
    "brand" : "Jeep",
    "model" : "Wrangler", 
    "year" : 2010
}
print( exDict ) 
print( exDict["model"] ) 
print( exDict.get( "model" ) ) # Dictionary's get function does the same
print()

exDict["year"] = 2020
print( exDict ) 
print()

# print the length of the dictionary with `len()` method
print( len( exDict ) )
print()

# adding a new key and value into the dictionary
exDict["color"] = 'blue'
print( exDict ) 
print()

# removing the keys and value from the dictionary
exDict.pop('color')
print( exDict )
# Iterate elements of the dictionary with loop 
exDict = {
    "brand" : "Jeep",
    "model" : "Wrangler", 
    "year" : 2010
}

# prints all `keys` of the dictionary
for x in exDict: 
  print( x ) 
# Iterate elements of the dictionary with loop 
exDict = {
    "brand" : "Jeep",
    "model" : "Wrangler", 
    "year" : 2010
}

# prints all `values` of the dictionary
for x in exDict: 
  print( exDict[x] ) 
print()

# can retrieve values  by using `values()` method
for x in exDict.values():
  print( x )
print()

# can loop through keys and values together by using `items()` method
for x, y in exDict.items():
  print( x, y ) 
# Check if `key` exists
exDict = {
    "brand" : "Jeep",
    "model" : "Wrangler", 
    "year" : 2010
}

if ( "model" in exDict ):
  print( "Yes, `model` is one of the keys in the exDict dictionary")
# Dictionary can be nested
myFamily = {
    "child1" :{
        "name" : "Jane", 
        "year" : 2004
    }, 
    "child2" :{
        "name" : "July", 
        "year" : 2007
    }, 
    "child3" :{
        "name" : "Lilly", "year" : 2011
    }
}

print( myFamily )<jupyter_output>{'child1': {'name': 'Jane', 'year': 2004}, 'child2': {'name': 'July', 'year': 2007}, 'child3': {'name': 'Lilly', 'year': 2011}}<jupyter_text>Let's move on to functions now. **Function** is a block of code which only runs when it is called. We can pass data, known as parameters, into a function. A function can return data as a result. The syntax for function definition is: ``` python   function with multiple inputs and outputs a data  def functionName( param1, param2, ... ):    statement1    statement2    ...     return dataValue   function doesn't need to have any inputs  def functionName( ):    statement1    statement2    ...     return dataValue   function doesn't need to output any data  def functionName():    statement1    statement2    ... ```<jupyter_code>def myAddition( a, b ):
  return a + b

c = myAddition( 3, 2 )
print( c ) 
d = myAddition( c, 10 )
print( d ) 
def welcomeStatements():
  print( 'Welcom to the ISE:4172')

def welcomeStudent(studentName):
  print( "Weclome " + studentName + ' to the ISE:4172' )

welcomeStatements()
welcomeStudent( "Jake" )<jupyter_output>Welcom to the ISE:4172
Weclome Jake to the ISE:417<jupyter_text>2.1. A Python integer is more than just an integerThe standard Python implementation is written in C. This means that every Python object is simply a cleverly-disguised C structure, which contains not only its value, but other information as well. For example, when we define an integer in Python, such as x = 10000, x is not just a "raw" integer. It's actually a pointer to a compound C structure, which contains several values. Looking through the Python 3.4 source code, we find that the integer (long) type definition effectively looks like this (once the C macros are expanded):```cppstruct _longobject {    long ob_refcnt;    PyTypeObject *ob_type;    size_t ob_size;    long ob_digit[1];};```A single integer in Python 3.4 actually contains four pieces:- `ob_refcnt`, a reference count that helps Python silently handle memory allocation and deallocation- `ob_type`, which encodes the type of the variable- `ob_size`, which specifies the size of the following data members- `ob_digit`, which contains the actual integer value that we expect the Python variable to represent.This means that there is some overhead in storing an integer in Python as compared to an integer in a compiled language like C, as illustrated in the following figure:![C int vs Python int](https://github.com/stephenbaek/bigdata/blob/master/in-class-assignments/ica01/figures/cint_vs_pyint.png?raw=1)Here `PyObject_HEAD` is the part of the structure containing the reference count, type code, and other pieces mentioned before.Notice the difference here: a C integer is essentially a label for a position in memory whose bytes encode an integer value. A Python integer is a pointer to a position in memory containing all the Python object information, including the bytes that contain the integer value. This extra information in the Python integer structure is what allows Python to be coded so freely and dynamically. All this additional information in Python types comes at a cost, however, which becomes especially apparent in structures that combine many of these objects.  2.2. A Python list is more than just a listLet's consider now what happens when we use a Python data structure that holds many Python objects. The standard multi-element container in Python is the list. We can create a list of integers as follows:<jupyter_code>L = list(range(10))
print(L)       # print the entire list
print(L[0])    # print the zero-th element
print(L[2:5])  # print from 2nd element (inclusive) to 5th element (exclusive)
type(L[0])<jupyter_output><empty_output><jupyter_text>Or, similarly a list of strings:<jupyter_code>L2 = [str(c) for c in L]  # this is one of the useful things about Python. It looks almost like the English language.
print(L2)
type(L2[0])<jupyter_output><empty_output><jupyter_text>Becuase of Python's dynamic typing, we can even create heterogenous lists:<jupyter_code>L3 = [True, "2", 3.0, 4]
[type(item) for item in L3]<jupyter_output><empty_output><jupyter_text>But this flexibility comes at a cost: to allow these flexible types, each item in the list must contain its own type info, reference count, and other information–that is, each item is a complete Python object. In the special case that all variables are of the same type, much of this information is redundant: it can be much more efficient to store data in a fixed-type array, like what you would do (and are forced to do) in C. The difference between a dynamic-type list and a fixed-type array is illustrated in the following figure:![Array vs List](https://github.com/stephenbaek/bigdata/blob/master/in-class-assignments/ica01/figures/array_vs_list.png?raw=1)The word 'Numpy' in the figure is something we will learn in a moment. For now, just substitute the word with 'fixed-type'.At the implementation level, the fixed-type array essentially contains a single pointer to one contiguous block of data. The Python list, on the other hand, contains a pointer to a block of pointers, each of which in turn points to a full Python object like the Python integer we saw earlier. Again, the advantage of the list is flexibility: because each list element is a full structure containing both data and type information, the list can be filled with data of any desired type. Fixed-type arrays lack this flexibility, but are much more efficient for storing and manipulating data.In many data science applications, efficient storage and manipulation of numerical arrays is absolutely fundamental. For those applications, fixed-type arrays would be much more useful than dynamic-type lists, as all the elements we will be dealing with are essentially numbers. Furthermore, there are a lot of mathematical operations that we will need to apply to those numbers as we get to extract, transform, and analyze data. Unfortunately, the native Python does not have such capacity. Instead, there is a specialized toolset available for handling such numerical arrays, called the `NumPy` package and we will see this more in depth in the below section.  3. Introduction to NumPyNumPy is the fundamental package for scientific computing with Python. It contains:- a fixed-type n-dimensional array object- operations defined on n-dimensional arrays- useful linear algebra and other mathematical capabilitiesIf you followed the Python installation manual I provided on the course website, your system must already have NumPy installed. To confirm this, you can run the following code cell.<jupyter_code>import numpy
numpy.__version__<jupyter_output><empty_output><jupyter_text>By convention, you'll find that most people in the real-world will import NumPy using np as an alias:<jupyter_code>import numpy as np<jupyter_output><empty_output><jupyter_text>In fact, we will use `np` as an alias throughout the whole semester.  3.1. Creating NumPy Arrays from Python ListsFirst, we can use np.array to create arrays from Python lists:<jupyter_code># integer array:
np.array([1, 4, 2, 5, 3])<jupyter_output><empty_output><jupyter_text>Remember that unlike Python lists, NumPy is constrained to arrays that all contain the same type. If types do not match, NumPy will upcast if possible (here, integers are up-cast to floating point):<jupyter_code>np.array([3.14, 4, 2, 3])<jupyter_output><empty_output><jupyter_text>If we want to explicitly set the data type of the resulting array, we can use the dtype keyword:<jupyter_code>np.array([1, 2, 3, 4], dtype='float32')<jupyter_output><empty_output><jupyter_text>Finally, unlike the native Python lists, NumPy arrays can explicitly be multi-dimensional; here's one way of initializing a multidimensional array using a list of lists:<jupyter_code># nested lists result in multi-dimensional arrays
np.array([range(i, i + 3) for i in [2, 4, 6]])<jupyter_output><empty_output><jupyter_text>The inner lists are treated as rows of the resulting two-dimensional array.<jupyter_code># Exercise Problem (Assignment)
# create a numpy array that contains the first five numbers of the Fibonacci sequence
# (https://www.mathsisfun.com/numbers/fibonacci-sequence.html)
a =<jupyter_output><empty_output><jupyter_text>3.2. Creating Arrays from ScratchEspecially for larger arrays, it is more efficient to create arrays from scratch using routines built into NumPy. Here are several examples:<jupyter_code># Create a length-10 integer array filled with zeros
np.zeros(10, dtype=int)
# Create a 3x5 floating-point array filled with ones
np.ones((3, 5), dtype=float)
# Create a 3x5 array filled with 3.14
np.full((3, 5), 3.14)
# Create an array filled with a linear sequence
# Starting at 0, ending at 20, stepping by 2
# (this is similar to the built-in range() function)
np.arange(0, 20, 2)
# Create an array of five values evenly spaced between 0 and 1
np.linspace(0, 1, 5)
# Create a 3x3 array of uniformly distributed
# random values between 0 and 1
np.random.random((3, 3))
# Create a 3x3 array of normally distributed random values
# with mean 0 and standard deviation 1
np.random.normal(0, 1, (3, 3))
# Create a 3x3 array of random integers in the interval [0, 10)
np.random.randint(0, 10, (3, 3))
# Create a 3x3 identity matrix
np.eye(3)
# Exercise Problem (Assignment)
a = # create an array of ten values evenly spaced between -1 and 1
b = # create an array filled with a linear sequence starting at 10, ending at 20, stepping by 1
c = # create a random 5x5 matrix whose elements follow the normal distribution with mean 1 and stddev 0.5
d = # create a constant 3x3 matrix filled with 7<jupyter_output><empty_output><jupyter_text>3.3. NumPy Standard Data TypesNumPy arrays contain values of a single type, so it is important to have detailed knowledge of those types and their limitations. Because NumPy is built in C, the types will be familiar to users of C, Fortran, and other related languages.The standard NumPy data types are listed in the following table. Note that when constructing an array, they can be specified using a string:```np.zeros(10, dtype='int16')```Or using the associated NumPy object:```np.zeros(10, dtype=np.int16)``` | Data type	    | Description ||---------------|-------------|| ``bool_``     | Boolean (True or False) stored as a byte || ``int_``      | Default integer type (same as C ``long``; normally either ``int64`` or ``int32``)| | ``intc``      | Identical to C ``int`` (normally ``int32`` or ``int64``)| | ``intp``      | Integer used for indexing (same as C ``ssize_t``; normally either ``int32`` or ``int64``)| | ``int8``      | Byte (-128 to 127)| | ``int16``     | Integer (-32768 to 32767)|| ``int32``     | Integer (-2147483648 to 2147483647)|| ``int64``     | Integer (-9223372036854775808 to 9223372036854775807)| | ``uint8``     | Unsigned integer (0 to 255)| | ``uint16``    | Unsigned integer (0 to 65535)| | ``uint32``    | Unsigned integer (0 to 4294967295)| | ``uint64``    | Unsigned integer (0 to 18446744073709551615)| | ``float_``    | Shorthand for ``float64``.| | ``float16``   | Half precision float: sign bit, 5 bits exponent, 10 bits mantissa| | ``float32``   | Single precision float: sign bit, 8 bits exponent, 23 bits mantissa| | ``float64``   | Double precision float: sign bit, 11 bits exponent, 52 bits mantissa| | ``complex_``  | Shorthand for ``complex128``.| | ``complex64`` | Complex number, represented by two 32-bit floats| | ``complex128``| Complex number, represented by two 64-bit floats|  More advanced type specification is possible, such as specifying big or little endian numbers; for more information, refer to the [NumPy documentation](http://numpy.org/).NumPy also supports compound data types, but we won't cover it in this course.  3.4. Basic operations on NumPy arraysData science is virtually synonymous with data preparation and data preparation in Python is nearly synonymous with NumPy array manipulation. Here I show you several examples of using NumPy array manipulation to access data and subarrays, and to split, reshape, and join the arrays. While the types of operations shown here may seem a bit dry and pedantic, they comprise the building blocks of many other examples used throughout the course. Get to know them well!  Attributes<jupyter_code>x1 = np.random.randint(10, size=10)  # One-dimensional array
x2 = np.random.randint(10, size=(3, 4))  # Two-dimensional array
x3 = np.random.randint(10, size=(3, 4, 5))  # Three-dimensional array<jupyter_output><empty_output><jupyter_text>Each array has attributes ndim (the number of dimensions), shape (the size of each dimension), and size (the total size of the array):<jupyter_code>print("x3 ndim: ", x3.ndim)
print("x3 shape:", x3.shape)
print("x3 size: ", x3.size)<jupyter_output><empty_output><jupyter_text>Another useful attribute is the dtype, the data type of the array (which we discussed previously in Understanding Data Types in Python):<jupyter_code>print("dtype:", x3.dtype)
# Exercise Problem (Assignment)
N = np.random.randint(low=3, high=5, size=1)
M = np.random.randint(low=1, high=4, size=N)
a = np.random.normal(0, 1, size=M)
del M, N
# The above lines generates a random sized, random shaped array `a`
print( ) # print the total number of elements in `a`
print( ) # print the dimension of `a`
print( ) # print the data type of `a`
print( ) # print the shape of `a`<jupyter_output><empty_output><jupyter_text>AccessAccessing elements of a NumPy array is pretty similar to many other programming languages. In a one-dimensional array, the $i^{\text{th}}$ value (counting from zero) can be accessed by specifying the desired index in square brackets:<jupyter_code>print(x1)     # This prints the entire array
print(x1[0])  # This prints the zero-th element
print(x1[4])  # This prints the 4th element<jupyter_output><empty_output><jupyter_text>Negative indices allow you to index elements from the end of the array:<jupyter_code>print(x1[-1]) # This prints the last element
print(x1[-2]) # This prints the second from the last element
print(x1[-3]) # This prints the third from the last element<jupyter_output><empty_output><jupyter_text>In a multi-dimensional array, items can be accessed using a comma-separated tuple of indices:<jupyter_code>print(x2)        # This prints the entire array
print(x2[0, 0])  # This prints the element at 0-th row and 0-th column
print(x2[2, 3])  # This prints the 2nd row and 3rd column
print(x2[1, -1]) # This prints the 1st row and the last column<jupyter_output><empty_output><jupyter_text>Just as we can use square brackets to access individual array elements, we can also use them to access subarrays with the *slice* notation, marked by the colon (``:``) character.The NumPy slicing syntax follows that of the standard Python list; to access a slice of an array ``x``, use this:``` pythonx[start:stop:step]```If any of these are unspecified, they default to the values ``start=0``, ``stop=``*``size of dimension``*, ``step=1``.We'll take a look at accessing sub-arrays in one dimension and in multiple dimensions.<jupyter_code>print(x1)        # original array (for reference)
print(x1[:5])    # first five elements
print(x1[5:])    # elements after index 5
print(x1[4:7])   # middle sub-array
print(x1[::2])   # every other element
print(x1[1::2])  # every other element, starting at index 1
print(x1[::-1])  # all elements, reversed
print(x1[5::-2]) # reversed every other from index 5
print(x2)              # original array (for reference)
print(x2[:2, :3])      # two rows, three columns
print(x2[:3, ::2])     # all rows, every other column
print(x2[::-1, ::-1])  # reversed rows, reversed column
print(x2[:, 0])        # first column of x2
print(x2[0, :])        # first row of x2
print(x2[0])           # equivalent to x2[0, :]
# Exercise Problem (Assignment)
a = np.random.randint(10, size=(10, 10))
# matrix `a` contains random integers of shape 10 x 10
print( ) # print the first (2 x 3) block of the matrix on the top-left corner.
print( ) # print the (3 x 4) block on the bottom-right corner.
print( ) # print the first 5 elements of column 6
print( ) # print the last 3 elements of row 4
print( ) # print the row 7 in reverse order
print( ) # print the column 5 every three element starting at index 1<jupyter_output><empty_output><jupyter_text>Views and CopiesOne important–and extremely useful–thing to know about array slices is that they return **views** rather than **copies** of the array data. This is one area in which NumPy array slicing differs from Python list slicing: in lists, slices will be copies. Consider our two-dimensional array from before:<jupyter_code>print(x2)<jupyter_output><empty_output><jupyter_text>Let's extract a $2 \times 2$ subarray from this:<jupyter_code>x2_sub = x2[:2, :2]
print(x2_sub)<jupyter_output><empty_output><jupyter_text>Now if we modify this subarray, we'll see that the original array is changed! Observe:<jupyter_code>x2_sub[0, 0] = 99
print(x2_sub)
print(x2)<jupyter_output><empty_output><jupyter_text>This default behavior is actually quite useful: it means that when we work with large datasets, we can access and process pieces of these datasets without the need to copy the underlying data buffer.On the other hand, despite the nice features of array views, it is sometimes useful to instead explicitly copy the data within an array or a subarray. This can be most easily done with the ``copy()`` method:<jupyter_code>x2_sub_copy = x2[:2, :2].copy()
print(x2_sub_copy)<jupyter_output><empty_output><jupyter_text>If we now modify this subarray, the original array is not touched:<jupyter_code>x2_sub_copy[0, 0] = 42
print(x2_sub_copy)
print(x2)<jupyter_output><empty_output><jupyter_text>ReshapingAnother useful type of operation is reshaping of arrays.The most flexible way of doing this is with the ``reshape`` method.For example, if you want to put the numbers 1 through 9 in a $3 \times 3$ grid, you can do the following:<jupyter_code>grid = np.arange(1, 10).reshape((3, 3))
print(grid)<jupyter_output><empty_output><jupyter_text>Note that for this to work, the size of the initial array must match the size of the reshaped array. Where possible, the ``reshape`` method will use a no-copy view of the initial array, but with non-contiguous memory buffers this is not always the case.Another common reshaping pattern is the conversion of a one-dimensional array into a two-dimensional row or column matrix.This can be done with the ``reshape`` method, or more easily done by making use of the ``newaxis`` keyword within a slice operation:<jupyter_code>x = np.array([1, 2, 3])

# row vector via reshape
print(x.reshape((1, 3)))

# column vector via reshape
print(x.reshape((3, 1)))
# Exercise Problem (Assignment)
# create a 5x5 grid filled with numbers 1 through 25, from top to bottom. 
# i.e.   1   6  11  ...
#        2   7  12  ...
#        3   8  ...
#        4   9  ...
#        5  10  ...
a =<jupyter_output><empty_output><jupyter_text>Array Concatenation and SplittingAll of the preceding routines worked on single arrays. It's also possible to combine multiple arrays into one, and to conversely split a single array into multiple arrays. We'll take a look at those operations here.First off, concatenation, or joining of two arrays in NumPy, is primarily accomplished using the routines ``np.concatenate``, ``np.vstack``, and ``np.hstack``.``np.concatenate`` takes a tuple or list of arrays as its first argument, as we can see here:<jupyter_code>x = np.array([1, 2, 3])
y = np.array([3, 2, 1])
np.concatenate([x, y])<jupyter_output><empty_output><jupyter_text>You can also concatenate more than two arrays at once:<jupyter_code>z = [99, 99, 99]
print(np.concatenate([x, y, z]))<jupyter_output><empty_output><jupyter_text>It can also be used for two-dimensional arrays:<jupyter_code>grid = np.array([[1, 2, 3],
                 [4, 5, 6]])
# concatenate along the first axis
np.concatenate([grid, grid])
# concatenate along the second axis (zero-indexed)
np.concatenate([grid, grid], axis=1)<jupyter_output><empty_output><jupyter_text>For working with arrays of mixed dimensions, it can be clearer to use the ``np.vstack`` (vertical stack) and ``np.hstack`` (horizontal stack) functions:<jupyter_code>x = np.array([1, 2, 3])
grid = np.array([[9, 8, 7],
                 [6, 5, 4]])

# vertically stack the arrays
np.vstack([x, grid])
# horizontally stack the arrays
y = np.array([[99],
              [99]])
np.hstack([grid, y])<jupyter_output><empty_output><jupyter_text>Similary, np.dstack will stack arrays along the third axis. The opposite of concatenation is splitting, which is implemented by the functions ``np.split``, ``np.hsplit``, and ``np.vsplit``.  For each of these, we can pass a list of indices giving the split points:<jupyter_code>x = [1, 2, 3, 99, 99, 3, 2, 1]
x1, x2, x3 = np.split(x, [3, 5])
print(x1, x2, x3)<jupyter_output><empty_output><jupyter_text>Notice that $N$ split-points, leads to $N + 1$ subarrays.The related functions ``np.hsplit`` and ``np.vsplit`` are similar:<jupyter_code>grid = np.arange(16).reshape((4, 4))
grid
upper, lower = np.vsplit(grid, [2])
print(upper)
print(lower)
left, right = np.hsplit(grid, [2])
print(left)
print(right)<jupyter_output><empty_output><jupyter_text>Similarly, np.dsplit will split arrays along the third axis.  3.5. Broadcasting For arrays of the same size, binary operations are performed on an element-by-element basis:<jupyter_code>a = np.array([0, 1, 2])
b = np.array([5, 5, 5])
print(a + b)
print(a * b)<jupyter_output><empty_output><jupyter_text>Broadcasting allows these types of binary operations to be performed on arrays of different sizes–for example, we can just as easily add a scalar (think of it as a zero-dimensional array) to an array:<jupyter_code>print(a + 5)<jupyter_output><empty_output><jupyter_text>We can think of this as an operation that stretches or duplicates the value ``5`` into the array ``[5, 5, 5]``, and adds the results.The advantage of NumPy's broadcasting is that this duplication of values does not actually take place, but it is a useful mental model as we think about broadcasting.We can similarly extend this to arrays of higher dimension. Observe the result when we add a one-dimensional array to a two-dimensional array:<jupyter_code>M = np.ones((3, 3))
print(M)
print(M + a)<jupyter_output><empty_output><jupyter_text>Here the one-dimensional array ``a`` is stretched, or broadcast across the second dimension in order to match the shape of ``M``.While these examples are relatively easy to understand, more complicated cases can involve broadcasting of both arrays. Consider the following example:<jupyter_code>a = np.arange(3)
b = np.arange(3)[:, np.newaxis]

print(a)
print(b)
a + b<jupyter_output><empty_output><jupyter_text>Just as before we stretched or broadcasted one value to match the shape of the other, here we've stretched *both* ``a`` and ``b`` to match a common shape, and the result is a two-dimensional array!The geometry of these examples is visualized in the following figure:![Broadcasting Visual](https://github.com/stephenbaek/bigdata/blob/master/in-class-assignments/ica01/figures/broadcasting.png?raw=1)The light boxes represent the broadcasted values: again, this extra memory is not actually allocated in the course of the operation, but it can be useful conceptually to imagine that it is.  3.6. Boolean operations NumPy implements comparison operators such as  (greater than) as element-wise universal functions. The result of these comparison operators is always an array with a Boolean data type. All six of the standard comparison operations are available:<jupyter_code>x = np.array([1, 2, 3, 4, 5])
print(x < 3)
print(x > 3)
print(x <= 3)  # less than or equal to
print(x >= 3)  # greater than or equal to
print(x != 3)  # not equal to
print(x == 3)  # equal to<jupyter_output><empty_output><jupyter_text>It is also possible to do an element-wise comparison of two arrays, and to include compound expressions:<jupyter_code>(2 * x) == (x ** 2)  # x ** 2 stands for 'x squared'<jupyter_output><empty_output><jupyter_text>To count the number of ``True`` entries in a Boolean array, ``np.count_nonzero`` is useful:<jupyter_code>print( np.count_nonzero(x < 3) )  # how many values less than 3?
print( np.sum(x < 3) )            # or equally, you can do this<jupyter_output><empty_output><jupyter_text>For multi-dimensional arrays, you count along different dimensions:<jupyter_code>x = np.random.randint(10, size=(3, 4))
print(x)
print(np.sum(x < 6, axis=1)) # how many values less than 6 in each row?<jupyter_output><empty_output><jupyter_text>If we're interested in quickly checking whether any or all the values are true, we can use (you guessed it) ``np.any`` or ``np.all``:<jupyter_code># are there any values greater than 8?
np.any(x > 8)
# are there any values less than zero?
np.any(x < 0)
# are all values less than 10?
np.all(x < 10)
# are all values equal to 6?
np.all(x == 6)<jupyter_output><empty_output><jupyter_text>``np.all`` and ``np.any`` can be used along particular axes as well. For example:<jupyter_code># are all values in each row less than 8?
np.all(x < 8, axis=1)<jupyter_output><empty_output><jupyter_text>A quick warning: Python has built-in ``sum()``, ``any()``, and ``all()`` functions. These have a different syntax than the NumPy versions, and in particular will fail or produce unintended results when used on multidimensional arrays. Be sure that you are using ``np.sum()``, ``np.any()``, and ``np.all()`` for these examples! Finally, Python's *bitwise logic operators*, ``&``, ``|``, ``^``, and ``~`` allows you to perform logic operations.Like with the standard arithmetic operators, these are element-wise operations on (usually Boolean) arrays.For example, we can address this sort of compound question as follows:<jupyter_code>np.sum((x > 1) & (x < 5))<jupyter_output><empty_output><jupyter_text>The following table summarizes the bitwise Boolean operators:| Operator	    | Name               || Operator	  | Name             ||---------------|--------------------||---------------|------------------||``&``          | element-wise AND   ||&124;         | element-wise OR  ||``^``          | element-wise XOR   ||``~``          | element-wise NOT |<jupyter_code># Exercise Problem (Assignment)
a = np.random.randint(10, (10, 10))

# is there any zero element in `a`? implement a code to answer the question.

# how many elements in `a` are smaller than 5? implement a code to answer the question.

# multiply 5 to the elements of `a` ONLY WHEN they are smaller than 5.<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10480274114201489, 'fraction_numerical': 0.022625906854769243, 'mean_word_length': 4.491065797771705, 'pattern_counts': {'":': 0, '<': 272, '<?xml version=': 0, '>': 256, 'https://': 10, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 3, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '64', 'dir': 'jupyter-structured-clean-dedup', 'id': '58189', 'license': 'MIT', 'n_tokens_mistral': 16323, 'n_tokens_neox': 15241, 'n_words': 7069, 'path': 'in-class-assignments/ica01/hello_world.ipynb', 'repo_name': 'stephenbaek/bigdata', 'size': '52250'}