<jupyter_start><jupyter_text>Python Best PracticesCode readability is key, Python syntax itself is close to plain english. Your variables, functions should be given descriptive identifiers!Identifiers for variables should be descriptive words separated by underscores (not spaces) and in all lower case. Classes can use CamelCase.BAD: `var6 = 25`, `AG3ofMoTh3R = 25`. GOOD : `age_of_mother = 25`Do this:```pythondef optimize_animals(cats, dogs):	return cats ** 2 + np.sqrt(dogs)```Not this:```pythondef feval(x, y):	return x**2+np.sqrt(y)```    White SpaceYou should use white space to increase readability. Apply spacing around operators, except in function calls and slices. Also insert newlines if that makes sense logically, perhaps with insightful comments.BAD: `x=[2,3,4]`, GOOD: `x = [2, 3, 4]`Do this:```python Normalize x by adding 12x = x + 12   Train on first 10 data pointstrain(data[:10], bias=x)```Not this:```pythonx=x+12 Add 12train( data[: 10], bias = x )```  Avoid name shadowing. Please use names that are different from built-in functions, so to make sure they can still be used.Do this:```pythontrain_data = data[:10] max_iterations = 100```Not this:```pythoninput = data[:10]   Shadows input() functioniter = 100   Shadows iter() function```  Additional- You should liberally intersperse your code with comments!- Proper indentation is non-negotiable. Be consistent in what you use. In Python, indentation matters!- If possible, do not use map and reduce where you can also run a for-loop (unless your environment, like Spark, specifically requires it)- Be consistent with variable names across all your work through the semester. For example you might want to always write standard deviation as ‘std_dev’ or ‘sigma’.   General GuidelinesThe general guideline for Python code is: try to stick by PEP8 as much as possible. The complete style guide is available here: https://www.python.org/dev/peps/pep-0008/. Please read this document: while we dont expect strict adherance to it, it will make your code readable by the entire python community.If you would like to adhere to it strictly, check out the code linter `flake8`. This is a tool that will complain if your code does not adhere to PEP8.The one place we differ is in writing documentation.  We will be using the `numpy` style guide instead. See  the [docstring section](https://numpydoc.readthedocs.io/en/latest/format.htmldocstring-standard) of the `numpy` style guide. A question may arise: If you are expected to provide documentation in code-cells, what is the notebook markdown cell for? The answer is that it is for English and Math explanation of your process, and a very high level overview  and skeleton of the code, rather than details.  Programming ExpectationsAll assignments for this class will use Python and the browser-based iPython notebook format you are currently viewing. Python experience is not a prerequisite for this course, as long as you are comfortable learning on your own as needed. While we strive to make the programming component of this course straightforward, we won't devote much time to teaching programming or Python syntax.  We will refer to the Python 3 [documentation](https://docs.python.org/3/) in this lab and throughout the course.  There are also many introductory tutorials to help build programming skills, which we are listed in the last section of this lab.  Table of Contents  Learning Goals  Getting Started Lists  Strings and Listiness  Dictionaries  Functions  Text Analysis of Hamlet  References   Part 0:  Learning Goals This introductory lab is a condensed tutorial in Python programming.  By the end of this lab, you will feel more comfortable:- Writing short Python code using functions, loops, arrays, dictionaries, strings,  if statements.- Manipulating Python lists and recognizing the listy properties of other Python containers.- Learning and reading Python documentation.    Part 1: Getting Started  Importing modulesAll notebooks should begin with code that imports *modules*, collections of built-in, commonly-used Python functions.  Below we import the Numpy module, a fast numerical programming library for scientific computing.  Future labs will require additional modules, which we'll import with the same `import MODULE_NAME as MODULE_NICKNAME` syntax.<jupyter_code>import numpy as np #imports a fast numerical programming library<jupyter_output><empty_output><jupyter_text>Now that Numpy has been imported, we can access some useful functions.  For example, we can use `mean` to calculate the mean of a set of numbers.<jupyter_code>np.mean([1.2, 2, 3.3])<jupyter_output><empty_output><jupyter_text>to calculate the mean of 1.2, 2, and 3.3. The code above is not particularly efficient, and efficiency will be important for you when dealing with large data sets. In Lab 1 we will see more efficient options.  Calculations and variables At the most basic level we can use Python as a simple calculator.<jupyter_code>1 + 2<jupyter_output><empty_output><jupyter_text>Notice integer division (//) and floating-point error below!<jupyter_code>1/2, 1//2, 1.0/2.0, 3*3.2<jupyter_output><empty_output><jupyter_text>The last line in a cell is returned as the output value, as above.  For cells with multiple lines of results, we can display results using ``print``, as can be seen below.<jupyter_code>print(1 + 3.0, "\n", 9, 7)
5/3<jupyter_output>4.0 
 9 7<jupyter_text>We can store integer or floating point values as variables.  The other basic Python data types -- booleans, strings, lists -- can also be stored as variables.<jupyter_code>a = 1
b = 2.0<jupyter_output><empty_output><jupyter_text>Here is the storing of a list:<jupyter_code>a = [1, 2, 3]<jupyter_output><empty_output><jupyter_text>Think of a variable as a label for a value, not a box in which you put the value![](images/sticksnotboxes.png)(image taken from Fluent Python by <NAME>)<jupyter_code>b = a
b<jupyter_output><empty_output><jupyter_text>This DOES NOT create a new copy of `a`. It merely puts a new label on the memory at a, as can be seen by the following code:<jupyter_code>print("a", a)
print("b", b)
a[1] = 7
print("a after change", a)
print("b after change", b)<jupyter_output>a [1, 2, 3]
b [1, 2, 3]
a after change [1, 7, 3]
b after change [1, 7, 3]<jupyter_text>Multiple items on one line in the interface are returned as a *tuple*, an immutable sequence of Python objects.<jupyter_code>a = 1
b = 2.0
a + a, a - b, b * b, 10*a<jupyter_output><empty_output><jupyter_text>We can obtain the type of a variable, and use boolean comparisons to test these types.<jupyter_code>type(a) == float
type(a) == int<jupyter_output><empty_output><jupyter_text>For reference, below are common arithmetic and comparison operations. >**EXERCISE**:  Create a tuple called `tup` with the following seven objects:> - The first element is an integer of your choice> - The second element is a float of your choice  > - The third element is the sum of the first two elements> - The fourth element is the difference of the first two elements> - The fifth element is first element divided by the second element> Display the output of `tup`.  What is the type of the variable `tup`? What happens if you try and chage an item in the tuple?<jupyter_code># create a tuple called tup with the above 5 objects
tup = (4, 2.0, 6.0, 2.0, 2.0)

# print the output of tup
print(tup)

# print the type of variable of tup
print(type(tup))

# an item can not be changed in a tuple as it is immutable. A tuple would first need to be changed to a list, item changed
# and then converted back to a tuple
tup = (4, 2.0, 6.0, 2.0, 2.0)
y = list(tup)
y[1] = 5.0
tup = tuple(y)
print(tup)<jupyter_output>(4, 2.0, 6.0, 2.0, 2.0)
<class 'tuple'>
(4, 5.0, 6.0, 2.0, 2.0)<jupyter_text>Part 2: ListsMuch of Python is based on the notion of a list.  In Python, a list is a sequence of items separated by commas, all within square brackets.  The items can be integers, floating points, or another type.  Unlike in C arrays, items in a Python list can be different types, so Python lists are more versatile than traditional arrays in C or in other languages. Let's start out by creating a few lists.<jupyter_code>empty_list = []
float_list = [1., 3., 5., 4., 2.]
int_list = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
mixed_list = [1, 2., 3, 4., 5]
print(empty_list)
print(int_list)
print(mixed_list, float_list)<jupyter_output>[]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
[1, 2.0, 3, 4.0, 5] [1.0, 3.0, 5.0, 4.0, 2.0]<jupyter_text>Lists in Python are zero-indexed, as in C.  The first entry of the list has index 0, the second has index 1, and so on.<jupyter_code>print(int_list[0])
print(float_list[1])<jupyter_output>1
3.0<jupyter_text>What happens if we try to use an index that doesn't exist for that list?  Python will complain!<jupyter_code>print(float_list[10])<jupyter_output><empty_output><jupyter_text>A list has a length at any given point in the execution of the code, which we can find using the `len` function.<jupyter_code>print(float_list)
len(float_list)<jupyter_output><empty_output><jupyter_text>Indexing on lists And since Python is zero-indexed, the last element of `float_list` is<jupyter_code>float_list[len(float_list)-1]<jupyter_output><empty_output><jupyter_text>It is more idiomatic in python to use -1 for the last element, -2 for the second last, and so on<jupyter_code>float_list[-1]<jupyter_output><empty_output><jupyter_text>We can use the ``:`` operator to access a subset of the list.  This is called *slicing.*<jupyter_code>print(float_list[1:5])
print(float_list[0:2])<jupyter_output><empty_output><jupyter_text>Below is a summary of list slicing operations: You can slice "backwards" as well:<jupyter_code>float_list[:-2] # up to second last
float_list[:4] # up to but not including 5th element<jupyter_output><empty_output><jupyter_text>You can also slice with a stride:<jupyter_code>float_list[:4:2] # above but skipping every second element<jupyter_output><empty_output><jupyter_text>We can iterate through a list using a loop.  Here's a for loop.<jupyter_code>for ele in float_list:
    print(ele)<jupyter_output><empty_output><jupyter_text>Or, if we like, we can iterate through a list using the indices using a for loop with  `in range`. This is not idiomatic and is not recommended, but accomplishes the same thing as above.<jupyter_code>for i in range(len(float_list)):
    print(float_list[i])<jupyter_output><empty_output><jupyter_text>What if you wanted the index as well? Python has other useful functions such as `enumerate`,  which can be used to create a list of tuples with each tuple of the form `(index, value)`.<jupyter_code>for i, ele in enumerate(float_list):
    print(i,ele)
list(enumerate(float_list))<jupyter_output><empty_output><jupyter_text>This is an example of an *iterator*, something that can be used to set up an iteration. When you call `enumerate`, a list if tuples is not created. Rather an object is created, which when iterated over (or when the `list` function is called using it as an argument), acts like you are in a loop, outputting one tuple at a time.  Appending and deleting We can also append items to the end of the list using the `+` operator or with `append`.<jupyter_code>float_list + [.333]
float_list.append(.444)
print(float_list)
len(float_list)<jupyter_output><empty_output><jupyter_text>Go and run the cell with `float_list.append` a second time.  Then run the next line.  What happens?  To remove an item from the list, use `del.`<jupyter_code>del(float_list[2])
print(float_list)<jupyter_output><empty_output><jupyter_text>List ComprehensionsLists can be constructed in a compact way using a *list comprehension*.  Here's a simple example.<jupyter_code>squaredlist = [i*i for i in int_list]
squaredlist<jupyter_output><empty_output><jupyter_text>And here's a more complicated one, requiring a conditional.<jupyter_code>comp_list1 = [2*i for i in squaredlist if i % 2 == 0]
print(comp_list1)<jupyter_output><empty_output><jupyter_text>This is entirely equivalent to creating `comp_list1` using a loop with a conditional, as below:<jupyter_code>comp_list2 = []
for i in squaredlist:
    if i % 2 == 0:
        comp_list2.append(2*i)
        
comp_list2<jupyter_output><empty_output><jupyter_text>The list comprehension syntax```[expression for item in list if conditional]```is equivalent to the syntax```for item in list:    if conditional:        expression``` >**EXERCISE**:  Build a list that contains every prime number between 1 and 100, in two different ways:1.  Using for loops and conditional if statements.2.  *(Stretch Goal)* Using a list comprehension.  You should be able to do this in one line of code, and it may be helpful to look up the function `all` in the documentation.<jupyter_code>for i in range(2, 101):
    for j in range(2, i-1):
        if i % j == 0: 
            break
    else:
        print (i)
# your code here<jupyter_output><empty_output><jupyter_text>Part 3:  Strings and listiness A list is a container that holds a bunch of objects.  We're particularly interested in Python lists because many other containers in Python, like strings, dictionaries, numpy arrays, pandas series and dataframes, and iterators like `enumerate`, have list-like properties.  This is known as [duck](https://en.wikipedia.org/wiki/Duck_typing) typing, a term coined by <NAME>, which refers to the notion that  *if it quacks like a duck, it is a duck*.  We'll soon see that these  containers quack like lists, so for practical purposes we can think of these containers as lists!  They are listy!Containers that are listy have a set length, can be sliced, and can be iterated over with a loop.  Let's look at some listy containers now.  StringsWe claim that strings are listy.  Here's a string.<jupyter_code>astring = "kevin"<jupyter_output><empty_output><jupyter_text>Like lists, this string has a set length, the number of characters in the string.<jupyter_code>len(astring)<jupyter_output><empty_output><jupyter_text>Like lists, we can slice the string.<jupyter_code>print(astring[0:2])
print(astring[0:6:2])
print(astring[-1])<jupyter_output><empty_output><jupyter_text>And we can iterate through the string with a loop.  Below is a while loop:<jupyter_code>i = 0
while i < len(astring):
    print(astring[i])
    i = i + 1<jupyter_output><empty_output><jupyter_text>This is equivalent to the for loop:<jupyter_code>for character in astring:
    print(character)<jupyter_output><empty_output><jupyter_text>So strings are listy.  How are strings different from lists?  While lists are mutable, strings are immutable.  Note that an error occurs when we try to change the second elemnt of `string_list` from 1 to b.<jupyter_code>print(float_list)
float_list[1] = 2.09
print(float_list)
print(astring)
astring[1] = 'b'
print(astring)<jupyter_output><empty_output><jupyter_text>We can't use `append` but we can concatenate with `+`. Why is this?<jupyter_code>astring = astring + ', pavlos, ' + 'rahul, ' + 'margo'
print(astring)
type(astring)<jupyter_output><empty_output><jupyter_text>What is happening here is that we are creating a new string in memory when we do `astring + ', pavlos, ' + 'rahul, ' + 'margo'`. Then we are relabelling this string with the old lavel `astring`. This means that the old memory that `astring` labelled is forgotten. What happens to it? We'll find out in lab 1. Or we could use `join`.  See below for a summary of common string operations.    To summarize this section, for  practical purposes all containers that are listy have the following properties:1.  Have a set length, which you can find using `len`2.  Are iterable (via a loop)3.  Are sliceable via : operationsWe will encounter other listy containers soon.<jupyter_code>list(range(-1, -5))<jupyter_output><empty_output><jupyter_text>>**EXERCISE**: Make three strings, called `first`, `middle`, and `last`, with your first, middle, and last names, respectively.  If you don't have a middle name, make up a middle name!  >Then create a string called `full_name` that joins your first, middle, and last name, with a space separating your first, middle, and last names.  >Finally make a string called `full_name_rev` which takes `full_name` and reverses the letters.  For example, if `full_name` is `<NAME>`, then `full_name_rev` is `eoD hteB enaJ`.<jupyter_code># create 3 strings called first, middle and last
first = 'Rebecca'
middle = 'Theresa'
last = 'Cole'

# create new string called full_name that joins first, middle and last
full_name = f'{first} {middle} {last}'

# print string
print(full_name)

# create a string called full_name_rev which takes full_name and reverses the letters

# calculate length of the string full_name
string_length =len (full_name)

# start at the end of string_length, end at position 0, move with the step -1 (or one step backward)
sliced_string = full_name[string_length::-1] 

# print the reversed string
print(sliced_string)<jupyter_output><empty_output><jupyter_text>Part 4: DictionariesA dictionary is another storage container.  Like a list, a dictionary is a sequence of items.  Unlike a list, a dictionary is unordered and its items are accessed with keys and not integer positions.  Dictionaries are the closest container we have to a database.Let's make a dictionary with a few Harvard courses and their corresponding enrollment numbers.<jupyter_code>enroll2016_dict = {'CS50': 692, 'CS109 / Stat 121 / AC 209': 312, 'Econ1011a': 95, 'AM21a': 153, 'Stat110': 485}
enroll2016_dict
enroll2016_dict.values()
enroll2016_dict.items()
for key, value in enroll2016_dict.items():
    print("%s: %d" %(key, value))<jupyter_output><empty_output><jupyter_text>Simply iterating over a dictionary gives us the keys. This is useful when we want to do something with each item:<jupyter_code>second_dict={}
for key in enroll2016_dict:
    second_dict[key] = enroll2016_dict[key]
second_dict<jupyter_output><empty_output><jupyter_text>The above is an actual copy to another part of memory, unlike, `second_dict = enroll2016_dict` which would have made both variables label the same meory location. In this example, the keys are strings corresponding to course names.  Keys don't have to be strings though.  Like lists, you can construct dictionaries using a *dictionary comprehension*, which is similar to a list comprehension. Notice the brackets {} and the use of `zip`, which is another iterator that combines two lists together.<jupyter_code>my_dict = {k:v for (k, v) in zip(int_list, float_list)}
my_dict<jupyter_output><empty_output><jupyter_text>You can also create dictionaries nicely using the *constructor* function `dict`.<jupyter_code>dict(a = 1, b = 2)<jupyter_output><empty_output><jupyter_text>While dictionaries have some similarity to lists, they are not listy.  They do have a set length, and the can be iterated through with a loop, but they cannot be sliced, since they have no sense of an order. In technical terms, they satisfy, along with lists and strings, Python's *Sequence* protocol, which is a higher abstraction than that of a list.  A cautionary word on iterators (read at home)Iterators are a bit different from lists in the sense that they can be "exhausted". Perhaps its best to explain with an example<jupyter_code>an_iterator = enumerate(astring)
type(an_iterator)
for i, c in an_iterator:
    print(i,c)
for i, c in an_iterator:
    print(i,c)<jupyter_output><empty_output><jupyter_text>What happens, you get nothing when you run this again! This is because the iterator has been "exhausted", ie, all its items are used up. I have had answers go wrong for me because I wasnt careful about this. You must either track the state of the iterator or bypass this problem by not storing `enumerate(BLA)` in a variable, so that you dont inadvertantly "use that variable" twice.  Part 5: FunctionsA *function* is a reusable block of code that does a specfic task.  Functions are all over Python, either on their own or on objects.  We've seen built-in Python functions and methods.  For example, `len` and `print` are built-in Python functions.  And at the beginning of the lab, you called `np.mean` to calculate the mean of three numbers, where `mean` is a function in the numpy module and numpy was abbreviated as `np`. This syntax allow us to have multiple "mean" functions" in different modules; calling this one as `np.mean` guarantees that we will pick up numpy's mean function. MethodsA function that belongs to an object is called a *method*. An example of this is `append` on an **existing** list. In other words, a *method* is a function on an **instance** of a type of object (also called **class**, here the list type).<jupyter_code>print(float_list)
float_list.append(56.7) 
float_list<jupyter_output><empty_output><jupyter_text>User-defined functionsWe'll now learn to write our own user-defined functions.  Below is the syntax for defining a basic function with one input argument and one output. You can also define functions with no input or output arguments, or multiple input or output arguments.```def name_of_function(arg):    ...    return(output)``` The simplest function has no arguments whatsoever.<jupyter_code>def print_greeting():
    print("Hello")
    
print_greeting()<jupyter_output>Hello<jupyter_text>We can write functions with one input and one output argument.  Here are two such functions.<jupyter_code>def square(x):
    x_sqr = x*x
    return(x_sqr)

def cube(x):
    x_cub = x*x*x
    return(x_cub)

square(5),cube(5)<jupyter_output><empty_output><jupyter_text>Lambda functionsOften we define a mathematical function with a quick one-line function called a *lambda*. No return statement is needed.The big use of lambda functions in data science is for mathematical functions.<jupyter_code>square = lambda x: x*x
print(square(3))


hypotenuse = lambda x, y: x*x + y*y

## Same as

# def hypotenuse(x, y):
#     return(x*x + y*y)

hypotenuse(3,4)<jupyter_output><empty_output><jupyter_text>Refactoring using functions>**EXERCISE**: Write a function called `isprime` that takes in a positive integer $N$, and determines whether or not it is prime.  Return the $N$ if it's prime and return nothing if it isn't.  You may want to reuse part of your code from the exercise in Part 2.  > Then, using a list comprehension and `isprime`, create a list `myprimes` that contains all the prime numbers less than 100.<jupyter_code># define a function called is_prime with parameter N
def is_prime(N):
# if N is equal to 1 return False as not prime
    if (N == 1):
        return False
# if N is equal to 2 return N as is prime
    elif (N == 2):
        return N;
# else for all numbers in the range from 2 to N, if N modulus i is zero return false otherwise return N
    else:
        for i in range(2, N):
            if(N % i == 0):
                return False
        return N 
# print
print(is_prime(97))<jupyter_output>97<jupyter_text>Notice that what you just did is a **refactoring** of the algorithm you used earlier to find primes smaller than 100. This implementation reads much cleaner, and the function `isprime` which containes the "kernel" of the functionality of the algorithm can be **re-used** in other places. You should endeavor to write code like this.  Default ArgumentsFunctions may also have *default* argument values.  Functions with default values are used extensively in many libraries.<jupyter_code># This function can be called with x and y, in which case it will return x*y;
# or it can be called with x only, in which case it will return x*1.
def get_multiple(x, y = 1):
    return x*y

print("With x and y:", get_multiple(10, 2))
print("With x only:", get_multiple(10))<jupyter_output><empty_output><jupyter_text>We can have multiple default values.<jupyter_code>def print_special_greeting(name, leaving = False, condition = "nice"):
    print("Hi", name)
    print("How are you doing on this", condition, "day?")
    if leaving:
        print("Please come back! ")

# Use all the default arguments.
print_special_greeting("Rahul")<jupyter_output><empty_output><jupyter_text>Or change all the default arguments:<jupyter_code>print_special_greeting("Rahul", True, "rainy")<jupyter_output><empty_output><jupyter_text>Or use the first default argument but change the second one.<jupyter_code>print_special_greeting("Rahul", condition="horrible")<jupyter_output><empty_output><jupyter_text>Positional and keyword arguments These allow for even more flexibility.  *Positional* arguments are used when you don't know how many input arguments your function be given.  Notice the single asterisk before the second argument name.<jupyter_code>def print_siblings(name, *siblings):
    print(name, "has the following siblings:")
    for sibling in siblings:
        print(sibling)
    print()
print_siblings("John", "Ashley", "Lauren", "Arthur")
print_siblings("Mike", "John")
print_siblings("Terry")<jupyter_output><empty_output><jupyter_text>In the function above, arguments after the first input will go into a list called siblings. We can then process that list to extract the names.*Keyword* arguments mix the named argument and positional properties.  Notice the double asterisks before the second argument name.<jupyter_code>def print_brothers_sisters(name, **siblings):
    print(name, "has the following siblings:")
    for sibling in siblings:
        print(sibling, ":", siblings[sibling])
    print()
    
print_brothers_sisters("John", Ashley="sister", Lauren="sister", Arthur="brother")<jupyter_output><empty_output><jupyter_text>Putting things togetherFinally, when putting all those things together one must follow a certain order:Below is a more general function definition.  The ordering of the inputs is key: arguments, default, positional, keyword arguments.```def name_of_function(arg1, arg2, opt1=True, opt2="CS109", *args, **kwargs):    ...    return(output1, output2, ...)```Positional arguments are stored in a tuple, and keyword arguments in a dictionary.<jupyter_code>def f(a, b, c=5, *tupleargs, **dictargs):
    print("got", a, b, c, tupleargs, dictargs)
    return a
print(f(1,3))
print(f(1, 3, c=4, d=1, e=3))
print(f(1, 3, 9, 11, d=1, e=3)) # try calling with c = 9 to see what happens!<jupyter_output><empty_output><jupyter_text>Functions are first classPython functions are *first class*, meaning that we can pass functions to other functions, built-in or user-defined.<jupyter_code>def sum_of_anything(x, y, f):
    print(x, y, f)
    return(f(x) + f(y))
sum_of_anything(3,4,square)<jupyter_output><empty_output><jupyter_text>Finally, it's important to note that any name defined in this notebook is done at the *global* scope.  This means if you define your own `len` function, you will overshadow the system `len.` >**EXERCISE:** Create a dictionary, called `ps_dict`, that contains with the primes less than 100 and their corresponding squares.<jupyter_code># your code here<jupyter_output><empty_output><jupyter_text>Part 6. NumpyScientific Python code uses a fast array structure, called the numpy array. Those who have worked in Matlab will find this very natural. Let's make a numpy array.<jupyter_code>my_array = np.array([1, 2, 3, 4])
my_array<jupyter_output><empty_output><jupyter_text>Numpy arrays are listy. Below we compute length, slice, and iterate. But these are very bad ideas, for efficiency reasons we will see in lab 1.<jupyter_code>print(len(my_array))
print(my_array[2:4])
for ele in my_array:
    print(ele)<jupyter_output><empty_output><jupyter_text>In general you should manipulate numpy arrays by using numpy module functions (`np.mean`, for example). You can calculate the mean of the array elements either by calling the method `.mean` on a numpy array or by applying the function np.mean with the numpy array as an argument.<jupyter_code>print(my_array.mean())
print(np.mean(my_array))<jupyter_output><empty_output><jupyter_text>You can construct a normal distribution with mean0 and standard deviation 1 by doing:<jupyter_code>normal_array = np.random.randn(1000)
print("The sample mean and standard devation are %f and %f, respectively." %(np.mean(normal_array), np.std(normal_array)))<jupyter_output><empty_output><jupyter_text>Numpy supports a concept known as broadcasting, which dictates how arrays of different sizes are combined together. There are too many rules to list here, but importantly, multiplying an array by a number multiplies each element by the number. Adding a number adds the number to each element.This means that if you wanted the distribution $N(5, 7)$ you could do:<jupyter_code>normal_5_7 = 5 + 7*normal_array
np.mean(normal_5_7), np.std(normal_5_7)<jupyter_output><empty_output><jupyter_text>Part 7: Text Analysis of Hamlet (try it at home)Below we'll walk you through an exercise analyzing the word distribution of the play *Hamlet* using what we learned in this lab.  First we need to read the file into Python.  If you haven't done so already, please download `hamlet.txt` from the github repo and save it in the `data` subfolder. (You could just clone and download a zip of the entire repository as well). >**EXERCISE**:  Read the [documentation](https://docs.python.org/3/) on reading and writing files.  Open and read in and save `hamlet.txt` as `hamlettext`.  Then close the file.<jupyter_code># your code here<jupyter_output><empty_output><jupyter_text>>**EXERCISE**:  Before moving on, let's make sure `hamlettext` actually contains the text of *Hamlet*.  First, what is the type of `hamlettext`?  What is its length?  Print the first 500 items of `hamlettext`.<jupyter_code># your code here<jupyter_output><empty_output><jupyter_text>Each item in `hamlettext` is a text character.  Let's see what we can learn about the distribution of words in the play.  To do this, you'll need to split the string `hamlettext` into a list of strings, where each element of the list is a word in the play.  Search the documentation, or refer back to the summary of string operations in the Strings section of this lab.>**EXERCISE**:  Create a list called `hamletwords` where the items of the `hamletwords` are the words of the play.  Confirm that `hamletwords` is a list, that each element of `hamletwords` is a string, and print the first 10 items in `hamletwords`.  Print "There are $N$ total words in Hamlet," where $N$ is the total number of words in Hamlet.<jupyter_code># your code here<jupyter_output><empty_output><jupyter_text>The total number of words above considers "The" and "the" as two different words.  So if we're going to find the distribution of the words in the play, we need to first make all the words lower-case.  From the lower-case list of words, we then find the set of unique words.>**EXERCISE**: Using a *list comprehension*, create a `hamletwords_lc` which converts the items in `hamletwords` to lower-case.  You will likely want to search the documentation for the string method `lower`.  Then count the number of occurences of "thou", making use of the string method `count`.<jupyter_code># your code here<jupyter_output><empty_output><jupyter_text>We need to find the unique set of words in `hamletwords_lc`.  >**EXERCISE**:  Use `set` to determine the set of unique words in `hamletwords_lc`. Then print "There are $M$ unique words in Hamlet," where $M$ is the number of unique words.  As a sanity check, verify that $M < N$.<jupyter_code># your code here<jupyter_output><empty_output><jupyter_text>Next we want a dictionary with the number of occurrences of each unique word, with the unique words as dictionary keys.>**EXERCISE**:  Create this dictionary.  To do it you will use either a loop or a dictionary comprehension and the `count` method.<jupyter_code># your code here<jupyter_output><empty_output><jupyter_text>Finally, we find the 100 most frequently occurring words using the built-in `sorted` function.>**EXERCISE**: *(Stretch Goal)* Search and read the documentation for the `sorted` function.  The documentation has a link to some examples, "sorting HOW TO."  Check out the example under the heading "Key Functions."  Then create a list called `top100words` with the 100 most frequently occurring words in Hamlet.<jupyter_code># your code here<jupyter_output><empty_output><jupyter_text>Now we'll plot the top distribution of the top 10 words on this list.  To do this you'll need to import a plotting library called `matplotlib`.  The `pyplot` module in this library contains a set of functions that are especially useful for generating a wide range of plots.  Go back to the first cell in this notebook and type    ````    import matplotlib    import matplotlib.pyplot as plt    ````In this cell, also type    ````    %matplotlib inline    ````Be sure to evaluate the cell before continuing.  Below is the code to generate the bar chart.<jupyter_code>%matplotlib inline
import matplotlib.pyplot as plt
topfreq = top100words[:10]
pos = range(len(topfreq)) # get positions on graph
plt.bar(pos, [e[1] for e in topfreq]); # get second members of tuples, plot against positions
plt.xticks([e+0.4 for e in pos], [e[0] for e in topfreq]); # get labels as first members of tuples
plt.ylabel('Frequency')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08505434782608695, 'fraction_numerical': 0.014039855072463768, 'mean_word_length': 5.146038226015959, 'pattern_counts': {'":': 1, '<': 284, '<?xml version=': 0, '>': 303, 'https://': 5, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '71', 'dir': 'jupyter-structured-clean-dedup', 'id': '287945', 'license': 'Apache-2.0', 'n_tokens_mistral': 10314, 'n_tokens_neox': 9752, 'n_words': 4327, 'path': 'Lab_IntroPython.ipynb', 'repo_name': 'beccycole/programming-itsligo', 'size': '33143'}