<jupyter_start><jupyter_text>Midterm ReviewCSCI 1360E: Foundations for Informatics and Analytics  Material  - Anything in Lectures 1 through 10 are fair game!  - Anything in assignments 1 through 4 are fair game!  Topics Data Science - Definition - Intrinsic interdisciplinarity - "Greater Data Science" Python Language - Philosophy - Compiled vs Interpreted - Variables, literals, types, operators (arithmetic and comparative) - Casting, typing system - Syntax (role of whitespace) Data Structures - Collections (lists, sets, tuples, dictionaries) - Iterators, generators, and list comprehensions - Loops (`for`, `while`), loop control (`break`, `continue`), and utility looping functions (`zip`, `enumerate`) - Variable unpacking - Indexing and slicing - Differences in indexing between collection types (tuples versus sets, lists versus dictionaries) Conditionals - `if` / `elif` / `else` structure - Boolean algebra (stringing together multiple conditions with `or` and `and`) Exception handling - `try` / `except` structure, and what goes in each block Functions - Defining functions - Philosophy of a function - Defining versus calling (invoking) a function - Positional (required) versus default (optional) arguments - Keyword arguments - Functions that take any number of arguments - Object references, and their behaviors in Python NumPy - Importing external libraries - The NumPy `ndarray`, its properties (`.shape`), and indexing - NumPy submodules - Vectorized arithmetic in lieu of explicit loops - NumPy array dimensions, or *axes*, and how they relate to the `.shape` property - Array broadcasting, uses and rules - Fancy indexing with boolean and integer arrays  Midterm Logistics  - The format will be very close to that of JupyterHub assignments (there may or may not be autograders to help).  - It will be **90 minutes**. Don't expect any flexibility in this time limit, so plan accordingly.  - You are **NOT** allowed to use internet resources or collaborate with your classmates (enforced by the honor system), but you **ARE** allowed to use lecture and assignment materials from this course, as well as terminals in the JupyterHub environment or on your local machine.  - I will be available on Slack for questions most of the day tomorrow, from 9am until about 3pm (then will be back online around 4pm until 5pm). Shoot me a direct message if you have a conceptual / technical question relating to the midterm, and I'll do my best to answer ASAP.  JupyterHub Logistics  - The midterm will be released on JupyterHub at **12:00am on Thursday, June 29**.  - It will be collected at **12:00am on Friday, June 30**. The release and collection will be done by automated scripts, so believe me when I say there won't be any flexibility on the parts of these mechanisms.  - Within that 24-hour window, you can start the midterm (by "Fetch"-ing it on JupyterHub) whenever you like.  - **ONCE YOU FETCH THE MIDTERM, YOU WILL HAVE 90 MINUTES FROM THAT MOMENT TO SUBMIT THE COMPLETED MIDTERM BACK.**  - Furthermore, it's **up to you** to keep track of that time. Look at your system clock when you click "Fetch", or use the timer app on your smartphone, to help you track your time use. Once the 90 minutes are up, the exam is considered late.  - In theory, this should allow you to take the midterm when it is most convenient for you. Obviously you should probably start no later than 10:30PM tomorrow, since any submissions after midnight on Friday will be considered late, even if you started at 11:58PM.  Tough Assignment Questions and Concepts  From A1 **Do NOT hard-code answers!** For example, take the question on taking the square root of a number and converting it to a string:<jupyter_code>number = 3.14159265359<jupyter_output><empty_output><jupyter_text>Answering this is *not* simply taking what's in the autograder and copy-pasting it into your solution:<jupyter_code>number = "1.7724538509055743"<jupyter_output><empty_output><jupyter_text>The whole point is that your code should *generalize* to any possible input. To that end, you want to perform the actual operations required: as stated in the directions, this involves taking the square root and converting the answer to a string:<jupyter_code>number = 3.14159265359
number = number ** 0.5  # Raise to the 0.5, which means square root.
number = str(number)    # Cast to a string.<jupyter_output><empty_output><jupyter_text>**With great looping power comes great looping responsibility**. The question that involved finding the first negative number in a list of numbers gave a lot of folks problems. By that, I mean folks combined simultaneous `for` and `while` loops, inadvertently creating more problems with some very difficult-to-follow program behavior. Thing to remember: **both loops can solve the same problems**, but they lend themselves to different ones. So in almost all cases, you'll only need 1 of them to solve a given problem. In this case: if you need to perform operations on every element of a list, `for` is your friend. If you need to do something repeatedly *until some condition is satisfied*, `while` is your operator. This question better fits the latter than the former.<jupyter_code>def first_negative(numbers):
    num = 0
    
    index = 0
    while numbers[index] > 0:
        index += 1
    num = numbers[index]
    
    return num
first_negative([1, 2, 3, -1])
first_negative([10, -10, -100, -50, -75, 10])<jupyter_output><empty_output><jupyter_text>From A2 **`zip()` is an amazing mechanism for looping with MULTIPLE collections at once** There were very few students who deigned to use `zip()`; if you can learn how to use it, it will make your life considerably easier whenever you need to loop through multiple lists at the same time. Take the question on computing magnitudes of 3D vectors.<jupyter_code>def compute_3dmagnitudes(X, Y, Z):
    magnitudes = []
    
    ### BEGIN SOLUTION
    
    ### END SOLUTION
    
    return magnitudes<jupyter_output><empty_output><jupyter_text>Since all three lists--`X`, `Y`, and `Z`--are the same length, you could run a `for` loop with an index through one of them, and use that index across all three. That would work just fine.<jupyter_code>def compute_3dmagnitudes(X, Y, Z):
    magnitudes = []
    
    length = len(X)
    for i in range(length):
        # Pull out the corresponding (x, y, z) coordinates.
        x = X[i]
        y = Y[i]
        z = Z[i]
        
        ### Do the magnitude computation ###
    
    return magnitudes<jupyter_output><empty_output><jupyter_text>...but it's very verbose, and can get a bit difficult to follow. If, instead, you use `zip`, you don't need to worry about using `range` or computing the `len` of a list or even extracting the right `x,y,z` from each index:<jupyter_code>def compute_3dmagnitudes(X, Y, Z):
    magnitudes = []
    
    for x, y, z in zip(X, Y, Z):
        pass
        ### Do the magnitude computation ###
    
    return magnitudes<jupyter_output><empty_output><jupyter_text>Look how much cleaner that is! The `zip` function is amazing. You can certainly get along without it, as shown in the previous slide, but it handles so much of that work for you, so I encourage you to practice using it. **Indentation in Python may be the most important rule of all.** I cannot overemphasize how important it is for your code indentation to be precise and exact. Indentation dictates whether a line of code is part of an `if` statement or not, part of a `for` loop or not, part of a `try` block or not, even part of a function or not. There were quite a few examples of code that was completely correct, but it wasn't indented properly--for example, it wasn't indented under a `for` loop, so the line was executed only once, *after* the `for` loop finished running.  From A3 **`if` statements don't always need an `else`.** I saw this a lot:<jupyter_code>def list_of_positive_indices(numbers):
    indices = []
    for index, element in enumerate(numbers):
        if element > 0:
            indices.append(index)
        else:
            pass  # Why are we here? What is our purpose? Do we even exist?
    return indices<jupyter_output><empty_output><jupyter_text>`if` statements are adults; they can handle being short-staffed, as it were. If there's literally nothing to do in an `else` clause, you're perfectly able to omit it entirely:<jupyter_code>def list_of_positive_indices(numbers):
    indices = []
    for index, element in enumerate(numbers):
        if element > 0:
            indices.append(index)
    return indices<jupyter_output><empty_output><jupyter_text>**An actual example of reference versus value.** This was the bonus question from A3 about building a list-of-lists matrix. Some of you had a very clever solution that *technically* worked, but would fail spectacularly the moment you actually tried to use the matrix built by the function. In short: rather than construct a matrix of 0s one element at a time, the strategy was to pre-construct a row of 0s, and then use just 1 loop to append this pre-built list a certain number of times.  It was clever in that it avoided the need for nested loops, which are certainly difficult to write and understand under the best of circumstances! But you'd see some odd behavior if you tried to *use* the matrix that came out...<jupyter_code>def make_matrix(rows, cols):
    pre_built_row = []
    
    # Build a single row that has <cols> 0s.
    for j in range(cols):
        pre_built_row.append(0)
        
    # Now build a list of the rows.
    matrix = []
    for i in range(rows):
        matrix.append(pre_built_row)

    return matrix        
m = make_matrix(3, 4)
print(m)<jupyter_output>[[0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0]]<jupyter_text>Certainly *looks* ok--3 "rows" (i.e. lists), each with 4 0s in them. Why is this a problem? Let's try changing one of the 0s to something else. Say, change the 0 in the upper left (position 0, 0) to a 99.<jupyter_code>m[0][0] = 99<jupyter_output><empty_output><jupyter_text>Now if we print this... what do you think we'll see?<jupyter_code>print(m)<jupyter_output>[[99, 0, 0, 0], [99, 0, 0, 0], [99, 0, 0, 0]]<jupyter_text>*cue The Thriller* This is a *pass-by-reference* problem. You've appended three references to the *same object*, so when you update one of them, you actually update them all. The fix is to use a nested-`for` loop to create everything on-the-fly:<jupyter_code>def make_matrix(rows, cols):
    matrix = []
    
    for i in range(rows):
        matrix.append([])  # First, append an empty list for the new row.
        for j in range(cols):
            matrix[i].append(0)  # Now grow that empty list.

    return matrix        
m = make_matrix(3, 4)
print(m)
m[0][0] = 99
print(m)<jupyter_output>[[99, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0]]<jupyter_text>From A4 **`len(ndarray)` versus `ndarray.shape`** For the question about checking that the lengths of two NumPy arrays were equal, a lot of people chose this route:<jupyter_code># Some test data
import numpy as np
x = np.random.random(10)
y = np.random.random(10)
len(x) == len(y)<jupyter_output><empty_output><jupyter_text>which works, but only for *one-dimensional arrays*. For anything other than 1-dimensional arrays, things get problematic:<jupyter_code>x = np.random.random((5, 5))  # A 5x5 matrix
y = np.random.random((5, 10))  # A 5x10 matrix
len(x) == len(y)<jupyter_output><empty_output><jupyter_text>These definitely are **not** equal in length. But that's because `len` doesn't measure *length* of matrices...it only measures the number of rows (i.e., the first axis--which in this case is 5 in both, hence it thinks they're equal). You definitely want to get into the habit of using the `.shape` property of NumPy arrays:<jupyter_code>x = np.random.random((5, 5))  # A 5x5 matrix
y = np.random.random((5, 10))  # A 5x10 matrix
x.shape == y.shape<jupyter_output><empty_output><jupyter_text>We get the answer we expect.  Other Questions from the Google Hangouts Review Session **The Tale of Two `for` Loops**<jupyter_code>import numpy as np

# Generate a random list to work with as an example.
some_list = np.random.random(10).tolist()
print(some_list)<jupyter_output>[0.6931832056888297, 0.6021857169653142, 0.03836475358595104, 0.3651604096030209, 0.2755364584885275, 0.30045938465583466, 0.47102308993526965, 0.8970256806073568, 0.15810029910258372, 0.21366871301862667]<jupyter_text>1: Looping through *elements*<jupyter_code>for element in some_list:
    print(element)<jupyter_output>0.6931832056888297
0.6021857169653142
0.03836475358595104
0.3651604096030209
0.2755364584885275
0.30045938465583466
0.47102308993526965
0.8970256806073568
0.15810029910258372
0.21366871301862667<jupyter_text>2: Looping over *indices* of elements<jupyter_code>list_length = len(some_list)
for index in range(list_length):  # [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
    element = some_list[index]
    print(element)<jupyter_output>0.6931832056888297
0.6021857169653142
0.03836475358595104
0.3651604096030209
0.2755364584885275
0.30045938465583466
0.47102308993526965
0.8970256806073568
0.15810029910258372
0.21366871301862667<jupyter_text>In general:  - If you don't care about list order, or where you are in the list--use "loop by element"  - If ordering of elements MATTERS, or where you are in the list during a loop is important--use "loop by index" **Sliding windows for finding substrings in a longer string** From A4, Bonus Part A:<jupyter_code>def count_substring(base_string, substring, case_insensitive = True):
    count = 0
    
    if case_insensitive == True:
        base_string = base_string.lower()
        #base_string = base_string.upper()
        
    length = len(substring)
    index = 0
    while (index + length) < len(base_string):
        
        # Sliding window.
        substring_to_test = base_string[index : (index + length)]
        if substring_to_test == substring:
            count += 1
            
        index += 1
    
    return count<jupyter_output><empty_output><jupyter_text>**Normalization by any other name** This confused some folks, namely because "normalization" can mean a lot of different things. In particular, two different types of normalization were conflated: 1: Rescale vector elements so the vector's magnitude is 1  - NOT the same thing as having all the vector elements SUM to 1 2: Rescale vector elements so they all sum to 1  - What the Bonus, Part B in A4 was actually asking for (even though the autograder was terrible) tl;dr These are both perfectly valid forms of normalization. It's just that the autograder was horrible. Here's what the spirit of the question was asking for:<jupyter_code>numbers = [10, 20, 30, 40]
print(sum(numbers))
numbers = [10/100, 20/100, 30/100, 40/100]
# (0.1 + 0.2 + 0.3 + 0.4)  = 1.0
print(sum(numbers))
import numpy as np
def normalize(something):

    # Compute the normalizing constant
    s = something.sum()
    
    # Use vectorized programming (broadcasting) to normalize each element
    # without the need for any loops
    normalized = (something / s)
    
    return normalized<jupyter_output><empty_output><jupyter_text>This can then be condensed into the 3 lines required by the question:<jupyter_code>import numpy as np  # 1
def normalize(something):  # 2
    return something / something.sum()  # 3<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08362097036795832, 'fraction_numerical': 0.05236079452946923, 'mean_word_length': 4.1289245156980625, 'pattern_counts': {'":': 0, '<': 85, '<?xml version=': 0, '>': 87, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '613604', 'license': 'MIT', 'n_tokens_mistral': 5112, 'n_tokens_neox': 4484, 'n_words': 1993, 'path': 'lectures/MidtermReview.ipynb', 'repo_name': 'eds-uga/csci1360e-su17', 'size': '15355'}