<jupyter_start><jupyter_text>Programming Bootcamp 2019 Lesson 4 Exercises--- **Required Files for this exercise*** genes.txt * init_sites.txt* gene_expr.txt* sequences2.txt**Earning points (optional)**If you would like to get points/feedback for your work, please **submit your notebook to Piazza**. To do this, follow these steps:1. Click "New Post"2. For "Post Type" select `Question`3. For "Post to" select `Individual Student(s)/Instructors(s)`4. Where is says "Enter one or more names..." type `Instructors`5. For "Select Folder(s)" select `lab4`6. In "Summary" type in "[Your Full Name] Lab4 Submission"7. In the "Details" click "Insert" > "Insert File" and then insert your python notebook.8. You can then write whatever else you want in the Summary.9. Then click "Post My Question to PROGRAMMING BOOTCAMP!":Please also **write your name below**. You do not need to complete all the problems to get points. Points will be assigned, but completion is all that actually matters. Those who consistenly participate throughout bootcamp will get a ~prize~. **Name**:  --- 1. Guess the output: list practice (1pt)For the following blocks of code, first try to guess what the output will be, and then run the code yourself. Points will be given for filling in the guesses; guessing wrong won't be penalized.<jupyter_code>ages = [65, 34, 96, 47]

print (len(ages))<jupyter_output><empty_output><jupyter_text>Your guess:<jupyter_code>names = ["Wilfred", "Manfred", "Wadsworth", "Jeeves"]
ages = [65, 34, 96, 47]

print (len(ages) == len(names))<jupyter_output><empty_output><jupyter_text>Your guess:<jupyter_code>ages = [65, 34, 96, 47]

for hippopotamus in ages:
    print (hippopotamus)<jupyter_output><empty_output><jupyter_text>Your guess:<jupyter_code>ages = [65, 34, 96, 47]

print (ages[1:3])<jupyter_output><empty_output><jupyter_text>Your guess:<jupyter_code>names = ["Wilfred", "Manfred", "Wadsworth", "Jeeves"]

if "Willard" not in names:
    names.append("Willard")
print (names)<jupyter_output><empty_output><jupyter_text>Your guess:<jupyter_code>names = ["Wilfred", "Manfred", "Wadsworth", "Jeeves"]
ages = [65, 34, 96, 47]

for i in range(len(names)):
    print (names[i],"is",ages[i])<jupyter_output><empty_output><jupyter_text>Your guess:<jupyter_code>ages = [65, 34, 96, 47]

ages.sort()
print (ages)<jupyter_output>[34, 47, 65, 96]<jupyter_text>Your guess:<jupyter_code>ages = [65, 34, 96, 47]

ages = None
print (ages)<jupyter_output>None<jupyter_text>Your guess:    > Remember that `.sort()` is an in-place function. Its return value is "None". (This is a special value in Python that basically means "null". It's used as a placeholder sometimes when we don't want to give something a value.)<jupyter_code>ages = [65, 34, 96, 47]

print (max(ages))<jupyter_output><empty_output><jupyter_text>Your guess:<jupyter_code>cat = "Mitsworth"

for i in range(len(cat)):
    print (cat[i])<jupyter_output><empty_output><jupyter_text>Your guess:<jupyter_code>cat = "Mitsworth"

print (cat[:4])<jupyter_output><empty_output><jupyter_text>Your guess:<jupyter_code>str1 = "Good morning, Mr. Mitsworth."

parts = str1.split()
print (parts)
print (str1)<jupyter_output><empty_output><jupyter_text>Your guess:<jupyter_code>str1 = "Good morning, Mr. Mitsworth."

parts = str1.split(",")
print (parts)<jupyter_output><empty_output><jupyter_text>Your guess:<jupyter_code>names = ["Wilfred", "Manfred", "Wadsworth", "Jeeves"]

print (names[-1])<jupyter_output><empty_output><jupyter_text>Your guess:<jupyter_code>oldList = [2, 2, 6, 1, 2, 6]
newList = []

for item in oldList:
    if item not in newList:
        newList.append(item)
print (newList)<jupyter_output><empty_output><jupyter_text>Your guess:  > This is an example of how to remove duplicates.   --- 2. On your own: Lists (3pts)Write code to accomplish each of the following tasks **using list functions**. Do not copy and paste the list to make changes. ***You must pretend you don't know what's in the list.***<jupyter_code># run this first!
geneNames = ["Ppia", "Gria2", "Mecp2", "Omd", "Zfp410", "Hsp1", "Mtap1a", "Cfl1", 
             "Slc25a40", "Dync1i1", "V1ra4", "Fmnl1", "Mtap2", "Atp5b", "Olfr259", 
             "Atf3", "Vapb", "Dhx8", "Slc22a15", "Orai3", "Ifitm7", "Kcna2", "Timm23", "Shank1"]<jupyter_output><empty_output><jupyter_text>**(A)** (1pt) Replace the 12th element in `geneNames` with the string `"Camk2a"`. **(B)** (1pt) Add the string `"Shank3"` to the end of the `geneNames` list. **(C)** (1pt) Using a loop, print the elements in `geneNames` that start with "S" or "O". --- 3. On your own: Strings & Splitting (3pts)Write code to accomplish each of the following tasks **using string slicing and splitting**.  **(A)** (1pt) Print the 3rd to 9th letters of the string stored in the variable `magicWords` using string slicing. (You should get "racadab")<jupyter_code>magicWords = "abracadabra"<jupyter_output><empty_output><jupyter_text>**(B)** (1pt) Prompt the user to input a sentence using `input()`. Print each word of their sentence on a separate line. **(C)** (1pt) Using `input()`, prompt the user to enter some good ideas for cat names (have them enter the names on one line, separating each name with a comma). Separate the names so that each one is a separate element of a list. Then choose one randomly (using the random module) and print it out. > Note: if you want to strip off the extra whitespace, you can do this:```print (catNamesList[randNum].strip(" "))``` --- 4. File reading and lists (3pts)For this problem, use the file `genes.txt` provided on Piazza. It contains some gene IDs, one on each line. **(A)** (1pt) Read the file and print to the screen only the gene IDs that contain "uc007".**[ Check your answer ]** You should end up printing 14 gene IDs. **(B)** (1 pt) Print to the screen only unique gene IDs (remove the duplicates). Do not assume repeat IDs appear consecutively in the file. *Hint: see problem 1 for an example of duplicate checking.***[ Check your answer ]** You should end up printing 49 gene IDs. **(C)** (1 pt) Print to the screen only the gene IDs that are still unique after removing the ".X" suffix (where X is a number).**[ Check your answer ]** You should end up printing 46 gene IDs. --- 5. Practice with `.split()` (4pts)Use `init_sites.txt` to complete the following. This file contains a subset of translation initiation sites in mouse, identified by Ingolia et al. (Cell, 2011). Note that this file has a header, which you will want to skip over. **(A)** (2 pt) Write a script that reads this file and computes the average CDS length (i.e. average the values in the 7th column).**[ Check your answer ]** You should get an average of 236.36. **(B)** (2 pt) Write a script that reads this file and prints the "Init Context" from each line (i.e. the 6th column) **if and only if** the "Codon" column (column 12) is "aug" for that line.**[ Check your answer ]** You should print 38 init contexts. **(C)** (0 pt) For fun (?), copy and paste your output from (b) into http://weblogo.berkeley.edu/logo.cgi to create a motif logo of the sequence around these initiation sites. What positions/nt seem to be most common? Your answer:  --- 6. Cross-referencing (4pts)Here you will extract and print the data from `init_sites.txt` that corresponds to genes with high expression. There isn't gene expression data in `init_sites.txt`, so we'll have to integrate information from another file.- First, use `gene_expr.txt` to create a list of genes with high expression. We'll say high expression is anything >= 50. - Then read through `init_sites.txt` and print the GeneName (2nd column) and PeakScore (11th column) from any line that matches an ID in your high-expression list. - Finally, separately compute the average PeakScore for high expression genes and non-high expression genes. Print both averages to the screen.**[ Check your answer ]** There should be 10 lines corresponding to high-expression genes that you print info about. Your average peak scores should be 4.371 and 4.39325 for high and non-high expression genes, respectively. --- 7. All-against-all comparisons (4pts)A common situation that arises in data analysis is that we have a list of data points and we would like to compare each data point to each other data point. Here, we will write a script that computes the "distance" between each pair of strings in a file and outputs a distance matrix. We will define "distance" between two strings as the number of mismatches between two strings when they are lined up, divided by their length.First we'll use a toy dataset. We'll create a list as follows:```things = [1, 2, 5, 10, 25, 50]```We'll start off by doing a very simple type of pairwise comparison: taking the numerical difference between two numbers. To systematically do this for all possible pairs of numbers in our list, we can make a nested for loop:<jupyter_code>things = [1, 2, 5, 10, 25, 50]

for i in range(len(things)):
    for j in range(len(things)):
        # print absolute value of the difference
        print (abs(things[i] - things[j]))<jupyter_output><empty_output><jupyter_text>Try running this code yourself and observe the output. Everything prints out on its own line, which isn't what we want -- we'd usually prefer a matrix-type format. Try this slightly modified code:<jupyter_code>things = [1, 2, 5, 10, 25, 50]

for i in range(len(things)):
    for j in range(len(things)):
        print (abs(things[i] - things[j]), "\t", end="")
    print ("")<jupyter_output><empty_output><jupyter_text>This gives us the matrix format we want. Make sure you understand how this code works. To make sure that the `print` function doesn't print a new line at the end of the string, in python3 we reset its end character to an empty string.  FYI, the "\t" is a tab character, and much like "\n", it is invisible once it's printed (it becomes a tab). So now we know how to do an all-against-all comparison. But how do we compute the number of mismatches between strings? As long as the strings are the same length, we can do something simple like the following:<jupyter_code>str1 = "Wilfred"
str2 = "Manfred"
diffs = 0

for k in range(len(str1)):
    if str1[k] != str2[k]: #compare the two strings at the same index
        diffs = diffs + 1
        
print ("dist =", round(float(diffs) / len(str1), 2))<jupyter_output><empty_output><jupyter_text>So this outputs the distance between the two strings, where the distance is defined as the fraction of the sequence length that is mismatched.**Using these two pieces of code as starting points, complete the following:** **(A)** (2 pt) Create a list of a few short strings *of the same length*. For example:```things = ["bear", "pear", "boar", "tops", "bops"]```Write code that prints a distance matrix for this list. As in the last example, use the fraction of mismatches between a given pair of words as the measure of their "distance" from each other. Round the distances to 2 decimals. **(B)** (2 pt) Now, instead of using a hard coded list like you did in (A), create a list of DNA sequences by reading in the file `sequences2.txt`. Compute the distance matrix between these sequences and print the distance matrix. Looking at this matrix, do you see a pair of sequences that are much less "distant" from each other than all the rest? --- 8. Writing and Using Custom Functions with Lists (5 pt) **(A)** (2pts) Create a function called "rand_seq" that takes an integer length as a parameter, and then returns a random DNA sequence of that length. *Hint: make a list of the possible nucleotides* **(B)** (2pts) Create a function called "shuffle_nt" that takes a single sequence as a parameter and returns a string that is a shuffled version of the sequence (i.e. the same nucleotides, but in a random order). *Hint: Look for Python functions that will make this easier. For example, the `random` module has some functions for shuffling. There may also be some built-in string functions that are useful. However, you can also do this just using things we've learned.* **(C)** (1pt) Run the unit tests below to show that all of your functions work. Try to fix any that have problems.<jupyter_code>##### testing rand_seq
randSeq1 = rand_seq(23)
randSeq2 = rand_seq(23)

if type(randSeq1) != str:
    print (">> Problem with rand_seq: answer is not a string, it is a %s." % type(randSeq1))
elif len(randSeq1) != 23:
    print (">> Problem with rand_seq: answer length (%s) does not match expected (%s)." % (len(randSeq1), 23))
elif randSeq1 == randSeq2:
    print (">> Problem with rand_seq: generated the same sequence twice (%s) -- are you sure this is random?" % randSeq1)
else:
    print ("rand_seq: Passed.")


##### testing shuffle_nt
shuffSeq = shuffle_nt("AAAAAAGTTTCCC")

if type(shuffSeq) != str:
    print (">> Problem with shuffle_nt: answer is not a string, it is a %s." % type(shuffSeq))
elif len(shuffSeq) != 13:
    print (">> Problem with shuffle_nt: answer length (%s) does not match expected (%s)." % (len(shuffSeq), 12))
elif shuffSeq == "AAAAAAGTTTCCC":
    print (">> Problem with shuffle_nt: answer is exactly the same as the input. Are you sure this is shuffling?")
elif shuffSeq.count('A') != 6:
    print (">> Problem with shuffle_nt: answer doesn't contain the same # of each nt as the input.")
else:
    print ("shuff_seq: Passed.")<jupyter_output><empty_output><jupyter_text>**(D)** (1pt) Use the **functions you created above** to create 20 random nucleotide sequences of length 50 and print them to the screen. ------ Extra questions (0pts)These questions are for people who would like extra practice. They will not be counted for points. **(A)** Following from problem 7 above: As you can see, the distance matrix you made in (B) is symmetrical around the diagonal. This means dist(i,j) is the same as dist(j,i), so we're doing some redundant calculations. Change the code so that we don't do any unnecessary calculations (including comparing a sequence to itself, which always is0). For any calculations you skip, you can print "-" or some other place-holder to keep the printed matrix looking neat. *Hint: There's a really simple way to do this! Think about the range of the second loop...* **(B)** Below is a loop that creates a list. Do the same thing but with a list comprehension instead.<jupyter_code># Loop version:
import random
randomNums = []

for i in range(100):
    randomNums.append(random.randint(0,10))
    
print (randomNums)
# Your list comprehension version here:<jupyter_output><empty_output><jupyter_text>**(C)** Below is a loop that creates a list. Do the same thing but with a list comprehension instead.<jupyter_code># Loop version:
import random
randomNums = []
for i in range(100):
    if (i % 2) == 0:
        randNum = random.randint(0,10)
        randNumStr = str(randNum)
        randomNums.append(randNumStr)
print (randomNums)
# Your list comprehension version here:<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10960005359415824, 'fraction_numerical': 0.020231794734373955, 'mean_word_length': 4.734921244717634, 'pattern_counts': {'":': 2, '<': 91, '<?xml version=': 0, '>': 110, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '514145', 'license': 'MIT', 'n_tokens_mistral': 4754, 'n_tokens_neox': 4408, 'n_words': 2049, 'path': 'class_materials/Data_Struture_List/2019/lab_exercise/lab4_exercises.ipynb', 'repo_name': 'sklasfeld/python_bootcamp', 'size': '14927'}