<jupyter_start><jupyter_text>Python basics continuing on with part 2 ....See [here](https://github.com/fomightez/Python_basics_4nanocourse) for information about this notebook.----------  First make sure file to work on is here:<jupyter_code>!curl -OL https://files.rcsb.org/download/1p3v.pdb<jupyter_output><empty_output><jupyter_text>Slicing short-cut syntax and using indexing/slicing from end of lineWe saw before something like this:<jupyter_code>with open("1p3v.pdb", 'r') as input:
    for line in input:
        print(line[31-1]) 
        print(line[0:31])<jupyter_output><empty_output><jupyter_text>Where `print(line[0:31])` gets everything up to but not including the item at the 31st index.Because we are starting with zero, we can leave that off here because it will default to that.<jupyter_code>with open("1p3v.pdb", 'r') as input:
    for line in input:
        # print(line[31-1]) 
        print(line[:31])<jupyter_output><empty_output><jupyter_text>Likewise if we want to go all the way to the end, leave off the number after the colon:<jupyter_code>with open("1p3v.pdb", 'r') as input:
    for line in input:
        # print(line[31-1]) 
        print(line[30:])<jupyter_output><empty_output><jupyter_text>If negative numbers are used you can get back from the end of the line. First with a single letter using indexing:<jupyter_code>with open("1p3v.pdb", 'r') as input:
    for line in input:
        # print(line[31-1]) 
        if line[-7] != " ":
            print(line[-7])<jupyter_output><empty_output><jupyter_text>Let's use slicing back from end of the line to put that in context:<jupyter_code>with open("1p3v.pdb", 'r') as input:
    for line in input:
        # print(line[31-1]) 
        if line[-7] != " ":
            print(line[-7])
            print(line[-7:].strip()) #Strip off the new line character, or you get an additional line in output<jupyter_output><empty_output><jupyter_text>**Better way to show?**:<jupyter_code>input='''0123456789
ABCDEFGHIJKLMNOPQRSTUVWXYZ
0123456789'''
for line in input.split("\n"):
    # print(line[31-1]) 
    if line[-3] != " ":
        print(line[-3])
        print(line[-3:])<jupyter_output><empty_output><jupyter_text>Lists and dictionaries for data handling`split()` in code we just ran illustrated making a list splititng on new line character.For example from the split like above we can see the list if we look at it.<jupyter_code>input='''0123456789
ABCDEFGHIJKLMNOPQRSTUVWXYZ
0123456789'''
l = input.split("\n")
print(type(l))
print(l)<jupyter_output><empty_output><jupyter_text>Making a set from a list will give of us the set of unique items in the list.<jupyter_code>input='''0123456789
ABCDEFGHIJKLMNOPQRSTUVWXYZ
0123456789'''
l = input.split("\n")
print(type(l))
print(l)
s = set(l)
print(type(s))
print(s)<jupyter_output><empty_output><jupyter_text>Dictionaries have unique keys linked to values.Examples:```python protein names as keys:d = {"Svu1p":"LVALACHYRSITWP",    "Svu2p":"LVALACHYRSITWP",    "Svu3p":"LVALACIYASITWN"} accession numbers as keys:a = {"GB1567661":"LVALACHYRSITWP",    "GB1567662":"LVALACHYRSITWP",    "GB1567663":"LVALACIYASITWN"}```  `for` loops and `enumerate()` to iterate on listsWe covered `for` loops in session 3 and you've seen several today.This is one place were whitespace is important. (`if` conditionals are another.)Let's illustrate that by writing a for loop to iterate over the `input` we've been using lately and splitting on new line character to make a list.We'll write a `for` loop to print the contents of each line.<jupyter_code>input='''0123456789
ABCDEFGHIJKLMNOPQRSTUVWXYZ
0123456789'''
l = input.split("\n")<jupyter_output><empty_output><jupyter_text>A useful addition to this is to use `enumerate()` on the list to get both the current index and current item. Then we can use that to do useful stuff. Often I use it to make file names specific to a point in the iteration. We'll use the shell command `ls` to list files in the directory before and after.<jupyter_code>!ls file*
input='''0123456789
ABCDEFGHIJKLMNOPQRSTUVWXYZ
0123456789'''
l = input.split("\n")
for indx, i in enumerate(l):
    filename = "file_"+str(indx)+".txt"
    with open(filename, 'w') as output:
        output.write(i)
!ls file*<jupyter_output><empty_output><jupyter_text>If you check the contents of each generated file they should now have the corresponding line in the list.Note that we had to cast the integer index to a string to concatenate it to the file name. (There are easier ways to do that now using `f-strings` but we'll save that for you to learn about later.) The way we did things here is very explicit.Note if you'd prefer your file names start with one, you could replace filename assignment with the following before running that code so you end up with `file_1.txt  file_2.txt	file_3.txt`:```pythonfilename = "file_"+str(indx)+".txt"``` You can also iterate over a dictionary. Let's use our dictionary where the keys corresponded to accessions to save files.<jupyter_code>!ls *.fa
# accession numbers as keys:
a = {"GB1567661":"LVALACHYRSITWP",
    "GB1567662":"LVALACHYRSITWP",
    "GB1567663":"LVALACIYASITWN"}
for k, v in a.items():
    filename = k+".fa"
    txt = ">"+k+"\n"+v
    with open(filename, 'w') as output:
        output.write(txt)
!ls *.fa<jupyter_output><empty_output><jupyter_text>You'll see FASTA files made from those sequences for each accession number now.  Adding items to a list and dictionary and accessing items in a dictionaryYou use the list method `append()` to add an item to a list.<jupyter_code>input='''0123456789
ABCDEFGHIJKLMNOPQRSTUVWXYZ
0123456789'''
l = input.split("\n")
l.append("Added this text")
print(l)<jupyter_output><empty_output><jupyter_text>You use the key and assign an additional item to a dictionary.<jupyter_code># accession numbers as keys:
a = {"GB1567661":"LVALACHYRSITWP",
    "GB1567662":"LVALACHYRSITWP",
    "GB1567663":"LVALACIYASITWN"}
print(a)
a["GB1568820"] = "GAVALATIYASITWS"
print(a)<jupyter_output><empty_output><jupyter_text>We've already seen how to access items in a list. We either use the index of the item, such as `line[16]` or iterate through the list looking for items that meet our conditions. (There is a way to also get the index of the first occurence of an item in a list if we only know the item we are looking for.)What about a dictionary?We can just use the key to get the value.<jupyter_code># accession numbers as keys:
a = {"GB1567661":"LVALACHYRSITWP",
    "GB1567662":"LVALACHYRSITWP",
    "GB1567663":"LVALACIYASITWN"}
print(a["GB1567662"])<jupyter_output><empty_output><jupyter_text>Alternatively we can use the `get()` method of a dicitonary object.<jupyter_code>print(a.get("GB1567662"))<jupyter_output><empty_output><jupyter_text>This second one has the ability to assign something special to return if the key doesn't exist.<jupyter_code>print(a.get("GB9999999", "This accession number is not represented in the data."))<jupyter_output><empty_output><jupyter_text>And then you could use an `if` / conditional to check if you get that and react in a special way if necessary.  dataframesPanel data, a.k.a, dataframes, are another useful data handling Python object. They aren't included in 'base' Python.We have to import a popular Python package, called 'Pandas' to handle them.  'base' Python and packages / libraries / modulesYou get 'base' python by default.Add other stuff beyond base python and what you code by importing modules/packages/libraries.This adds a lot of convenience because you aren't stuck re-inventing the wheel from scratch. You can plug in to code developed by others.Some included as built-in to Python, but you have to still import them, such as `math` which gives you pi as a constant.  Why if built-in to Python, not included in 'base'? The idea is that you don't wanted a crowded namespace. It's kind of an artitificial case, but imagine someone was coding where they had pi defined as soemthing involving interpreting ancient greek letters that match that glyph, that would conflict or maybe lead to odd results if the math pi was already defined as pi. This is why you want to keep the namespace environment free and barebones at the start.Lots of not built-ins importable modules/packages/libraries are made by third parties. (You use package manager, [pip](https://pypi.org/project/pip/) to easily install them on system before you import import them. What we'll import today has already been installed in this remote system.)Some are super popular , essential for Python work, like numpy/matplotlib/pandas, others by big companies like Netflix (papermill; scrapbook). Lots just done by one a few folks.   Import pandas and make a dataframeSo let's import Pandas and make a dataframe to illustrate this. (Keep in mind. Pandas has already been installed using `pip` in the environment you are working in.)<jupyter_code>import pandas as pd
# accession numbers as keys:
a = {"GB1567661":"LVALACHYRSITWP",
    "GB1567662":"LVALACHYRSITWP",
    "GB1567663":"LVALACIYASITWN"}
df = pd.DataFrame(list(a.items()), 
    columns=['accession', 'aas'])
df<jupyter_output><empty_output><jupyter_text>Pandas is like what you'd get if combined a spread sheet with Python. It is great for data wrangling and analyses. We won't get into here but you can access data by row and column, iterate over any row or columns. Do operations across any row or column. You can even export to Excel if you absolutely need to.It is particularly useful for taking supplemental data published along with scientific articles and analyzing them further, example [here](https://github.com/fomightez/cl_sq_demo-binder/blob/master/notebooks/GSD/GSD%20Add_Supplemental_data_info_to_nt_count%20data%20for%201011_cerevisiae_collection.ipynb). Or taking data from BLAST and bringing it into Python, example [here](https://github.com/fomightez/blast-binder).  BioPandasFotunately, someone has already combined the power of Pandas for PDB file data wrangling.[BioPandas](http://rasbt.github.io/biopandas/)An associated article is [here](http://dx.doi.org/10.21105/joss.00279).Let's use it to look at a PDB file and make some plots<jupyter_code>from biopandas.pdb import PandasPdb
# Initialize a new PandasPdb object & fetch the PDB file from rcsb.org
ppdb = PandasPdb().fetch_pdb('3eiy')
# print first 10 lines of atoms
ppdb.df['ATOM'].head(10)
import matplotlib.pyplot as plt
from matplotlib import style
style.use('ggplot')
ppdb.df['ATOM']['b_factor'].plot(kind='hist')
plt.title('Distribution of B-Factors')
plt.xlabel('B-factor')
plt.ylabel('count')
plt.show()
ppdb.df['ATOM']['b_factor'].plot(kind='line')
plt.title('B-Factors Along the Amino Acid Chain')
plt.xlabel('Residue Number')
plt.ylabel('B-factor in $A^2$')
plt.show()
ppdb.df['ATOM']['element_symbol'].value_counts().plot(kind='bar')
plt.title('Distribution of Atom Types')
plt.xlabel('elements')
plt.ylabel('count')
plt.show()<jupyter_output><empty_output><jupyter_text>Modularity and DRY (Don't Repeat Yourself)Often you'll want to run the same processes over different data. Sometimes it will be in a loop, and so you could consider leaving the list of the steps there in the code. However, sometimes you may want to run those steps at one point in the whole list of steps and then maybe again later, not in that same loop. Or what if you write something you use quite often in many different scripts or notebooks with different data and slightly different options.If you could modularize your code into a portable unit that you could then invoke with different data inputs and options, then you could increase your efficiency dramatically.This is what Python functions allow. You define a function name and parameters you'll pass into it. And then you define a list of steps to run on input passed into those parameters when you call that function. Optionally, you can also have the function return something, such as a result when it is done. For example, if it was doing a calculation you can have it return the calculated value and assign that to a variable, or append it to a list or dictionary where you are collecting the results.**Whitespace matters here.**Okay, that sounds rather daunting at first but we've been building towards this today. We can put together some of our basics and make a function. We'll make something simple for now but hopefully you'll see you can keep building on that concept to make functions that do very complex analyses.As an example for today we'll make a function that calculates the number of histidines in a PDB file. This is based on a simple example we developed in Session 3 along the lines of:```pythonwith open("1p3v.pdb", 'r') as input:    for line in input:        if line.startswith("ATOM") and "CA" in line:            if 'HIS' in line:                print(line)```<jupyter_code># make sure we have the involved PDB files
pdb_ids = ["1p3v","1pin","1f8a","1f8a","1trn","1ehz"]
pdb_files = []
for id in pdb_ids:
    !curl -OL https://files.rcsb.org/download/{id}.pdb
    pdb_files.append(id + ".pdb") #there's fancier ways to do this automatically 
pdb_files<jupyter_output><empty_output><jupyter_text>Make the function to take the PDB file and return the number of histidines. This will get you started. You'll need to edit it:<jupyter_code>def histidine_count(pdb_fn):
    '''
    This function takes a filename of a PDB file and
    opens that file and counts the number of histidines.
    
    Returns:
    Count of the number of histidines
    '''
    with open("1p3v.pdb", 'r') as input:
    for line in input:
        if line.startswith("ATOM") and "CA" in line:
            if 'HIS' in line:
                print(line)<jupyter_output><empty_output><jupyter_text>Let's use the function now to do some analyses:<jupyter_code>histidine_counts = []
for pdb_file in pdb_files:
    histidine_counts.append(histidine_count(pdb_file))
print("The maximum number of histidines among the PDB files analyzed is:",max(histidine_counts))
print("The minimum number of histidines among the PDB files analyzed is:",min(histidine_counts))
histidine_counts<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09518072289156626, 'fraction_numerical': 0.025372076541459957, 'mean_word_length': 4.899247491638796, 'pattern_counts': {'":': 21, '<': 89, '<?xml version=': 0, '>': 90, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '544792', 'license': 'MIT', 'n_tokens_mistral': 4636, 'n_tokens_neox': 4285, 'n_words': 1702, 'path': 'basics_pt2.ipynb', 'repo_name': 'fomightez/Python_basics_4nanocourseRISE', 'size': '14110'}