<jupyter_start><jupyter_text>Processing Multiple Files and Writing Files================================== OverviewQuestions             How do I analyze multiple files at once?    Objectives:             Import a python library.         Use python library funtions.         Process multiple files using a `for` loop.         Print output to a new text file.    Key Points:             Use the glob function in the python library `glob` to find all the files you want to analyze.         You can have multiple `for` loops nested inside each other.         Python can only print strings to files.         You must close files so python will actually write them. You can do this either manually or automatically.          Processing multiple filesIn our previous lesson, we parsed values from output files.  While you might have seen the utility of doing such a thing, you might have also wondered why we didn't just search the file and cut and paste the values we wanted into a spreadsheet.  If you only have 1 or 2 files, this might be a very reasonable thing to do.  But what if you had 100 files to analyze?  What if you had 1000?  In such a case the cutting and pasting method would be very tedious and time consuming.  One of the real powers of writing a program to analyze your data is that you can just as easily analyze 100 files as 1 file.  In this example, we are going to parse the PDB files for a series of enzyme structures in the [Protein Data Bank](https://www.rcsb.org/) and extract resolution data and atom counts for each one.  The PDB files are all saved in a folder called PDB_files that you should have downloaded in the setup for this lesson.  Make sure the folder is in the same directory as the directory where you are writing and executing your code.To analyze multiple files, we will need to import a python **library**.  A **library** is a set of modules which contain functions. The functions within a library or module are usually related to one another. Using libraries in Python reduces the amount of code you have to write. In the last lesson, we imported `os.path`, which was a module that handled filepaths for us.In this lesson, we will be using the `glob` library, which will help us read in multiple files from our computer.  Within a library there are modules and functions which do a specific computational task.  Usually a function has some type of input and gives a particular output.  To use a function that is in a library, you often use the dot notation introduced in the previous lesson.```pythonimport library_nameoutput = library_name.function_name(input)```  Importing librariesWe are going to import two libraries.  One is the `os` library which controls functions related to the operating system of your computer. We used this library in the last lesson to handle filepaths.  The other is the `glob` library which contains functions to help us analyze multiple files.  If we are going to analyze multiple files, we first need to specify where those files are located. Check your understanding    How would you use the `os.path` module to point to the directory where your PDB files are located?    ```{admonition} Solution:class: dropdown```python        outfile = os.path.join('data', 'PDB_files')    ```   In order to get all of the files which match a specific pattern, we will use the wildcard character `*`.<jupyter_code>import os
file_location = os.path.join('data', 'PDB_files', '*.pdb')
print(file_location)<jupyter_output><empty_output><jupyter_text>This specifies that we want to look for all the files in a directory called `data/PDB_files` that end in ".pdb".  The * is the wildcard character which matches any character or series of characters.  Next we are going to use a function called `glob` in the library called `glob`.  It is a little confusing since the function and the library have the same name, but we will see other examples where this is not the case later.  The output of the function `glob` is a list of all the filenames that fit the pattern specified in the input.   The input is the file location.<jupyter_code>import glob
filenames = glob.glob(file_location)
print(filenames)<jupyter_output><empty_output><jupyter_text>Reading multiple files with nested for loopsNow you have a list of all the files which end in `*.pdb` in the `PDB_files` directory. To parse every file we just read in, we will use a `for` loop to go through each file.<jupyter_code>for f in filenames:
    with open(f, 'r') as outfile:
        data = outfile.readlines()
        
    for line in data:
        if 'RESOLUTION.' in line:
            res_line = line
            words = res_line.split()
            resolution = float(words[3])
            print(resolution)<jupyter_output><empty_output><jupyter_text>Notice that in this code we actually used two `for` loops, one nested inside the other.  The outer `for` loop counts over the filenames we read in earlier.  The inner `for` loop counts over the line in each file, just as we did in our previous file parsing lesson.  The output our code currently generates is not that useful.  It doesn't show us which file each resolution value came from.  We want to print the name of the molecule with the resolution. We can use `os.path.basename`, which is another function in `os.path` to get just the name of the file.<jupyter_code>first_file = filenames[0] # look above to recall the content of filenames
print(first_file)

file_name = os.path.basename(first_file)
print(file_name)<jupyter_output><empty_output><jupyter_text>Check your understanding    How would you extract the PDB ID from the example above? ```{admonition} Solution:class: dropdownYou can use the str.split function introduced in the last lesson, and split at the '.' character.split_filename = file_name.split('.')  molecule_name = split_filename[0]  print(molecule_name)  ``` Using the solution above, we can modify our loop so that it prints the file name along with each resolution value.<jupyter_code>for f in filenames:
    # Get the PDB ID
    file_name = os.path.basename(f)
    split_filename = file_name.split('.')
    molecule_name = split_filename[0]

    # Read the data
    with open(f,"r") as outfile:
        data = outfile.readlines()

    # Loop through the data
    for line in data:
        if 'RESOLUTION.' in line:
            res_line = line
            words = res_line.split()
            resolution = float(words[3])
            print(molecule_name, ": ", resolution, " Angstroms", sep = "")<jupyter_output><empty_output><jupyter_text>Printing to a FileFinally, it might be useful to print our results in a new file, such that we could share our results with colleagues or e-mail them to our advisors.  Much like when we read in a file, the first step to writing output to a file is opening that file for writing.  In general, to open a file for writing you have two options. The first is uses the `open` command.```pythonfilehandle = open('file_name.txt', 'w+'):take some actionsuse filehandle.write('content') to the filefilehandle.close()```The filehandle.close() command is very important here. Think about a computer as someone who has a very good memory, but is very slow at writing. Therefore, when you tell the computer to write a line, it remembers what you want it to write, but it doesn't actually write the new file until you tell it you are finished. The datafile.close() command tells the computer you are finished giving it lines to write and that it should go ahead and write the file now. If you are trying to write a file and the file keeps coming up empty, it is probably because you forgot to close the file. The second (preferred) approach in python is to use the `with` context manager that we have already used for reading a file. The advantage to this approach is that all of the steps that generate input for the file are indented under the initial `with` statement and the file closes automatically when all of the actions indented beneath the `with` statement are completed.```pythonwith open('file_name.txt', 'w') as filehandle:    take some actions    use filehandle.write('content') to add content the file```Let's examine the syntax of the `with` statement.```pythonwith open('file_name.txt', 'w') as filehandle:```The `w` instructs python to open the file for writing.  If you use `w+` that means open the file for writing and if the file does not exist, create it.  You can also use `a` for append to an existing file or `a+`.  The difference between `w+` and `a+` is that `w+` will overwrite the file if it already exists, whereas `a+` will keep what is already there and just add additional text to the file.  Python can only write strings to files.  In the next cell, we want to print the contents of two variables, `molecule_name` and `resolution`.  To convert what we have now to a string, you place a capital **F** in front of the line you want to print and enclose the content to be printed in single quotes.  Each python variable is placed in braces `{}`. Then you can either print the line (as we have done before) or you can use the `filehandle.write()` command to print it to a file.To make the printing neater, we will separate the PDB ID from the resolution using a tab. To insert a tab, we use the special character `\t`.<jupyter_code>with open('resolutions.txt', 'w+') as datafile:
    for f in filenames:
        # Get the PDB ID
        file_name = os.path.basename(f)
        split_filename = file_name.split('.')
        molecule_name = split_filename[0]
        
        # Read the data
        with open(f,"r") as outfile:
            data = outfile.readlines()  
            
        # Loop through the data
        for line in data:
            if 'RESOLUTION.' in line:
                res_line = line
                words = res_line.split()
                resolution = float(words[3])
                datafile.write(F'{molecule_name} \t {resolution} \n')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06475615887380594, 'fraction_numerical': 0.002011060834590246, 'mean_word_length': 3.6541881141787553, 'pattern_counts': {'":': 1, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '561863', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2576, 'n_tokens_neox': 2501, 'n_words': 1438, 'path': 'biochemist-python/chapters/Processing_Multiple_Files_and_Writing_Files.ipynb', 'repo_name': 'janash/python-scripting-biochemistry', 'size': '9945'}