<jupyter_start><jupyter_text>An Introduction to Python for Physical Chemistry Python is a computer programming language that has become ubiquitous in scientific programming. Our initial lessons will run Python interactively through a Jupyter notebook.  First, a brief introduction to the Jupyter notebook. Notebooks are living documents that combine code with inline output and annotations, allowing you to combine your data analysis with your writing. In this way, you can ensure that your written reports always reflect the latest work you've done, as well as ensuring that you (and your audience) always know what steps you took while performing your analysis. Notebooks are made up of three main types of cells: - Markdown (used for general writing and annotation)- Code (used to write code which can be evaluated for computation or display purposes)- Output (generated by running code cells and displaying the result)You can change the type of a cell at any time by clicking the Cell Type button in the toolbar at the top of the notebook. (usually says "Code", but can also say "Markdown" or "Raw"). The contents of a cell appear as raw text until the cell is evaluated or "run". At that point, Jupyter processes the contents of the cell according to the cell type and outputs a result. To execute a cell, type Shift+Enter.   Markdown cells To start, let's introduce the idea of [Markdown](https://daringfireball.net/projects/markdown/). Markdown is a lightweight markup language, allowing you to write with simple formatting using only plain text (no fancy word processor required to display things like italics, bold, or links). Though it was invented in 2004, the language was standardized in 2014 under the [CommonMark](https://commonmark.org) moniker. A brief overview of the standardized syntax is available on the [CommonMark help page](https://commonmark.org/help/). Writing with Markdown lets us style text in all the ways possible with HTML without requiring many of the verbose tags that appear in HTML. One point to remember about Markdown is that it is a superset of HTML. In other words, any HTML code will also work in Markdown, so you can make text bold by typing `**bold text**` or `bold`.All of the text in this tutorial is written in Markdown. To se the source, just double click on any text cell. Try it on this one (double-click, look at the source text, maybe add your own comments, then evaluate with Shift+Enter).On execution, Markdown cells are processed into displayed text.  Code cells Jupyter notebooks are associated with a "kernel", which is just a term for a code interpreter. For our purposes, we will only be working with a Python kernel called "PChem Lab". The kernel can be changed within a notebook by clickin on the kernel name in the upper right corner of the notebook window. Upon execution, the contents of code cells are evaluated and any outputs are inserted into an Output cell directly beneath the code cell. By default, the output (if any) of the last line is displayed.   Note: Jupyter notebooksThe variables created in a cell are available for use by any cell evaluated after that point. Notice I say _evaluated_, rather than _located_. It is possible to evaluate Jupyter notebooks out of order (_i.e.,_ you could evaluate a cell near the bottom of a notebook, then evaluate the second cell in a notebook and use the information calculated in that bottom cell). As you work through labs, you might be tempted to do this… I urge you to resist this temptation. Similarly, if you open your Jupyter notebook later, and try to run a code block in the middle, it may tell you that your variables are undefined, even though you can clearly see them defined in earlier code blocks. If you didn’t _re-run_ those code blocks, then Python doesn’t know they exist. When in doubt about your notebook or the state of your calculations, use the Kernel menu options starting with "Restart Kernel and Run…" to rerun your notebook with an empty slate.   Structure of this lesson As you work through this lesson, you will be presented with code samples followed by an empty code cell. To practice writing code, _type_ (don't copy) the sample code into the code cell (make any necessary modifications), then execute the code cell to see the output. If you just copy and paste the sample code, you're losing out on valuable practice and muscle-memory development that will help you feel more comfortable with code later in the semester. The only person losing out in that situation is you… As a mentor once told me (about electronics), "Like mathematics and computer programming, this is not a spectator sport."   Python Basics  Data types and assigning variables Any Python interpreter can work just like a calculator. This is not very useful. Execute the following cell in the Jupyter notebook.<jupyter_code>3+7<jupyter_output><empty_output><jupyter_text>Here, Python has performed a calculation for us. To save this value, or other values, we assign them to a variable for later use. The syntax for assigning variables is the following:```pythonvariable_name = variable_value```Let’s see this in action with a calculation. Type the following into the next cell of your Jupyter notebook, then evaluate that cell.```pythondeltaH = -541.5   kJ/moledeltaS =  10.4    kJ/(mole * K)temp = 298        KelvindeltaG = deltaH - temp * deltaS```<jupyter_code># Input the sample code here<jupyter_output><empty_output><jupyter_text>Notice several things about this code. Text following the `` are comments. The computer does not do anything with these comments. They have been used here to remind the user what units each of their values are in. Comments are also often used to explain what the code is doing or leave information for future people who might use the code. You can make a whole line a comment (by making the first character a ``), or just leave a brief comment at the end of a line.When choosing variable names, you should choose informative names so that someone reading your code can tell what they represent. Naming a variable `temp` or `temperature` is much more informative than naming that variable `t`.We can now access any of the variables from other cells. Let’s print the value that we calculated. In the next cell, type then evaluate the following:```pythonprint(deltaG)``` In the previous code block, we introduced the `print()` function. Often, we will use this function just to make sure our code is working correctly.Note that if you do not specify a new name for a variable, then it doesn’t automatically change the value of the variable; this is called being immutable. For example evaluate the following:```pythonprint(deltaG)deltaG * 1000print(deltaG)``` Nothing happened to the value of `deltaG`. If we wanted to change the value of `deltaG` we would have to re-save the variable using the same name to overwrite the existing value.```pythonprint(deltaG)deltaG = deltaG * 1000print(deltaG)``` There are situations where it is reasonable to overwrite a variable with a new value, but you should always think carefully about this. Usually it is a better practice to give the variable a new name and leave the existing variable as is, like so:```pythonprint(deltaG)deltaG_joules = deltaG * 1000print(deltaG)print(deltaG_joules)```  Assigning several variables at once Python can do what is called multiple assignment where you assign several variables their values on one line of code. The following code block does the exact same thing as the previous code block. This structure of comma-separated variables inside parentheses is called "tuple", a data type we will discuss later in the lesson. In this situation, the parentheses are optional, but it's generally better to be explicit, rather than implicit in our statements. ```python I can assign all these variables at once(deltaH, deltaS, temp) = (-541.5, 10.4, 298)deltaG = deltaH - temp * deltaSprint(deltaG)```  Data types Each variable is some particular _type_ of data. The most common (and fundamental) types of data are strings (`str`), integers (`int`), and floating point numbers (`float`). Integers are, as the label suggests, integer valued numbers, _i.e.,_ positive or negative numbers with no decimal component (_e.g.,_ 1, 2, 524, -10000045). Floats are numbers with a decimal component (_e.g.,_ 0.25, 1005.5, -3284.673, 3.33333333). Strings can be simply thought of as text surrounded by quotes (note: Python doesn't distinguish between single and double quotes, so the following are equivalent: - `'sample string'`- `"sample string"`You can identify the data type of any variable with the function `type(variable_name)`.```pythontype(deltaG)``` You can change the data type of a variable like this. This is called casting.```pythondeltaG_string = str(deltaG)type(deltaG_string)```  Lists Another common data structure in Python is the list. Lists can be used to group several values or variables together, and are declared using square brackets `[ ]`. List values are separated by commas. Python has several built in functions which can be used on lists. The built-in function `len()` can be used to determine the length of a list. This code block also demonstrates how to print multiple variables.``` python This is a listenergy_kcal = [-13.4, -2.7, 5.4, 42.1] I can determine its lengthenergy_length = len(energy_kcal) print the list lengthprint('The length of this list is', energy_length)``` If I want to operate on a particular element of the list, you use the list name and then put in brackets which element of the list you want. **In Python, counting starts at zero.** So the first element of a list is `list_name[0]`.In the cell below, `print` the first value of the list created in the previous cell (`energy_kcal`).  You can use an element of a list as a variable in a calculation.```python Convert the second list element to kilojoules.energy_kilojoules = energy_kcal[1] * 4.184print(energy_kilojoules)```  Slices Sometimes you will want to make a new list that is a subset of an existing list. For example, we might want to make a new list that is just the first few elements of our previous list. This is called a `slice`. The general syntax is```pythonnew_list = list_name[start:end]```When taking a slice, it is very important to remember how counting works in Python. Remember that counting starts at zero so the first element of a list is `list_name[0]`. When you specify the last element for the slice, it goes _up to but not including_ that element of the list. So a slice like```pythonshort_list = energy_kcal[0:2]```includes `energy_kcal[0]` and `energy_kcal[1]` but not `energy_kcal[2]`. Print the contents of `short_list` below. If you do not include a start index, the start of the slice defaults to `list_name[0]`. If you do not include an end index, the slice automatically goes to the end of the list. What does the following code print? Try to guess before trying it for yourself. ```pythonslice1 = energy_kcal[1:]slice2 = energy_kcal[:3]print('slice1 is', slice1)print('slice2 is', slice2)``` If you don’t specify a new variable name nothing happens. Looking at our example above if we only type```pythonprint(energy_kcal)energy_kcal[0:2]print(energy_kcal)```nothing happens to `energy_kcal`.  Repeating an operation many times: for loops Often, you will want to do something to every element of a list. The structure to do this is called a `for` loop. The general structure of a `for` loop is```pythonfor variable in list:    do things using variable```There are two very important pieces of syntax for the `for` loop. Notice the colon `:` after the word list. You will always have a colon at the end of a `for` statement. If you forget the colon, you will get an error when you try to run your code.The second thing to notice is that the lines of code under the `for` loop (the things you want to do several times) are indented. Indentation is very important in Python. There is nothing like an `end` or `exit` statement that tells you that you are finished with the loop. The indentation shows you what statements are in the loop. Each indentation is 4 spaces by convention in Python 3. However, if you are using an editor which understands Python, it will do the correct amount of indentation for you when you press the Tab key on your keyboard. In fact, some editors (like the Jupyter notebook you're currently using) will notice that you used a colon (`:`) in the previous line, and will indent for you (so you will not need to press Tab). Let’s use a loop to change all of our energies in kcal to kJ.```pythonfor number in energy_kcal:    kJ = number * 4.184    print(kJ)``` Now it seems like we are really getting somewhere with our program! But it would be even better if instead of just printing the values, it saved them in a new list. To do this, we are going to use the `append()` function. The `append` function adds a new item to the end of an existing list. The general form of the `append` function is```pythonlist_name.append(new_thing)```Try running the following block of code. See if you can figure out why it doesn’t work.```pythonfor number in energy_kcal:    kJ = number * 4.184    energy_kJ.append(kJ)print(energy_kJ)``` This code doesn’t work because on the first iteration of our loop, the list `energy_kJ` doesn’t exist. To make it work, we have to start the list outside of the loop. The list can be blank when we start it, but we have to start it.```pythonenergy_kJ = []for number in energy_kcal:    kJ = number * 4.184    energy_kJ.append(kJ)print(energy_kJ)```  Making choices: logic statements Within your code, you may need to evaluate a variable and then do something if the variable has a particular value. This type of logic is handled by an `if` statement. In the following example, we only append the negative numbers to a new list.```pythonnegative_energy_kJ = []for number in energy_kJ:    if number < 0:        negative_energy_kJ.append(number)print(negative_energy_kJ)``` Other logic operations include- equal to `==`- not equal to `!=`- greater than `>`- less than `<`- greater than or equal to `>=`- less than or equal to `<=`You can use `not` to check if a condition is absent. The keywords `and` and `or` can be used to check more than one condition.```pythonnegative_numbers = []for number in energy_kJ:    if number < 0 or number == 0:        negative_numbers.append(number)print(negative_numbers)``` If you are comparing strings, not numbers, you use different logic operators like `is`, `in`, or `is not`. You will use the `is` operator in the next exercise.  Exercise The following list contains some floating point numbers and some numbers which have been saved as strings. Copy this list exactly into your code.```pythondata_list = ['-12.5', 14.4, 8.1, '42']```Set up a for loop to go over each element of `data_list`. If the element is a string (`str`), recast it as a `float`. Save all of the numbers to a new list called `number_list`. Pay close attention to your indentation!  File Parsing  Working with files One of the most common tasks in research is analyzing data. Many computational chemistry programs output text files that include a large amount of information including text and data that you need to analyze. Often, you need to sort through the output file and identify particular pieces of information that are most important to you. In general, this is called file parsing.  Working with file paths - the `pathlib` module For this section, we will be working with the file `ethanol.out` in the `outfiles` directory in our `data` folder.To see this, go to a new cell and type `ls`. `ls` stands for ‘list’, and will list all of the contents of the current directory. This command is not a Python command, but will work in the Jupyter notebook. To see everything in the `data` directory, type```bashls data``` Here, `03_Prod.mdout`, `sapt.out`, and `water.xyz` are all files, while outfiles is another directory.In order to parse a file, you must tell Python the location of the file, or the “file path”. For example, you can see what folder your Jupyter notebook is in by typing `pwd` into a cell in your notebook and evaluating it. `pwd` stands for ‘print working directory’, and can also be used in a terminal to see what directory you’re in. Try this in the cell below. After evaluating the cell with `pwd`, you should see an output similar to the following if you are on Mac or Linux:```'/Users/YOUR_USER_NAME/357_python_intro'```or similar to this if you were on Windows```'C:\Users\YOU_USER_NAME\357_python_intro'```Notice that the file paths are different for these two systems. The Windows system uses a backslash (`\`), while Mac and Linux use a forward slash (`/`) for filepaths.When we write a script, we want it to be usable on any operating system, thus we will use a Python module called `pathlib` that will allow us to define file paths in a general way (actually, we're using the `Path` class from the `pathlib` module).In order to get the path to the `ethanol.out` file in a general way, type```pythonfrom pathlib import Pathethanol_file = Path('data', 'outfiles', 'ethanol.out')print(ethanol_file)``` You should see something similar to the following```data/outfiles/ethanol.out```Here, we have specified that our filepath contains the ‘data’ and ‘outfiles’ directory, and the `pathlib.Path` module has made this into a filepath that is usable by our system. If you were on Windows, you would instead see that a backslash is used.  Absolute and relative paths File paths can be _absolute_ or _relative_.A relative file path gives the location relative to the directory we are in. Thus, if we are in the `Python_intro` directory, the relative filepath for the `ethanol.out` file would be `data/ethanol.out`An absolute filepath gives the complete path to a file. This could file path could be used from anywhere on a computer, and would access the same file. For example, the absolute filepath to the `ethanol.out` file on Linux might be `home/YOUR_USER_NAME/357_python_intro/data/ethanol.out`. You can get the absolute path of a file using `Path.absolute(path)`, where `path` is the relative path of the file.  Note: `os.path`We are working with the `pathlib` module here. Prior to Python 3.6, the `os.path` module was the standard, and this is how you will see people handle file paths in most Python code. `os.path` works with filepaths as strings, while in the `pathlib` module, paths are objects. `pathlib` is the standard way to handle paths in code moving forward.   Reading a file In Python, there are many ways to read in information from a text file. The best method to use depends on the type of data and the type of analysis you are performing. If you have a file with lots of different types of information, text and numbers, with different types of formatting, the most generic way to read in information is the `read()` and `readlines()` functions. Before you can read in a file, you have to `open` the file using the file path we defined above. This will create a file object, or filehandle. The file we will be analyzing in this example is a [Psi4](http://psicode.org) output file for a SCF/cc-pVDZ energy calculation for an ethanol molecule (the output information from a quantum chemical calculation using a Python program called Psi4). The `read()` function reads in all of the information in a file character-by-character as a single long string, while `readlines()` reads in a file line-by-line, storing each line as a string inside a list. The same information is read, but it's presented to the user in a slightly different manner. For now, let's use `readlines()`. ```pythonoutfile = open(ethanol_file, "r")data = outfile.readlines()``` This code opens a file for reading and assigns it to the filehandle outfile. The `"r"` argument in the function stands for `read`. Other arguments might be `"w"` for `write` if we want to write new information to the file, or `"a"` for `append` if we want to add new information at the end of the file.In the next line, we use the `readlines` function to get our file as a list of strings. Notice the "dot notation" introduced last lesson; readlines acts on the file object given right before the dot. The function creates a list called `data` where each element of the list is a string that is one line of the file. This is always how the `readlines()` function works.  Note: `readlines` function behaviorThe readlines function can only be used on a file object a single time. If you forget to set `outfile.readlines()` equal to a variable, you must open the file again in order to get the contents of the file.After you open and read information from a file object, you should always `close` the file.```pythonoutfile.close()```  Note: Two alternatives to open a fileAlternatively, you can open a file using a "context manager". In this case, the context manager will automatically handle closing of the file. To use a context manager to open and close the file, you use the word `with`, and put everything you want to be done while the file is open in an indented block.```pythonwith open(ethanol_file, "r") as outfile:    data = outfile.readlines()```This is often the preferred way to deal with files because you do not have to remember to close the file.The second option is to use the `read_text()` method on a `Path` object. This will open the `Path` object in text mode (this assumes your object _is_ a text file), read all of the text in the file, then close the file. The two blocks of code below are functionally equivalent:```pythonwith open(ethanol_file, "r") as outfile:    data1 = outfile.read()```compared to ```pythondata2 = ethanol_file.read_text()```The objects `data1` and `data2` are both strings the length of the text file. Try this yourself and verify that the strings have the same length (using the `len()` function). In fact, you can use `==` to compare the two objects and verify that they are the same (the output of `data1 == data2` should be `True`).   Check Your Understanding Check that your file was read in correctly with the `readlines()` function by determining how many lines are in the file.  Searching for a pattern in your file The file we opened is an output file which calculates the energy (and a lot of other stuff!) for an ethanol molecule. As stated previously, the `readlines()` function put the file contents into a list where each element is a line of the file. You may remember from [the earlier section on `for` loops](for_loops) that a `for` loop can be used to execute the same code repeatedly. As we learned in that section, we can use a `for` loop to iterate through elements in a list.Let’s take a look at what’s in the file.```pythonfor line in data:    print(line)```This will print exactly what is in the file. If you look through the output, you will see that the critical line says “Final Energy”. We want to search through this file and find that line, and print only that line. We can do this using an `if` statement.Returning to our file example,```pythonfor line in data:    if 'Final Energy' in line:        energy_line = line        print(energy_line)``` Remember that `readlines()` saves each line of the file as a string, so `energy_line` is a string that contains the whole line. For our analysis, if we are most interested in the energy, we need to split up the line so we can save just the number as a different variable name. To do this, we use a new function called `split`. The `split` function takes a string and divides it into its components using a delimiter.The delimiter is specified as an argument to the function (put in the parenthesis `()`). If you do not specify a delimiter, a space is used by default. Let’s try this out.```pythonenergy_line.split()``` Or, we can use the colon (‘:’) as the delimiter.```pythonenergy_line.split(':')``` When we use ‘:’ as the delimiter, a list with two elements is returned. It is split where a colon was found.We can save the output of this function to a variable as a new list. In the example below, we take the line we found in the for loop and split it up into its individual words.```pythonwords = energy_line.split()print(words)``` From this `print` statement, we now see that we have a list called `words` where we have split `energy_line`. The energy is actually the fourth element of this list, so we can now save it as a new variable.```pythonenergy = words[3]print(energy)```  Note: Python negative indexingWe also recognize that `energy` is the last element of the list. Therefore, an alternative way to assign energy is:```pythonenergy = words[-1]print(energy)```In the example above, the index value of `-1` gives the last element, and `-2` would give the second last element of a list, and so on. An excelent tutorial on Python lists accessed by index can be found at [RealPython](https://realpython.com/python-lists-tuples/list-elements-can-be-accessed-by-index). If we now try to do a math operation on energy, we receive an error message. Input the following and try to figure out what went wrong:```pythonenergy + 50``` Even though `energy` looks like a number to us, it is really a string, so we can not add an integer to it. We need to change the data type of energy to a float. This is called casting, as we saw before.```pythonenergy = float(energy)``` Now the math operation will work. If we thought ahead we could have changed the data type when we assigned the variable originally.```pythonenergy = float(words[3])```  Exercise on File Parsing Use the provided `sapt.out` file. In this output file, the program calculates the interaction energy for an ethene-ethyne complex. The output reports four interaction energy components: electrostatics, induction, exchange, and dispersion. Parse each of these energies, in kcal/mole, from the output file. (Hint: study the file in a text editor to help you decide what to search for. You can do this by navigating to the file in the left panel, right-clicking on it, and selecting "Open with… > Editor") Calculate the total interaction energy by adding the four components together. Your code’s output should look something like this:```pythonElectrostatics : -2.25850118 kcal/molExchange : 2.27730198 kcal/molInduction : -0.5216933 kcal/molDispersion : -0.9446677 kcal/molTotal Energy : 1.4475602000000003 kcal/mol```  Searching for a particular line number in your file There is a lot of other information in the output file we might be interested in. For example, we might want to pull out the initial coordinates for the molecule. If we look through the file in a text editor, we notice that the coordinates begin with a line that says```Center X Y Z Mass```and then the coordinates begin on the next line. In this case, we don’t want to pull something out of this line, as we did in our previous example, but we want to know which line of the file this is so that we can then pull the coordinates from the next few lines.When you use a `for` loop, it is easy to have Python keep up with the line numbers using the `enumerate` command. The general syntax is```pythonfor linenum, line in enumerate(list_name):    do things in the loop```In this notation, there are now two variables you can use in your loop commands. 1. `linenum` (which can be named something else) will keep up with what iteration you are on in the loop, in this case what line you are on in the file. 2. `line` (which could be named something else) functions exactly as it did before, holding the actual information from the list. Finally, instead of just giving the list name you use `enumerate(list_name)`.  Note: Enumerate with index other than 0`enumerate(list_name)` will start with a 0-index, so the first line will be labeled as ‘0’. To change this behavior, use the `start` argument in `enumerate`. For example, to start with index of “1” you can do: ```pythonfor linenum, line in enumerate(data, start=1):  do something with ‘linenum’ and  ‘line’``` The following block of code searches our file for the line that contains “Center” and reports the line number.```pythonfor linenum, line in enumerate(data):    if 'Center' in line:        print(linenum)        print(line)```   Now we know which line this is in our file (remember that the counting starts at zero!).  Exercise What would be printed if you entered the following:```pythonprint(data[77])print(data[78])print(data[79])print(data[80])print(data[81])```  A final note about regular expressions Sometimes you will need to match something more complex than just a particular word or phrase in your output file. Sometimes you will need to match a particular word, but _only_ if it is found at the beginning of a line. Or perhaps you will need to match a particular pattern of data, like a capital letter followed by a number, but you won’t know the exact letter and number you are looking for. These types of matching situations are handled with something called regular expressions which is accessed through the Python module `re`. While using regular expressions (regex) is outside the scope of this tutorial, they are very useful and you might want to learn more about them in the future. An excellent tutorial can be found in the book ["Automate the Boring Stuff with Python" by <NAME>](https://automatetheboringstuff.com). It is free to read on the linked site. A great test site for regular expressions is [RegEx101.com](https://regex<EMAIL>/).   Processing multiple files In our previous lesson, we parsed values from an output files. While you might have seen the utility of doing such a thing, you might have also wondered why we didn’t just search the file and cut and paste the values we wanted into a spreadsheet. If you only have one or two files, this might be a very reasonable thing to do. But what if there are 50 files to analyze? How about 1000? In such a case the cutting and pasting method would be very tedious and time consuming.One of the real powers of writing a program to analyze your data is that you can just as easily analyze one file or one _hundred_ files. In this example, we are going to parse the output files for a whole series of aliphatic alcohol compounds and parse the energy value for each one. The output files are all saved in a folder called `outfiles` that should be in the `data` folder. Make sure the folder is in the same directory as the directory where you are writing and executing your code.To analyze multiple files, we will need to import a Python _library_. While we've seen this done, I'll give you a proper explanation here. A library is a set of _modules_ which contain _functions_. The functions within a library or module are usually related to one another. Using libraries in Python reduces the amount of code you have to write. In the last lesson, we imported `pathlib.Path`, which was a module that handled filepaths for us.In this lesson, we will be using the `glob` module in `pathlib`, which will help us read in multiple files from our computer. Within a library there are modules and functions which do a specific computational task. Usually a function has some type of input and gives a particular output. To use a function that is in a library, you often use the dot notation introduced in the previous lesson. In general```pythonimport library_nameoutput = library_name.funtion_name(input)```We will import the `Path` module from `pathlib` and work with the functions contained within it.   Exercise How would you use the `pathlib.Path` module to point to the directory where your outfiles are? In order to get all of the files which match a specific pattern, we will use the wildcard character `*`.```pythonfile_path = Path('data/outfiles')file_filter = '*.out'``` This specifies that we want to look for files in a directory called `data/outfiles`, and we'd like to filter the list to only include files that end in “.out”. The `*` is the wildcard character which matches any character.Next we are going to use a function called `glob`. There is also a library called `glob`, but the capabilities are similar enough that we won't worry about importing it.  The output of the function `glob` is a _generator_, an object that creates output objects on demand. As we want a _list_ of all the filenames that fit the pattern specified in the input, we'll need to cast the `glob` output to a list (`list(filenames)`). The input of the `glob` function is the search pattern.```pythonfilenames = file_path.glob(file_filter)print(list(filenames))``` This will give us a list of all the files which end in `*.out` in the outfiles directory. Now if we want to parse every file we just read in, we can use a `for` loop to go through each file.```pythonfor f in filenames:    with open(f, 'r') as outfile:        data = outfile.readlines()    for line in data:        if 'Final Energy' in line:            energy_line = line            words = energy_line.split()            energy = float(words[3])            print(energy)``` Notice that in this code we actually used two `for` loops, one nested inside the other. The outer `for` loop counts over the filenames we read in earlier. The inner `for` loop counts over the line in each file, just as we did in our previous file parsing lesson.The output our code currently generates is not that useful. It doesn’t show us which file each energy value came from.We want to print the name of the molecule with the energy. We can use `Path.name`, which is another function in `pathlib.Path` to get just the name of the file.```pythonfirst_file = filenames[0]print(first_file)file_name = first_file.nameprint(file_name)```  ExerciseHow would you extract _just_ the molecule name (_i.e.,_ "heptanol") from the example above? Using the solution above, we can modify our loop so that it prints the file name along with each energy value.```pythonfor f in filenames:     Get the molecule name    file_name = f.name    split_filename = file_name.split('.')    molecule_name = split_filename[0]     Read the data    with open(f, 'r') as outfile:        data = outfile.readlines()     Loop through the data    for line in data:        if 'Final Energy' in line:            energy_line = line            words = energy_line.split()            energy = float(words[3])            print(molecule_name, energy)``` Now that we've done all that work, you should know that `pathlib.Path` has another method called `stem` that can give you just the first part of the filename (without the extension). Documentation on `pathlib` is available as part of the [Python documentation library](https://docs.python.org/3/library/pathlib.html). If you read through the documentation, you can find all sorts of useful information about the capabilities of a given library or function. When in doubt, read the docs!  Printing to a File Finally, it might be useful to print our results in a _new_ file, perhaps so we can share our results with colleagues or or e-mail them to our advisor. Much like when we read in a file, the first step to writing output to a file is opening that file for writing. In general to open a file for writing you use the syntax```pythonfilehandle = open('file_name.txt', 'w+')```The `w` means open the file for writing. If you use `w+` that means open the file for writing and if the file does not exist, create it. You can also use `a` for append to an existing file or `a+`. The difference between `w+` and `a+` is that `w+` will _overwrite_ the file if it already exists, whereas `a+` will keep what is already there and just add additional text to the file. When in doubt, `a+` is safer, as it won't erase your previous output. Think carefully about which one is appropriate in a given situation.Python can only write strings to files. Our current `print` statement is not a string; it prints two Python variables. To convert what we have now to a string, you place a letter `f` in front of the line you want to print and enclose it in quotes (single or double makes no difference). The `f` means that the string is a [_formatted string literal_](https://docs.python.org/3/reference/lexical_analysis.htmlf-strings). These objects give us a way to use code and variables within strings to give formatted output. Each Python variable is placed in braces. Then you can either print the line (as we have done before) or you can use the `filehandle.write()` command to print it to a file.```pythondatafile = open('energies.txt','w+')  This opens the file for writingfor f in filenames:     Get the molecule name    molecule_name = f.stem     Read the data    with open(f, 'r') as outfile:        data = outfile.readlines()     Loop through the data    for line in data:        if 'Final Energy' in line:            energy_line = line            words = energy_line.split()            energy = float(words[3])            datafile.write(f'{molecule_name} \t {energy} \n')datafile.close()``` After you run this command, look in the directory where you ran your code and find the “energies.txt” file. Open it in a text editor and look at the file.In the file writing line, notice the `\t` in the middle of the line and `\n` at the end of the line. These are known as ["escape sequences"](https://www.pyblog.in/programming/python-escape-sequence-list-and-uses/List_of_Escapes_Sequences), and are two of many. The most common are `\t` for tab, `\n` for a new line, `\\` for a literal backslash, and `\'` or `\"` for single or double quotes (necessary when you want quotes inside of a quoted string). To make the printing neater, we will separate the file name from the energy using a tab. The newline character prevents the text in our file from all being smushed together on one line. Finally, the `filehandle.close()` command at the end is very important. Think about a computer as someone who has a very good memory, but is very slow at writing. Therefore, when you tell the computer to write a line, it remembers what you want it to write, but it doesn’t actually write the new file until you tell it you are finished. The `datafile.close()` command tells the computer you are finished giving it lines to write and that it should go ahead and write the file now. If you are trying to write a file and the file keeps coming up empty, it is probably because you forgot to close the file.  Note: One more thing about string formattingThe `f’string’` notation that you can use with the `print` or the `write` command lets you format strings in many ways. You could include other words or whole sentences. For example, we could change the file writing line to```pythondatafile.write(f'For the file {molecule_name} the energy is {energy} in kcal/mole.')```where anything in the braces is a Python variable and it will print the value of that variable.  File parsing projectIn the `data` folder, there is a file called `03_Prod.mdout`. This is a file output by the Amber molecular dynamics simulation program.If you open the file and look at it, you will see sections which look like this:``` NSTEP =      100   TIME(PS) =      20.200  TEMP(K) =   296.43  PRESS =  -300.8 Etot   =     -4585.1049  EKtot   =      1129.2368  EPtot      =     -5714.3417 BOND   =         1.5160  ANGLE   =         6.9846  DIHED      =        11.7108 1-4 NB =         4.3175  1-4 EEL =        49.9911  VDWAALS    =       882.6741 EELEC  =     -6671.5358  EHBOND  =         0.0000  RESTRAINT  =         0.0000 EKCMT  =       583.7810  VIRIAL  =       786.8823  VOLUME     =     31270.0410                                                    Density    =         0.6104 Ewald error estimate:   0.3214E-03 ------------------------------------------------------------------------------```Your assignment is to parse this file, and write a new file containing a list of the total energies. Name your file `Etot.txt`. When you open it, it should look like this:```-4585.1049-4573.5326-4548.1223-4525.341-4542.8995-4550.9376-4543.8652-4570.4109-4550.4225-4585.2078...````...` indicates that you will have many more rows. We’ve only shown the first 10 here.If you are unsure of where to start with this assignment, check the hint section! Hints:     Click here for some ideas on how to proceed.    It helps when you are writing code to break up what you have to do into steps. Overall, we want to get information from the file. How do we do that?If you think about the steps you will need to do this assignment you might come up with a list that is like this:     Open the file for reading     Read the data in the file     Loop through the lines in the file.     Get the information from the line we are interested in.     Write the information to a file.It can be helpful when you code to write out these steps and work on it in pieces. Try to write the code using these steps. Note that as you write the code, you may come up with other steps! First, thing about what you have to do for step 1, and write the code for that. Next, think about how you would do step 2 and write the code for that. You can troubleshoot each step using print statements. The steps build on each other, so you can work on getting each piece written before moving on to the next step.<jupyter_code># Place your project solution here<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06439145336541967, 'fraction_numerical': 0.011376066506234959, 'mean_word_length': 4.193788663047595, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 18, 'https://': 9, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '369334', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 10728, 'n_tokens_neox': 10125, 'n_words': 6463, 'path': 'py_intro.ipynb', 'repo_name': 'mskblackbelt/357_python_intro', 'size': '41143'}