<jupyter_start><jupyter_text>Lab.01 / IBM3202 – Warm-up on COLAB & Brief Review of Biomolecular Databases  Part I. What is Google Colab? 🤔 Colaboratory, or "Colab" for short, allows you to write and execute Python in your browser, with - Zero configuration required- Free access to GPUs- Easy sharingYou can watch the [Introduction to Colab](https://www.youtube.com/watch?v=inN8seMm7UI) video recommended by Google Colab to learn more, or just get started below! Note that at your left you have three icons. The first one looks like this:>and corresponds to the Table of Contents of this tutorial.The last folder icon corresponds to the File Explorer of the virtual machine hosted in Google Cloud assigned to your session.   The central concept of Google Colab Notebook: CellsA notebook is a list of cells. Cells contain either explanatory text or executable code and its output. Click a cell to select it. **Adding and moving cells**You can add new cells by using the **+ CODE** and **+ TEXT** buttons that show when you hover between cells. These buttons are also in the toolbar above the notebook, and they can be used to add a cell below the currently selected cell.You can move a cell by selecting it and clicking **Cell Up** or **Cell Down** in the top toolbar. Consecutive cells can be selected by "lasso selection", i.e. by dragging from outside one cell and through the group.  Non-adjacent cells can be selected concurrently by clicking one and then holding down **Ctrl** while clicking another.  Similarly, using **Shift** instead of Ctrl will select all cells between two non-adjacent selections.  Text cells Colaboratory has two types of cells: text and code. The text cells are formatted using a simple markup language called **markdown**, based on [the original](https://daringfireball.net/projects/markdown/syntax) markdown project. This is a **text cell**. You can **double-click** to edit this cell. Text cellsuse markdown syntax. To learn more, see the [markdownguide](/notebooks/markdown_guide.ipynb) recommended by Google Colab.  Markdown ⌨️ To see the markdown source, double-click a text cell, showing both the markdown source (left) and the rendered version (right). Above the markdown source there is a toolbar to assist editing. You can also use tags to format your text. The following are examples of markdown text formats. Each word/phrase is shown in the desired format, and the tags around it are those required to achieve each specific format.**Text Formats:**\**italics*\* or \__italics__**\*\*bold\*\***\~\~~~strikethrough~~\~\~\``monospace`\`**Indentations:**No indent>\>One level of indentation>>\>\>Two levels of indentation**An ordered list:**1. 1\. One1. 1\. Two1. 1\. Three**An unordered list:*** \* One* \* Two* \* Three ******If you are interested in learning more about markdown in Google Colab you can read this nice article which includes a cheat sheet [here](https://towardsdatascience.com/cheat-sheet-for-google-colab-63853778c093)**  Math 🧮 & Equations ✏️ You can also add math to text cells using [$\LaTeX$](http://www.latex-project.org/)to be rendered by [MathJax](https://www.mathjax.org). Just place the statementwithin a pair of **`$`** signs. For example `$\sqrt{3x-1}+(1+x)^2$` becomes$\sqrt{3x-1}+(1+x)^2.$ Also, if you double the **`$`** tags in your $\LaTeX$ equations, you can set the contents off on its own centered line. For example, `$$y = 0.1 x$$` renders the following equation: $$y = 0.1 x$$  Tables 📍 Tables:```First column name | Second column name--- | ---Row 1, Col 1 | Row 1, Col 2Row 2, Col 1 | Row 2, Col 2```becomes:>First column name | Second column name>--- | --->Row 1, Col 1 | Row 1, Col 2>Row 2, Col 1 | Row 2, Col 2Horizontal rule done with three dashes (\-\-\-):---  Gifs 😱 YES! you can add animated gifs  Code cellsBelow is a **code cell**. To execute the contents of a code cell, you first must connect to a hosted runtime by clicking on the **Connect** button located in the toolbar menu.  Once the toolbar button changes to **'Connected'**, click in the code cell below to select it and execute the contents in the following ways: * Click the **Play icon** in the left gutter of the cell;* Type **Cmd/Ctrl+Enter** to run the cell in place;* Type **Shift+Enter** to run the cell and move focus to the next cell (adding one if none exists); or* Type **Alt+Enter** to run the cell and insert a new code cell immediately below it.There are additional options for running some or all cells in the **Runtime** menu.<jupyter_code>W = 'Tryptophan'
C = 'Cysteine'
W,C<jupyter_output><empty_output><jupyter_text>Part II. BASH warm up 🏃🏃‍♀️🏃‍♂️  Part II A. Shell introduction  • IPython and Shell Commands *This part of this tutorial contains an excerpt from the [Python Data Science Handbook](http://shop.oreilly.com/product/0636920034919.do) by <NAME>; the content is available [on GitHub](https://github.com/jakevdp/PythonDataScienceHandbook).**The text is released under the [CC-BY-NC-ND license](https://creativecommons.org/licenses/by-nc-nd/3.0/us/legalcode), and code is released under the [MIT license](https://opensource.org/licenses/MIT). If you find this content useful, please consider supporting the work by [buying the book](http://shop.oreilly.com/product/0636920034919.do)!* When working interactively with the standard Python interpreter, one of the frustrations is the need to switch between multiple windows to access Python tools and system command-line tools. IPython bridges this gap, and gives you a syntax for executing shell commands directly from within the IPython terminal.The magic happens with the exclamation point: anything appearing after ``!`` on a line will be executed not by the Python kernel, but by the system command-line.The following assumes you're on a Unix-like system, like this Google Colab cloud instance.Some of the examples that follow will fail on Windows, which uses a different type of shell by default.If you're unfamiliar with shell commands, I'd suggest reviewing the [Shell Tutorial](http://swcarpentry.github.io/shell-novice/) put together by the always excellent Software Carpentry Foundation.  • Quick Introduction to the Shell A full intro to using the shell / terminal / command-line is well beyond the scope of this lab, but for the uninitiated we will offer a quick introduction here.>The shell is a way to interact textually with your computer.Ever since the mid 1980s, when Microsoft and Apple introduced the first versions of their now ubiquitous graphical operating systems, most computer users have interacted with their operating system through familiar clicking of menus and drag-and-drop movements.But operating systems existed long before these graphical user interfaces, and were primarily controlled through sequences of text input: at the prompt, the user would type a command, and the computer would do what the user told it to.Those early prompt systems are the precursors of the shells and terminals that most active data scientists still use today.Someone unfamiliar with the shell might ask why you would bother with this, when many results can be accomplished by simply clicking on icons and menus.A shell user might reply with another question: why hunt icons and click menus when you can accomplish things much more easily by typing?While it might sound like a typical tech preference impasse, when moving beyond basic tasks it quickly becomes clear that the shell offers much more control of advanced tasks, though admittedly the learning curve can intimidate the average computer user.As an example, the following are the most common bash commands and a short description:```bashcd: change directoryls: listmv: movecp: copymkdir: make new directoryhistory: terminal history (i.e. history of the commands you have executed) help: command list & helpecho: outputs to terminalexpr: evaluate expression and outputs to terminalwc: word countcat: “concatenate” streams all input to the terminalsed: “stream editor” edits input (most importantly substitute) and outputs to the terminal vim: “visual” visualizes input and allows for editiongrep: “Global regular expression print”, searches a given expression and outputs to terminal awk: pattern scanning & processing language```Notice that all of this is just a compact way to do familiar operations (navigating a directory structure, creating a directory, moving a file, etc.) by typing commands rather than clicking icons and menus.Note that with just a few commands (``pwd``, ``ls``, ``cd``, ``mkdir``, and ``cp``) you can do many of the most common file operations.It's when you go beyond these basics that the shell approach becomes really powerful.  • Shell Commands in iPython Any command that works at the bash command-line can be used in IPython by prefixing it with the ``!`` character.For example, the ``ls``, ``pwd``, and ``echo`` commands can be run as follows:```ipythonIn [1]: !lssample_dataIn [2]: !pwd/contentIn [3]: !echo "printing from the shell"printing from the shell```Try these commands in the next code cells!<jupyter_code>#List all files and directories with ls

#Print the working directory of Google Colab cloud linux instance with pwd

#Print a Hello world using echo<jupyter_output><empty_output><jupyter_text>• Passing Values to and from the Shell<jupyter_code>directory = !pwd
print(directory)<jupyter_output><empty_output><jupyter_text>Shell commands can not only be called from IPython, but can also be made to interact with the IPython namespace.For example, you can save the output of any shell command to a Python list using the assignment operator:```ipythonIn [4]: contents = !lsIn [5]: print(contents)['sample_data']In [6]: directory = !pwdIn [7]: print(directory)['/content']```Try them out yourself!<jupyter_code>#Try the commands indicated above<jupyter_output><empty_output><jupyter_text>Note that these results are not returned as lists, but as a special shell return type defined in IPython:```ipythonIn [8]: type(directory)IPython.utils.text.SList```This looks and acts a lot like a Python list, but has additional functionality, such asthe ``grep`` and ``fields`` methods and the ``s``, ``n``, and ``p`` properties that allow you to search, filter, and display the results in convenient ways.For more information on these, you can use IPython's built-in help features. Communication in the other direction–passing Python variables into the shell–is possible using the ``{varname}`` syntax:```ipythonIn [9]: message = "hello from Python"In [10]: !echo {message}hello from Python```The curly brackets contain the variable name, which is replaced by the variable's contents in the shell command.<jupyter_code>#Try assigning a string to variable and the print it using echo
holi = 'hello mundo'
!echo {holi}<jupyter_output><empty_output><jupyter_text>• Shell-Related Magic Commands If you play with IPython's shell commands for a while, you might notice that you cannot use ``!cd`` to navigate the filesystem:```ipythonIn [11]: !pwd/content/In [12]: !cd ..In [13]: !pwd/content/```<jupyter_code>#Try for yourself!
!pwd
!cd ..
!pwd<jupyter_output><empty_output><jupyter_text>The reason is that shell commands in the notebook are executed in a temporary subshell.If you'd like to change the working directory in a more enduring way, you can use the ``%cd`` magic command:```ipythonIn [14]: %cd ../```This is known as an ``automagic`` function, and this behavior can be toggled with the ``%automagic`` magic function.Besides ``%cd``, other available shell-like magic functions are ``%cat``, ``%cp``, ``%env``, ``%ls``, ``%man``, ``%mkdir``, ``%more``, ``%mv``, ``%pwd``, ``%rm``, and ``%rmdir``, any of which can be used without the ``%`` sign if ``automagic`` is on.This makes it so that you can almost treat the IPython prompt as if it's a normal shell:This access to the shell from within the same terminal window as your Python session means that there is a lot less switching back and forth between interpreter and shell as you write your Python code.<jupyter_code>#Try it here using the !cd to access and exit the 'sample_data' folder 
!pwd
!cd sample_data
!pwd
!cd ..
#Try using the %cd 
!pwd
%cd sample_data
!pwd
%cd ..<jupyter_output><empty_output><jupyter_text>💡 Hint: You can use the %%bash at the beginning of your cell code instead of the `!` approach to active automagic for the whole cell code. Try it below!<jupyter_code>%%bash
pwd
cd sample_data
pwd
cd ..<jupyter_output><empty_output><jupyter_text>• Lets practice! The terminal enters the **/content** directory by default in Google Colab, which contains all the main folders that you will use.The `ls` command will list all the folders and files within the current folder.Sadly, there is no color code in Google Colab to differentiate between files and folders in your code cells – which is very common on Linux. Therefore, you will have to manually inspect your directories on the File Explorer built in Google Colab and available on the menu to your left.<jupyter_code>#List the files using ls
#!pwd
#%ls<jupyter_output><empty_output><jupyter_text>Now, we can start making new directories by typing `mkdir NAME`, where NAME is the name for our new folder.To change directory to a particular folder, type `%cd` followed by the folder name<jupyter_code>#Make two directories named folder and folder/folder2
#%mkdir prueba
#%ls
#%cd prueba
#%mkdir prueba2
#%ls
#!pwd
#%cd ..<jupyter_output><empty_output><jupyter_text>Files and folders can be renamed by using the  `mv` command as follows: `mv file1 file2`.This will move the content from the first file into a second file. In the cell below rename folder to newfolder using `mv`<jupyter_code>#Rename folder to newfolder and list
#!pwd
#%ls
#%mv prueba prueba3
#%ls<jupyter_output><empty_output><jupyter_text>In order to create and edit text files in Google Colab you can use the included text editor. Use `touch` to create and empty file<jupyter_code>#Use touch to generate a empty file
#%cd prueba3/
#!touch testfile
#ls<jupyter_output><empty_output><jupyter_text>Refresh the files in the file explorer at your left and double click the generated text file. Add whaterever you want inside and remember to click Ctrl+S or Cmd+S to save the changes. Use `cat` to show what is inside your just edited text file<jupyter_code>#Use cat here
#%cat testfile<jupyter_output><empty_output><jupyter_text>This information and exercises should be sufficient for you to start using the bash shell commands more often. As with any other piece of software, it requires practice, but we are sure you will get used to it by the end of this course.Now here are some tasks for you to complete for more practice!1. Make a copy of your file using `cp` just like `mv`2. Move your copy to the HOME directory3. Output the contents or your file to the terminal using `cat`4. Use the `paste` command instead of the `cat` and describe what happens.<jupyter_code>#Your turn!<jupyter_output><empty_output><jupyter_text>Part II B. Database retrieval  • Protein Data Bank Now that you can use the command-line in Google Colab to do some basic commands, let's go see a PDB file.Go to the [Protein Data Bank ](https://www.rcsb.org) and search the structure whose accession ID is 6ANE. This is a crystal structure of a PET hydrolase that was solved by our lab in 2018 [[ <NAME> & <NAME> et al. (2018) *Biophys J 114*, 1302-1312 ]](https://www.cell.com/biophysj/fulltext/S0006-3495(18)30208-X).For this, you just need to write 6ANE on the search area, hit 'Enter' and you will get the PDB entry corresponding to the accession code alongside related information of this protein structure.In order to directly retrieve this structure in Google Colab, we are going to use the **urllib.request** module from python, go and run the cell below.💡 Hint: **remember** to go back to the default directory by executing `%cd /content` before retrieving the PDB file<jupyter_code>import urllib.request 
urllib.request.urlretrieve('http://files.rcsb.org/download/6ane.pdb', '6ane.txt')<jupyter_output><empty_output><jupyter_text>Note that we are changing the extension from **.pdb** to **.txt** in order to quickly visualize the information as text file using the text editor built in Google Colab. To open this text file, go to the File Explorer, click on refresh to show the 6ane.txt file (if not readily available) and double click on it.💡 Hint: try resizing and or changing the layout from vertical split to horizontal split by clicking the three rectangle icon.   Use the Google Colab text editor to examine the contents of the PDB file. This is a human-readable file, which contains information of how the experiment was conducted and what the results are. Well, it is human-readable if we actually tell you how to read it...The head of the file contains information of all the experimental conditions that allowed solving the structure contained in the PDB file, as well as information about the biological features of the protein and the overall features of the crystal structure (cell unit, dimensions, number of monomers, biological assembly, etc). All of this information is followed by several lines led by **'ATOM'** or **'HETATM'**. These lines of information contain the positions, defined by the X-ray diffracting electron densities, of the atoms of a crystal structure of a protein. It also includes water molecules and other non-protein ligands that could also contribute to the electron densities detected during X-ray diffraction. This information is contained in the PDB file as a text of atom positions, numbers, types and the residues to which they belong to, as indicated in the following figure:![alt text](https://raw.githubusercontent.com/pb3lab/ibm3202/master/images/pdbformat_01.png) Navigating through files this long can be a bit challenging. If you wish to look for a particular word, such as PET, there is a program called `grep` that can help you do about the same without entering into a visual text editor (i.e. without leaving the terminal).By using `grep pattern1 file`, this program will output to the terminal all lines containing ‘pattern1’ within a ‘file’.For example:```!grep PET 6ane.txt```<jupyter_code>#Try it yourself here<jupyter_output><empty_output><jupyter_text>Now is your turn!Let’s use `grep` to quickly check what is the RESOLUTION of that crystal structure.💡Hint: The uppercase in RESOLUTION is not random...1. What METHOD was used for determining the structure? 2. Which ORGANISM is this protein from?3. In which CONDITIONS was the enzyme crystallized?4. What PLASMID was used for protein expression?Now that we are more comfortable using BASH, we’ll get to the real advantages of using the terminal.<jupyter_code>#Use this cell to use grep over the 6ane file<jupyter_output><empty_output><jupyter_text>The function `wc` helps you count characters (-m), lines (-l) and words (-w) in files.1. On average, how long is each word in the file? 2. How many words does the average line have? 3. And how many characters?<jupyter_code>#Use wc in this cell<jupyter_output><empty_output><jupyter_text>The good stuff of BASH starts with 'pipe'.This means feed the output from one program as the input to the next one.For instance, use `grep` to grab the lines containing ATOM.That’s plenty! We can pipe this output to `wc`, so it can tell us how many lines there are. For this, you have to write the first command, then put a pipe '|' and then write the second command:`!grep ATOM 6ane.pdb | wc -l`<jupyter_code>#Try it yourself<jupyter_output><empty_output><jupyter_text>Pipes work by redirecting the output from the left of the pipe as the input to the right of the pipe. That is why you should not give `wc -l` any input argument.Pipes are also independent, so you can use any number of pipes to process the text until you have what you need. In some programs, such as `grep`, there are other options that you might find useful to extract specific information from a given file. We will perform some activities to review them now:1. How many of the atoms in the PDB are alpha-carbon?💡Hint: You want to extract the atoms from the PDB file AND specifically those that are alpha-carbon.Try first by doing a pipe with `grep` (you only need to execute the command twice). Now, compare that with the AND (.*) conditional of `grep` as follows:  `grep 'pattern1'.*'pattern2' file`Did you get the same results? Use `wc` to check if you obtained the same number of lines.<jupyter_code>#Try it yourself<jupyter_output><empty_output><jupyter_text>2. How many arginine residues are in the PDB structure?💡 Hint: You want to extract the ARG residues AND you just need one atom (say, alpha-carbon) of the whole residue to count the number of ARG.This exercise is pretty similar to the previous one, so we really think you will not need much help...<jupyter_code>#Try it yourself<jupyter_output><empty_output><jupyter_text>3. Are there any HETATM in the file? How many are NOT water?💡 Hint: You want to obtain the HETATM from the file AND NOT those that correspond to water (HOH). Pipe is not sufficient for this part; you actually need a specific `grep` conditional to disregard the water molecules`grep -v 'pattern1' file1`The -v option allows `grep` to eliminate the lines containing a given pattern. Try it out with your PDB and let us know how many non-protein/non-water atoms you have in your crystal structure.<jupyter_code>#Try it yourself<jupyter_output><empty_output><jupyter_text>4. The TRP residue 158 of this protein is known to have two different conformations. How many conformations are present in the crystallographic structure?💡 Hint: Alternative conformations of a residue in a PDB are indicated by a leading letter (eg. ATRP, BTRP, CTRP, etc). You would be looking for 'ATRP' OR 'BTRP' in this exercise.You can do this exercise by using `grep` and pipe once for ATRP and a second time for BTRP (try it out). However, you would end up with two separate outputs. If you want to do a so-called one-liner, we can do the same by using the appropriate OR (\|) conditional in `grep``grep 'pattern1\|pattern2' file1`Please check how many chains of the protein contain a TRP residue in two conformations and if all atoms are occupying both conformations.<jupyter_code>#Try it yourself<jupyter_output><empty_output><jupyter_text>5. If at neutral pH the sidechains of ARG, HIS and LYS hold a charge 1+, and ASP and GLU a 1-, what is the total charge of each chain?Now it is time for you to ponder how to use all of the previous commands, conditionals and pipe to achieve this task. 💡 Hint: You can count all ARG, HIS and LYS residues first (they are all +1) and all ASP and GLU residues second (they are all -1).<jupyter_code>#Try it yourself<jupyter_output><empty_output><jupyter_text>• Other Databases Biological databases contain information derived from experiment and/or analysis of scientific experiments. They are categorized as primary if they contain experimentally-derived data (i.e. nucleotide or protein sequences, macromolecular structures) provided by the researchers who acquired it. The database archives this data which is then never changed (as they form part of a scientific record), and an accession number is assigned to it. Since this record is archival in nature, the curators only care for the data to be properly stored (to be consistent and accurate) and not for their quality or underlying biological or technical errors; they are not reviewers. This also means that the data cannot be changed, as it is a public reference, and only corrections are attached to it if required.On the other hand, secondary databases utilize the information from primary databases to derive new knowledge by using complex algorithms and manual analysis. This means putting together primary information from databases and literature, grouping and comparing entries, and often including relevant predictions from external sites. Therefore, this type of databases has become increasingly popular as reference library, as they provide extensive integrated information on most genes or gene products that have been investigated by the scientific community.Some examples of primary databases and the information they hold:>**International Nucleotide Sequence Database Collaboration:** A collaborative effort from the American GenBank, the European ENA and the Japanese DDBJ, for archiving data raw reads, alignments and assemblies of functional annotation, alongside information regarding their experimental setup. American site [www.ncbi.nlm.nih.gov/genbank](https://www.ncbi.nlm.nih.gov/genbank)>**World-wide Protein Data Bank (wwPDB)**: A joined effort from the American, European & Japanese PDB alongside the Magnetic Resonance DB to hold a repository of information about 3D structure of biological molecules. American site [www.rcsb.org](https://www.rcsb.org)>**Proteome Xchange**: A consortium established to provide public proteomics repositories. Composed of American, English, Chinese and Japanese member databases. Site [www.proteomexchange.org](https://www.proteomexchange.org)Some examples of secondary databases and their information:>**Uniprot:** Resources of protein sequence and functional information. It includes ~600,000 manually annotated and reviewed entries from Swiss-prot, and ~120.000.000 automatically annotated yet unreviewed records from TrEMBL. Site [www.uniprot.org](https://www.uniprot.org)>**CATH:** Provides structural classification for proteins deposited in the PDB. Site www.cathdb.info Extended list of available databases: [www.oxfordjournals.org/nar/database/c](https://www.oxfordjournals.org/nar/database/c) We will retrieve some sequences and protein structures. At this time, we will not visualize them, only check the file types. We will start with sequences from the GenBank. We will retrieve the sequences using a Basic Local Alignment Search Tool or [BLAST](https://blast.ncbi.nlm.nih.gov/Blast.cgi) to find regions of local similarity between sequences our query and the database. Now open the BLAST website and go to Nucleotide BLAST. We will perform 5 searches to illustrate what is in the database. First, fill the query with this sequence ```>sequence1aaatgaagat gaatttagaa acatggtgac tagatgcaac aatgttgggg ttcgtattta tgtggatgct gtaattaatc atatgtgtgg taatgctgtg agtgcaggaa caagcagtac``` And then blast and wait for the results.To us, this looks like randomly jumbled letters, yet you will notice this is quite accurate information. At the top, there is a summary of the blast search, and then further down all the entries ordered from highest to lowest score. This score is based on the alignment between your query sequence and its matches.To the right you will see the **coverage** (what % of your query is covered by the match), **e-value** (the number of matches with that score that you would find just by random chance), and finally the **%identity** between query and match. For identification, you want your coverage and identity to be as high as possible, and your e-value as near to 0.For your query sequence, you will actually find **exact matches** (coverage 100%, e-value 0, %identity 100%) because this sequence belongs to the database. Now, in a new tab, just using the first 20 base pairs repeat the alignment. ```>sequence2aaatgaagat gaatttagaa acatggtgac tagatgcaac``` **Are the hits the same? What happened to the e-value? How can you explain this?**Now we will test the blast search. Use your imagination and create a 15 bp-long sequence (just using a, c, t & g!), and blast it.**What gene have you re-created? How big is the e-value?**In this example you are recreating what is a random hit in the database. The smaller the sequence, the most likely at least some bit of it already exists in the database. So, to increase your chances of a true hit, you need to have a long-enough sequence to provide reliable comparison.To illustrate further this effect, randomize a new 30 bases long sequence a blast it. See? Now, we will move onto protein sequences. We will use the same sequence as before but translated into an amino acid sequence. The translated sequence was obtained by clicking on the first hit after blast search. The deposited sequence contains all the information known about what species it belongs to, and a plethora of experimental and annotation information, one of which is Translation, hence our new sequence.Now, again just grab the first 20 residues of the sequence and blast them on Protein BLAST.```>sequence3MKLFWLLFTIGFCWAQYSPNT``` What happens with the score and e-value?Finally, we will edit our search parameters and look for our sequence in the database **UNIPROT**. Click on the top hit report and in the accession code under **DBSOURCE**, this will take you to the **UNIPROT** page for the protein.**What information is deposited there in this secondary database?**This page has a detailed PDB integration. Get the PDB identifier for the highest resolution structure and go directly to the PDB to retrieve it.If miraculously you have managed to finish all the tasks in time, then the next section is for you.  Optional Challenge: Using awk. A script is a series of commands that can be executed without requiring human input. `awk` is a scripting language that can help you retrieve, edit and analyze text data.As with bash, `awk` scripting can be used as in-line commands if they are compact enough, otherwise, we would have to write a file which includes all the commands we wish to run. In this case, we will not go into scripting, just basic `awk` stream editing. We highly recommend you learn how to use `awk`.`awk` can quickly evaluate conditions. Let’s say we wish to print all the lines whose fourth column says HIS, we just say```awk '$4=="HIS"' 6ane.txt```Notice that we initialize and end the use of awk with '. Then, `$0` refers to the whole line, `$1` refers to the first column, `$2` the second and so on. The `==` is asking for an exact match to a given pattern. In this case, we want this to be a string (a sequence of characters), and strings are defined by “string”.What the previous command line says is *print if 4th column equals HIS in 6ane.txt*. We can pass this in and out of a pipe, so plenty more conditions can be used (!= means not equal).1. How many residues are in the chain C?2. How many oxygens are there in total in the crystal? How many from water? 3. Figure out how you can calculate the center of mass for chain B.<jupyter_code>#Write down your code here!<jupyter_output><empty_output><jupyter_text>**This is the end of the first tutorial. Good science!**  Appendix A (Homework) 🤪 Quick Python 🐍 warm up   Before you start The following are excerpts from many different resources that we included in this tutorial so that you can familiarize with the use of **Python** for many different analysis. Some of the tools described here will be used in the following tutorials.  **What is Python?** Executive SummaryPython is an interpreted, object-oriented, high-level programming language with dynamic semantics. Its high-level built in data structures, combined with dynamic typing and dynamic binding, make it very attractive for Rapid Application Development, as well as for use as a scripting or glue language to connect existing components together. Python's simple, easy to learn syntax emphasizes readability and therefore reduces the cost of program maintenance. Python supports modules and packages, which encourages program modularity and code reuse. The Python interpreter and the extensive standard library are available in source or binary form without charge for all major platforms, and can be freely distributed.<jupyter_code># In order to "print" in python we use the print() function
print("Hello world!")
#Install packages with PIP package manager
!pip install mdanalysis<jupyter_output><empty_output><jupyter_text>Variables and data types<jupyter_code>a=1
print(a)<jupyter_output><empty_output><jupyter_text>Note that we did not need to introduce the variable `a` in any way. No type was given for the variable. Python automatically detected that the type of `a` must be `int` (an integer). We can query the type of a variable with the builtin function `type`:<jupyter_code>type(a)<jupyter_output><empty_output><jupyter_text>Note also that the type of a variable is not fixed:<jupyter_code>a="some text"
type(a)<jupyter_output><empty_output><jupyter_text>In Python the type of a variable is not attached to the name of the variable, like in C for instance, but instead with the actual value. This is called dynamic typing. ![typing.svg](https://github.com/csmastersUH/data_analysis_with_python_2020/blob/master/typing.svg?raw=1)  ExpressionsAn *expression* is a piece of Python code that results in a value. It consists of values combined together with *operators*. Values can be literals, such as `1`, `1.2`, `"text"`, or variables. Operators include arithmetics operators, comparison operators, function call, indexing, attribute references, among others. Below there are a few examples of expressions: ```1+27/(2+0.1)acos(0)mylist[1]c > 0 and c !=1(1,2,3)a<5obj.attr(-1)**2 == 1``` Note that in Python the operator `//` performs integer division and operator `/` performs float division. The `**` operator denotes exponentiation. These operators might therefore behave differently than in many other common languages. As another example the following expression computes the kinetic energy of a non-rotating object:`0.5 * mass * velocity**2`  Modules 📦  • Using modulesLet’s say that we need to use the cosine function.This function, and many other mathematical functions arelocated in the `math` module.To tell Python that we want to access the features offered bythis module, we can give the statement `import math`.Now the module is loaded into memory.We can now call the function like this:```pythonmath.cos(0)1.0```Note that we need to include the module name where the `cos`function is found.This is because other modules may have a function (or otherattribute of a module) with the same name.This usage of different namespace for each module preventsname clashes. For example, functions `gzip.open`, `os.open` are not to be confusedwith the builtin `open` function.<jupyter_code>#Try it here!<jupyter_output><empty_output><jupyter_text>• Breaking the namespaceIf the cosine is needed a lot, then it might be tedious toalways specify the namespace, especially if the name of thenamespace/module is long.For these cases there is another way of importing modules.Bring a name to the current scope with`from math import cos` statement.Now we can use it without the namespace specifier: `cos(1)`.Several names can be imported to the current scope with`from math import name1, name2, ...`Or even all names of the module with `from math import *`The last form is sensible only in few cases, normally it justconfuses things since the user may have no idea what nameswill be imported.  • Module lookupWhen we try to import a module `mod` with the importstatement, the lookup proceeds in the following order:* Check if it is a builtin module* Check if the file `mod.py` is found in any of the folders inthe list `sys.path`. The first item in this list is the currentfolderWhen Python is started, the `sys.path` list is initialised withthe contents of the `PYTHONPATH` environment variable  • Module hierarchyThe standard library contains hundreds of modules.Hence, it is hard to comprehend what the library includes.The modules therefore need to be organised somehow.In Python the modules can be organised into hierarchies using*packages*.A package is a module that can contain other packages andmodules.For example, the `numpy` package contains subpackages `core`,`distutils`, `f2py`, `fft`, `lib`, `linalg`, `ma`, `numarray`, `oldnumeric`,`random`, and `testing`.And package `numpy.linalg` in turn contains modules `linalg`,`lapack_lite` and `info`.  • Importing from packagesThe statement `import numpy` imports the top-level package `numpy`and its subpackages. * `import numpy.linalg` imports the subpackage only, and* `import numpy.linalg.linalg` imports the module onlyIf we want to skip the long namespace specification, we canuse the form```pythonfrom numpy.linalg import linalg```or```pythonfrom numpy.linalg import linalg as lin```if we want to use a different name for the module. The following command imports the function `det` (computes the determinant of a matrix) from the module linalg, which is contained in a subpackage linalg, which belongs to package numpy:```pythonfrom numpy.linalg.linalg import det```Had we only imported the top-level package `numpy` we would have to refer to the `det` function with the full name `numpy.linalg.linalg.det`.Here's a recap of the module hierarchy:```numpy    package  .linalg   subpackage  .linalg   module  . det     function```  • Correspondence between folder and module hierarchiesThe packages are represented by folders in the filesystem.The folder should contain a file named `__init__.py` thatmakes up the package body. This handles the initialisation ofthe package.The folder may contain also further folders(subpackages) or Python files (normal modules).```a/    __init__.py    b.py    c/        __init__.py        d.py        e.py```![package.svg](https://github.com/csmastersUH/data_analysis_with_python_2020/blob/master/package.svg?raw=1)  • Contents of a moduleSuppose we have a module named `mod.py`.All the assignments, class definitions with the `class` statement,and function definitions with `def` statement will create newattributes to this module.Let’s import this module from another Python file using the`import mod` statement.After the import we can access the attributes of the moduleobject using the normal dot notation: `mod.f()`,`mod.myclass()`, `mod.a`, etc.Note that Python doesn’t really have global variables that arevisible to all modules. All variables belong to some modulenamespace.One can query the attributes of an object using the `dir` function. With noparameters, it shows the attributes of the current module. Try executing `dir()` inan IPython shell or in a Jupyter notebook! After that, define the following attributes, and try running `dir()`again:```pythona=5def f(i):    return i + 1```The above definitions created a *data attribute* called `a` and a *function attribute* called `f`.We will talk more about attributes next week when we will talk about objects.Just like other objects, the module object contains itsattributes in the dictionary `modulename.__dict__`Usually a module contains at least the attributes `__name__` and`__file__`. Other common attributes are `__version__`,`__author__` and `__doc__` , which contains the docstring of themodule.If the first statement of a file is a string, this is taken as thedocstring for that module. Note that the docstring of the module really must be the first non-empty non-comment line.The attribute `__file__` is always the filename of the module.The module attribute `__name__` has value `“__main__”` if we in are the main program,otherwise some other module has imported us and nameequals `__file__`. In Python it is possible to put statements on the top-level of our module `mod` so that they don't belong to any function. For instance like this:```pythonfor _ in range(3):    print("Hello")```But if somebody imports our module with `import mod`, then all the statements at the top-level will be executed. This may be surprising to the user who imported the module. The user will usually say, explicitly when he/she wants to execute some code from the imported module.It is better style to put these statements inside some function. If they don't fit in any other function, then you can use, for example, the function named `main`, like this:```pythondef main():    for _ in range(3):        print("Hello")if __name__ == "__main__":     We call main only when this module is not being imported, but directly executed    main()                     for example with 'python3 mod.py'```You probably have seen this mechanism used in the exercise stubs.Note that in Python the `main` has no special meaning, it is just our convention to use it here.Now if somebody imports `mod`, the `for` loop won't be automatically executed. If we want, we can call it explicitly with `mod.main()`.  ```pythonfor _ in range(3):    print("Hello")```  Interactive ways of representing your data 📊 📈<jupyter_code>import ipywidgets as widgets

slider = widgets.IntSlider(20, min=0, max=100)
slider
# load an example dataset
from vega_datasets import data
cars = data.cars()

# plot the dataset, referencing dataframe column names
import altair as alt
alt.Chart(cars).mark_bar().encode(
  x=alt.X('Miles_per_Gallon', bin=True),
  y='count()',
)
# load an example dataset
from vega_datasets import data
cars = data.cars()

# plot the dataset, referencing dataframe column names
import altair as alt
alt.Chart(cars).mark_point().encode(
  x='Horsepower',
  y='Miles_per_Gallon',
  color='Origin'
).interactive()
import altair as alt
import ipywidgets as widgets
from vega_datasets import data

source = data.stocks()

stock_picker = widgets.SelectMultiple(
    options=source.symbol.unique(),
    value=list(source.symbol.unique()),
    description='Symbols')

# The value of symbols will come from the stock_picker.
@widgets.interact(symbols=stock_picker)
def render(symbols):
  selected = source[source.symbol.isin(list(symbols))]

  return alt.Chart(selected).mark_line().encode(
      x='date',
      y='price',
      color='symbol',
      strokeDash='symbol',
  )<jupyter_output><empty_output><jupyter_text>Appendix B (Homework) 🤪 Python DeepDive  InterleaveWrite function `interleave` that gets arbitrary number of lists as parameters. You may assume that all the lists have equal length. The function should return one list containing all the elements from the input lists interleaved.Test your function from the `main` function of the program.Example:`interleave([1,2,3], [20,30,40], ['a', 'b', 'c'])`should return`[1, 20, 'a', 2, 30, 'b', 3, 40, 'c']`.Use the `zip` function to implement `interleave`. Remember the `extend` method of list objects.  FunctionsA function is defined with the `def` statement. Let's do a doubling function.<jupyter_code>def double(x):
    "This function multiplies its argument by two."
    return x*2
print(double(4), double(1.2), double("abc")) # It even happens to work for strings!<jupyter_output><empty_output><jupyter_text>The double function takes only one parameter. Notice the *docstring* on the second line. It documents the purpose and usage of the function. Let's try to access it.<jupyter_code>print("The docstring is:", double.__doc__)
help(double)   # Another way to access the docstring<jupyter_output><empty_output><jupyter_text>Most of Python's builtin functions, classes, and modules should contain a docstring.<jupyter_code>help(print)<jupyter_output><empty_output><jupyter_text>Here's another example function:<jupyter_code>def sum_of_squares(a, b):
    "Computes the sum of arguments squared"
    return a**2 + b**2
print(sum_of_squares(3, 4))<jupyter_output><empty_output><jupyter_text>Note the terminology: in the function definition the names a and b are called parameters of the function; in the function call, however, 3 and 4 are called arguments to the function. It would be nice that the number of arguments could be arbitrary, not just two. We could pass a list to the function as a parameter.<jupyter_code>def sum_of_squares(lst):
    "Computes the sum of squares of elements in the list given as parameter"
    s=0
    for x in lst:
        s += x**2
    return s
print(sum_of_squares([-2]))
print(sum_of_squares([-2,4,5]))<jupyter_output><empty_output><jupyter_text>This works perfectly! There is however some extra typing with the brackets around the lists. Let's see if we can do better:<jupyter_code>def sum_of_squares(*t):
    "Computes the sum of squares of arbitrary number of arguments"
    s=0
    for x in t:
        s += x**2
    return s
print(sum_of_squares(-2))
print(sum_of_squares(-2,4,5))<jupyter_output><empty_output><jupyter_text>The strange looking argument notation (the star) is called *argument packing*. It packs all the given positional arguments into a tuple `t`. We will encounter tuples again later, but it suffices now to say that tuples are *immutable* lists. With the `for` loop we can iterate through all the elements in the tuple.Conversely, there is also syntax for *argument unpacking*. It has confusingly exactly same notation as argument packing (star), but they are separated by the location where used. Packing happens in the parameter list of the functions definition, and unpacking happens where the function is called:<jupyter_code>lst=[1,5,8]
print("With list unpacked as arguments to the functions:", sum_of_squares(*lst))
# print(sum_of_squares(lst))    # Does not work correctly<jupyter_output><empty_output><jupyter_text>The second call failed because the function tried to raise the list of numbers to the second power. Inside the function body we have `t=([1,5,8])`, where the parentheses denote a tuple with one element, a list. In addition to positional arguments we have seen so far, a function call can also have *named arguments*. An example will explain this concept best:<jupyter_code>def named(a, b, c):
    print("First:", a, "Second:", b, "Third:", c)
named(5, c=7, b=8)<jupyter_output><empty_output><jupyter_text>Note that the named arguments didn't need to be in the same order as in the function definition.The named arguments must come after the positional arguments. For example, the following function call is illegal `named(a=5, 7, 8)`.One can also specify an optional parameter by giving the parameter a default value. The parameters that have default values must come after those parameters that don't. We saw that the parameters of the `print` function were of form `print(value, ..., sep=' ', end='\n', file=sys.stdout, flush=False)`. There were four parameters with default values. If some default values don't suit us, we can give them in the function call using the name of the parameter:<jupyter_code>print(1, 2, 3, end=' |', sep=' -*- ')
print("first", "second", "third", end=' |', sep=' -*- ')<jupyter_output><empty_output><jupyter_text>We did not need to specify all the parameters with default values, only those we wanted to change.Let's go through another example of using parameters with default values:<jupyter_code>def length(*t, degree=2):
    """Computes the length of the vector given as parameter. By default, it computes
    the Euclidean distance (degree==2)"""
    s=0
    for x in t:
        s += abs(x)**degree
    return s**(1/degree)
print(length(-4,3))
print(length(-4,3, degree=3))<jupyter_output><empty_output><jupyter_text>With the default parameter this is the Euclidean distance, and if $p\ne 2$ it is called [p-norm](https://en.wikipedia.org/wiki/P-norm). We saw that it was possible to use packing and unpacking of arguments with the * notation, when one wants to specify arbitrary number of *positional arguments*. This is also possible for arbitrary number of named arguments with the `**` notation. We will talk about this more in the data structures section.   • Visibility of variablesFunction definition creates a new *namespace* (also called local scope). Variables created inside this scope are not available from outside the function definition. Also, the function parameters are only visible inside the function definition. Variables that are not defined inside any function are called `global variables`.Global variable are readable also in local scopes, but an assignment creates a new local variable without rebinding the global variable. If we are inside a function, a local variable hides a global variable by the same name:<jupyter_code>i=2           # global variable
def f():
    i=3       # this creates a new variable, it does not rebind the global i
    print(i)  # This will print 3    
f()
print(i)      # This will print 2<jupyter_output><empty_output><jupyter_text>If you really need to rebind a global variable from a function, use the `global` statement. Example:<jupyter_code>i=2
def f():
    global i
    i=5       # rebind the global i variable
    print(i)  # This will print 5
f()
print(i)      # This will print 5<jupyter_output><empty_output><jupyter_text>Unlike languages like C or C++, Python allows defining a function inside another function. This *nested* function will have nested scope:<jupyter_code>def f():            # outer function
    b=2
    def g():        # inner function
        #nonlocal b # Without this nonlocal statement,
        b=3         # this will create a new local variable
        print(b)
    g()
    print(b)
f()<jupyter_output><empty_output><jupyter_text>Try first running the above cell and see the result. Then uncomment the nonlocal stamement and run the cell again. The `global` and `nonlocal` statements are similar. The first will force a variable refer to a global variable, and the second will force a variable to refer to the variable in the nearest outer scope (but not the global scope).  Data structuresThe main data structures in Python are strings, lists, tuples, dictionaries, and sets. We saw some examples of lists, when we discussed `for` loops. And we saw briefly tuples when we introduced argument packing and unpacking. Let's get into more details now. SequencesA *list* contains arbitrary number of elements (even zero) that are stored in sequential order. The elements are separated by commas and written between brackets. The elements don't need to be of the same type. An example of a list with four values:<jupyter_code>[2, 100, "hello", 1.0]<jupyter_output><empty_output><jupyter_text>A *tuple* is fixed length, immutable, and ordered container. Elements of tuple are separated by commas and written between parentheses. Examples of tuples:<jupyter_code>(3,)               # a singleton
(1,3)              # a pair
(1, "hello", 1.0); # a triple<jupyter_output><empty_output><jupyter_text>Note the difference between `(3)` and `(3,)`. Because the parentheses can also be used to group expressions, the first one defines an integer, but the second one defines a tuple with single element. As we can see, both lists and tuples can contain values of different type.List, tuples, and strings are called *sequences* in Python, and they have several commonalities:* their length can be queried with the `len` function* `min` and `max` function find the minimum and maximum element of a sequence, and `sum` adds all the elements of numbers together* Sequences can be concatenated with the `+` operator, and repeated with the `*` operator: `"hi"*3=="hihihi"`* Since sequences are ordered, we can refer to the elements of a sequences by integers using the *indexing* notation: `"abcd"[2] == "c"`* Note that the indexing begins from 0* Negative integers start indexing from the end: -1 refers to the last element, -2 refers to the second last, and so on Above we saw that we can access a single element of a sequence using *indexing*. If we want a subsequence of a sequence, we can use the *slicing* syntax. A slice consists of elements of the original sequence, and it is itself a sequence as well. A simple slice is a range of elements:<jupyter_code>s="abcdefg"
s[1:4]<jupyter_output><empty_output><jupyter_text>Note that Python ranges exclude the last index. The generic form of a slice is`sequence[first:last:step]`. If any of the three parameters are left out, they are set to default values as follows: first=0, last=len(L), step=1. So, for instance "abcde"[1:]=="bcde". The step parameter selects elements that are step distance apart from each other. For example:<jupyter_code>print([0,1,2,3,4,5,6,7,8,9][::3])<jupyter_output><empty_output><jupyter_text>• Modifying listsWe can assign values to elements of a list by indexing or by slicing. An example:<jupyter_code>L=[11,13,22,32]
L[2]=10          # Changes the third element
print(L)<jupyter_output><empty_output><jupyter_text>Or we can assign a list to a slice:<jupyter_code>L[1:3]=[4]
print(L)<jupyter_output><empty_output><jupyter_text>We can also modify a list by using *mutating methods* of the `list` class, namely the methods `append`, `extend`, `insert`, `remove`, `pop`, `reverse`, and `sort`. Try Python's help functionality to find more about these methods: e.g. `help(list.extend)` or `help(list)`.Note that we cannot perform these modifications on tuples or strings since they are *immutable*  • Generating numerical sequencesTrivial lists can be tedious to write: `[0,1,2,3,4,5,6]`. The function `range` creates numeric ranges automatically. The above sequence can be generated with the function call `range(7)`. Note again that then end value is not included  in the sequence. An example of using the `range` function:<jupyter_code>L=range(3)
for i in L:
    print(i)
# Note that L is not a list!
print(L)<jupyter_output><empty_output><jupyter_text>So `L` is not a list, but it is a sequence. We can for instace access its last element with `L[-1]`. If really needed, then it can be converted to a list with the `list` constructor:<jupyter_code>L=range(10)
print(list(L))<jupyter_output><empty_output><jupyter_text>Note that using a range consumes less memory than the corresponding list. This is because in a list all the elements are stored in the memory, whereas the range generates the requested elements only when needed. For example, when the for loop asks for the next element from the range at each iteration, only a single element from the range exists in memory at the same time. This makes a big difference when using large ranges, like range(1000000). The `range` function works in similar fashion as slices. So, for instance the step of the sequence can be given:<jupyter_code>print(list(range(0, 7, 2)))<jupyter_output><empty_output><jupyter_text>•  Sorting sequencesIn Python there are two ways to sort sequences. The `sort` *method* modifies the original list, whereas the `sorted` *function* returns a new sorted list and leaves the original intact. A couple of examples will demonstrate this:<jupyter_code>L=[5,3,7,1]
L.sort()      # here we call the sort method of the object L
print(L)
L2=[6,1,7,3,6]
print(sorted(L2))
print(L2)<jupyter_output><empty_output><jupyter_text>The parameter `reverse=True` can be given (both to `sort` and `sorted`) to get descending order of elements:<jupyter_code>L=[5,3,7,1]
print(sorted(L, reverse=True))<jupyter_output><empty_output><jupyter_text>• Zipping sequencesThe `zip` function combines two (or more) sequences into one sequence. If, for example, two sequences are zipped together, the resulting sequence contains pairs. In general, if `n` sequences are zipped together, the elements of the resulting sequence contains `n`-tuples. An example of this:<jupyter_code>L1=[1,2,3]
L2=["first", "second", "third"]
print(zip(L1, L2))               # Note that zip does not return a list, like range
print(list(zip(L1, L2)))         # Convert to a list<jupyter_output><empty_output><jupyter_text>Here's another example of using the `zip` function.<jupyter_code>days="Monday Tuesday Wednesday Thursday Friday Saturday Sunday".split()
weathers="rainy rainy sunny cloudy rainy sunny sunny".split()
temperatures=[10,12,12,9,9,11,11]
for day, weather, temperature in zip(days,weathers,temperatures):
    print(f"On {day} it was {weather} and the temperature was {temperature} degrees celsius.")

# Or equivalently:
#for t in zip(days,weathers,temperatures):
#    print("On {} it was {} and the temperature was {} degrees celsius.".format(*t))<jupyter_output><empty_output><jupyter_text>If the sequences are not of equal length, then the resulting sequence will be as long as the shortest input sequence is.  • Enumerating sequencesIn some other programming languages one iterates through the elements using their indices (0,1, ...) in the sequence. In Python we normally don't need to think about indices when iterating, because the `for` loop allows simpler iteration through the elements. But sometimes you really need to know the index of the current element in the sequence. In this case one uses Python's `enumerate` function. In the next example we would like find the second occurrence of integer 5 in a list.<jupyter_code>L=[1,2,98,5,-1,2,0,5,10]
counter = 0
for i, x in enumerate(L):
    if x == 5:
        counter += 1
        if counter == 2:
            break
print(i)<jupyter_output><empty_output><jupyter_text>The `enumerate(L)` function call can be thought to be equivalent to `zip(range(len(L)), L)`.  • DictionariesA *dictionary* is a dynamic, unordered container. Instead of using integers to access the elements of the container, the dictionary uses *keys* to access the stored *values*. The dictionary can be created by listing the comma separated key-value pairs in braces. Keys and values are separated by a colon. A tuple (key,value) is called an *item* of the dictionary.Let's demonstrate the dictionary creation and usage:<jupyter_code>d={"key1":"value1", "key2":"value2"}
print(d["key1"])
print(d["key2"])<jupyter_output><empty_output><jupyter_text>Keys can have different types even in the same container. So the following code is legal:`d={1:"a", "z":1}`. The only restriction is that the keys must be *hashable*. That is, there has to be a mapping from keys to integers. Lists are *not* hashable, but tuples are!There are alternative syntaxes for dictionary creation:<jupyter_code>dict([("key1", "value1"), ("key2", "value2"), ("key3", "value3")]) # list of items
dict(key1="value1", key2="value2", key3="value3");<jupyter_output><empty_output><jupyter_text>If a key is not found in a dictionary, the indexing `d[key]` results in an error (*exception* `KeyError`). But an assignment with a non-existing key causes the key to be added in the dictionary associated with the corresponding value:<jupyter_code>d={}
d[2]="value"
print(d)
# d[1]   # This would cause an error<jupyter_output><empty_output><jupyter_text>Dictionary object contains several non-mutating methods:```d.copy()d.items()d.keys()d.values()d.get(k[,x])```Some methods mutate the dictionary:```d.clear()d.update(d1)d.setdefault(k[,x])d.pop(k[,x])d.popitem()```Try out some of these in the below cell. You can find more info with `help(dict)` or `help(dict.keys)`.<jupyter_code>d=dict(a=1, b=2, c=3, d=4, e=5)
d.values()<jupyter_output><empty_output><jupyter_text>• SetsSet is a dynamic, unordered container. It works a bit like dictionary, but only the keys are stored. And each key can be stored only once. The set requires that the keys to be stored are hashable. Below are a few ways of creating a set:<jupyter_code>s={1,1,1}
print(s)
s=set([1,2,2,'a'])
print(s)
s=set()  # empty set
print(s)
s.add(7) # add one element
print(s)<jupyter_output><empty_output><jupyter_text>A more useful example:<jupyter_code>s="mississippi"
print(f"There are {len(set(s))} distinct characters in {s}")<jupyter_output><empty_output><jupyter_text>The `set` provides the following non-mutating methods:<jupyter_code>s=set()
s1=set()
s.copy()
s.issubset(s1)
s.issuperset(s1)
s.union(s1)
s.intersection(s1)
s.difference(s1)
s.symmetric_difference(s1);<jupyter_output><empty_output><jupyter_text>The last four operation can be tedious to write to create a more complicated expression. The alternative is to use the corresponding operator forms: `|`, `&`, `-`, and `^`. An example of these:<jupyter_code>s=set([1,2,7])
t=set([2,8,9])
print("Union:", s|t)
print("Intersection:", s&t)
print("Difference:", s-t)
print("Symmetric difference", s^t)<jupyter_output><empty_output><jupyter_text>There are also the following mutating methods:```s.add(x)s.clear()s.discard()s.pop()s.remove(x)```And the set operators `|`, `&`, `-`, and `^` have the corresponding mutating, augmented assignment forms: `|=`, `&=`, `-=`, and `^=`.  Compact way of creating data structuresWe can now easily create complicated data structures using `for` loops:<jupyter_code>L=[]
for i in range(10):
    L.append(i**2)
print(L)<jupyter_output><empty_output><jupyter_text>Because this kind of pattern is often used, Python offers a short-hand for this. A *list comprehension* is an expression that allows creating complicated lists on one line. The notation is familiar from mathematics: $\{a^3 : a \in \{1,2, \ldots, 10\}\}$ The same written in Python as a list comprehension:<jupyter_code>L=[ a**3 for a in range(1,11)]
print(L)<jupyter_output><empty_output><jupyter_text>The generic form of a list comprehension is:`[ expression for element in iterable lc-clauses ]`.Let's break this syntax into pieces. The iterable can be any sequence (or something more general). The lc-clauses consists of zero or more of the following clauses:* for elem in iterable* if expressionA more complicated example. How would you describe these numbers?<jupyter_code>L=[ 100*a + 10*b +c for a in range(0,10)
                    for b in range(0,10)
                    for c in range(0,10) 
                    if a <= b <= c]
print(L)<jupyter_output><empty_output><jupyter_text>• Miscellaneous stuffTo find out whether a container includes an element, the `in` operator can be used. The operator returns a truth value. Some examples of the usage:<jupyter_code>print(1 in [1,2])
d=dict(a=1, b=3)
print("b" in d)
s=set()
print(1 in s)
print("x" in "text")<jupyter_output><empty_output><jupyter_text>As a special case, for strings the `in` operator can be used to check whether a string is part of another string:<jupyter_code>print("issi" in "mississippi")
print("issp" in "mississippi")<jupyter_output><empty_output><jupyter_text>Elements of a container can be unpacked into variables:<jupyter_code>first, second = [4,5]
a,b,c = "bye"
print(c)
d=dict(a=1, b=3)
key1, key2 = d
print(key1, key2)<jupyter_output><empty_output><jupyter_text>In membership testing and unpacking only the keys of a dictionary are used, unless either values or items (like below) are explicitly asked.<jupyter_code>for key, value in d.items():
    print(f"For key '{key}' value {value} was stored")<jupyter_output><empty_output><jupyter_text>To remove the binding of a variable, use the `del` statement. For example:<jupyter_code>s="hello"
del s
# print(s)    # This would cause an error<jupyter_output><empty_output><jupyter_text>To delete an item from a container, the `del` statement can again be applied:<jupyter_code>L=[13,23,40,100]
del L[1]
print(L)<jupyter_output><empty_output><jupyter_text>In similar fashion `del` can be used to delete a slice. Later we will see that `del` can delete attributes from an object. If one needs only to iterate through the list once, it is more memory efficient to use a *generator expression* instead. The only thing that changes syntactically is that the surrounding brackets are replaced by parentheses:<jupyter_code>G = ( 100*a + 10*b + c for a in range(0,10)
                       for b in range(0,10)
                       for c in range(0,10) 
                       if a <= b <= c )
print(sum(G))   # This iterates through all the elements from the generator
print(sum(G))   # It doesn't restart from the beginning, so all elements are already consumed<jupyter_output><empty_output><jupyter_text>Note above that one can only iterate through the generator once. Similary a *dictionary comprehension* creates a dictionary:<jupyter_code>d={ k : k**2 for k in range(10)}
print(d)<jupyter_output><empty_output><jupyter_text>And a *set comprehension* creates a set:<jupyter_code>s={ i*j for i in range(10) for j in range(10)}
print(s)<jupyter_output><empty_output><jupyter_text>• Creating stringsA string is a sequence of characters commonly used to store input or output data in a program. The characters of a string are specified either between single (`'`) or double (`"`) quotes. This optionality is useful if, for example, a string needs to contain a quotation mark:"I don't want to go!". You can also achieve this by *escaping* the quotation mark with the backslash: 'I don\\'t want to go'.The string can also contain other escape sequences like `\n` for newline and `\t` for a tabulator. See [literals](https://docs.python.org/3/reference/lexical_analysis.htmlliterals) for a list of all escape sequences.<jupyter_code>print("One\tTwo\nThree\tFour")<jupyter_output><empty_output><jupyter_text>A string containing newlines can be easily given within triple double or triple single quotes:<jupyter_code>s="""A string
spanning over
several lines"""
s<jupyter_output><empty_output><jupyter_text>Although we can concatenate strings using the `+` operator, for effiency reasons, one should use the `join` method to concatenate larger number of strings:<jupyter_code>a="first"
b="second"
print(a+b)
print(" ".join([a, b, b, a]))   # Here we introduce the join function<jupyter_output><empty_output><jupyter_text>Sometimes printing by concatenation from pieces can be clumsy:<jupyter_code>print(str(1) + " plus " + str(3) + " is equal to " + str(4))
# slightly better
print(1, "plus", 3, "is equal to", 4)<jupyter_output><empty_output><jupyter_text>The multiple catenation and quotation characters break the flow of thought. *String interpolation* offers somewhat easier syntax.There are multiple ways to do sting interpolation:* Python format strings* the `format` method* f-stringsExamples of these can be seen below:<jupyter_code>print("%i plus %i is equal to %i" % (1, 3, 4))     # Format syntax

print("{} plus {} is equal to {}".format(1, 3, 4)) # Format method

print(f"{1} plus {3} is equal to {4}")             # f-string<jupyter_output><empty_output><jupyter_text>The `i` format specifier in the format syntacs corresponds to integers and the specifier `f` corresponds to floats. When using f-strings or the `format` method, integers use `d` instead. In format strings specifiers can usually be omitted and are generally used only when specific formatting is required. For example in f-strings `f"{4:3d}"` would specify the number 4 left padded with spaces to 3 digits.It is often useful to specify the number of decimals when printing floats:<jupyter_code>print("%.1f %.2f %.3f" % (1.6, 1.7, 1.8))               # Old style
print("{:.1f} {:.2f} {:.3f}".format(1.6, 1.7, 1.8))     # newer style
print(f"{1.6:.1f} {1.7:.2f} {1.8:.3f}")                 # f-string<jupyter_output><empty_output><jupyter_text>The specifier `s` is used for strings. An example:<jupyter_code>print("%s concatenated with %s produces %s" % ("water", "melon", "water"+"melon"))
print("{0} concatenated with {1} produces {0}{1}".format("water", "melon"))
print(f"{'water'} concatenated with {'melon'} produces {'water' + 'melon'}")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08392133313316319, 'fraction_numerical': 0.00992343491968173, 'mean_word_length': 5.0870876359316455, 'pattern_counts': {'":': 4, '<': 349, '<?xml version=': 0, '>': 365, 'https://': 20, 'lorem ipsum': 0, 'www.': 16, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '85', 'dir': 'jupyter-structured-clean-dedup', 'id': '339360', 'license': 'MIT', 'n_tokens_mistral': 18986, 'n_tokens_neox': 18035, 'n_words': 9026, 'path': 'tutorials/lab01_intro.ipynb', 'repo_name': 'pablo-arantes/ibm3202', 'size': '66630'}