<jupyter_start><jupyter_text>Introduction to Jupyter Notebooks*Original Authors: <NAME>, <NAME>, <NAME>**Purpose*: We're going to use programming tools throughout this workshop. There are many benefits to using programming to do data science, but to reap these benefits you will need to know some basics about coding in Python, and working in Jupyter notebooks.  Highest-level ViewThroughout this day of the workshop, we will be writing computer software. Before diving into the details, it is beneficial to understand what tools we'll be using, and why. PythonPython is a [*programming language*](https://www.python.org), and an extremely popular one at that. Python is praised for its readability, is consistently ranked as a top programming language, and is used by both top [software companies](https://en.wikipedia.org/wiki/Python_(programming_language)Uses) and scientists doing computation. All of this is to say that knowing how to write Python code is a valuable skill. JupyterJupyter is a [*programming environment*](https://jupyter.org), often used for writing and sharing Python code. Just as one could choose to write English on stone tablets or on paper, one could choose to write Python code in simple files or in *jupyter notebooks*. These notebooks are useful for a number of reasons:1. Multimedia: This document you're reading now is a jupyter notebook -- it is a combination of (executable) Python code and (formatted) plain text. By allowing the juxtaposition of different media (code and natural language, to name a couple options), one can provide human-readable context next to executable software.2. Saves state: Jupyter notebooks *save state*, allowing you to run some code, save the results, close the document and come back to it later. This is certainly possible in other coding environments, but jupyter has built-in facilities to make this easy.3. Reproducible: [Reproducibility](https://en.wikipedia.org/wiki/Reproducibility) is a critical issue in science -- results cannot be considered scientifically valid unless they can be reproduced under similar conditions. The same principles apply to computational experiments; all too often software is written under time pressure, resulting in scattered documentation and analysis. Consolidating documentation and analysis within a single notebook facilitates scientific reproducibility.Jupyter notebooks are not *always* the best media for writing code; for instance, python *packages* are not written in jupyter notebooks. However, for our purposes, jupyter is a good solution. --- How to use this NotebookThis jupyter notebook is an *interactive exercise* -- it contains some reading, but the focus is on fill-in-the-blank sections that are meant to test your understanding and spark further learning. Pedagogy research shows that so-called *active learning* -- learning by doing -- is far more effective than lecture alone. Thus, you will get the most out of this workshop if you do the work! Try -- to the best of your ability -- to complete the exercises in these notebooks, and please feel free to ask the instructors questions if you get stuck. (It's what we're here for.)```{admonition} Run this notebook locally!:class: warningIf you're reading this on the workshop website---and *not* in your local jupyter reader---then it is *necessary* that you download this notebook locally and open it in jupyter. Look for the download bar in the top-right of this page, as pictured below.**TODO UPDATE SETUP INSTRUCTIONS**``` Exercise portions of the notebook will be marked with a "q" subheading, like the following: __q0__: An example questionHere's an example of an exercise prompt. Follow the directions here to start the exercise.  Jupyter Introduction  Cells A Jupyter notebook is organized as a sequence of different **cells**. Cells contain chunks of information and allow you to switch between text and code.**Current cell**: The currently-selected cell will have a color bar appear on the far left. Use your cursor or arrow keys to select different cells.**Create new cell**: To create a new cell, click the "**+**" icon at the top. This will create a cell below the currently selected cell.**Delete a cell**: To delete the currently-selected cell, click the "**scissors**" icon at the top.  __q1__ Create a new cell**Create a new cell below this one.** Note that it has a different background color! This is because cells have different *types*.  Types of cellsThere are two types of cells we will discuss here: `Markdown` and `Code`. A single cell can only have 1 type.**Markdown**: These are text cells that use the Markdown language for formatting. You can put instructions, headings, links, images, and much more; even $\LaTeX$ ([a language for typesetting mathematics](https://www.latex-project.org)). Here is a [cheatsheet](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet) for Markdown syntax.**Code**: These are cells where you can type Python code, just as you would in a `.py` file. They will have a label `In [ ]:` to their left.To switch the type of cell, you can use the dropdown menu at the top of the cell. (Note: While you're moused-over this cell, the dropdown menu should read `Markdown`.)  Edit and format / execute cellsTo edit a Markdown cell, **triple-click** within the cell if you currently see a blue bar. The color bar on the left will turn green. Then type and edit as normal.To format ("execute") a Markdown cell, press **Ctrl+Enter** when in "edit" mode. The color bar on the left will turn blue and the cursor will disappear. Your formatting will also take effect.---To edit a Code cell, click within the cell. The color bar on the left will turn green. Then type and edit as normal.To execute a Code cell, press **Ctrl+Enter** while it is selected. The color bar on the left will turn blue and the cursor will disappear.  Code cellsSo far we've only seen Markdown cells. Let's see what we can do with Code cells! Try selecting the code cell below this one and executing it.<jupyter_code># This is a python code cell
6 * 7<jupyter_output><empty_output><jupyter_text>Note that code cells are visually distinct from markdown cells -- they have a grey background and the `[-]:` prefix.  Python Introduction  Simple operationsWe can use python like a calculator to carry out simple operations. Some of these include:- Simple math operators `(+, -, *, /)`- Other math operators; exponentiation `10 ** 2 == 100`; modulus `3 % 2 == 1`- Parentheses `(, )` for grouping operations- Python functions; console output `print("Hello")`  __q2__ Average some valuesUse basic python operations to average the following numbers: `[2, 4, 6, 8]`.<jupyter_code>###
# TASK: Average the numbers [2, 4, 6, 8]
# TODO: Use basic python operations to compute the arithmetic mean
###

# task-begin
# -- WRITE YOUR CODE HERE -----
# task-end
# solution-begin
(2 + 4 + 6 + 8) / 4
# solution-end<jupyter_output><empty_output><jupyter_text>VariablesJupyter notebooks will save variables across different cells. We can assign values with the `=` operator. Furthermore, typing just the variable on the last line of a code cell will typically display the variable's values/attributes.<jupyter_code>a = 1
b = 2
c = a * b<jupyter_output><empty_output><jupyter_text>__q3__ Display a variableDisplay the value of `c` in the chunk below.<jupyter_code>###
# TASK: Display a variable
# TODO: Type the variable name below and execute
###
# solution-begin
c
# solution-end<jupyter_output><empty_output><jupyter_text>A couple things happened here. First, the variable was saved. Second, Jupyter notebooks will display the value of any lone variables in the last line of each Code cell. This is helpful for debugging.If you want to display multiple quantities from a single cell, you can always use a print statement `print(variable)`. For instance, we could write```print(a)print(b)print(c)```You can also see that Code cells with output will have an associated `Out []` right below. Notice what happens to the cell below when the code is running.  ListsA *list* is a Python [data structure](https://en.wikipedia.org/wiki/Data_structure). We'll use lists (and other *iterable* data structures) quite a bit when doing data analysis -- generally we have many observations (elements) to consider.Lists (i.e. arrays) in Python are created with square brackets. The entries of a list are called *elements*.<jupyter_code>l = [1, 2, 3]<jupyter_output><empty_output><jupyter_text>The `append()` method adds a single element to the end of the array. Note how we can add elements of different types to the same list (unlike Java or C++).<jupyter_code>l = [1, 2, 3]
l.append("a")
l<jupyter_output><empty_output><jupyter_text>Note that I *needed* to place `l` at the end of the cell to display the result. `append()` modified the list *in place* -- it makes the list longer.We can also *combine* lists with the `+` operator.<jupyter_code>[1, 2, 3] + [4, 5, 6]<jupyter_output><empty_output><jupyter_text>Note that I *did not need* to write an additional line to print the result. This is because the `+` operator *returns* a list as a result, which is printed by jupyter.  __q4__ Working with listsTest your understanding of lists; make a prediction about what the following code will display:```list_a = [1, 2, 3]list_b = [4, 5, 6]print(list_a + list_b)print(list_a.append(list_b))```*After* you've made a prediction, **make a new cell and test your prediction**.  Indexing and SlicingWe can access specific elements of a list (and other *iterables*) by *indexing*. Python is uses zero-based indexing, meaning the first element of an iterable is `0`, the second is `1`, and so on.```Index    0    1    2    3    4List  [ "a", "b", "c", "d", "e" ]```For instance:<jupyter_code>l = ["a", "b", "c"]
print("l[0] = {}".format(l[0]))
print("l[1] = {}".format(l[1]))
print("l[2] = {}".format(l[2]))<jupyter_output>l[0] = a
l[1] = b
l[2] = c<jupyter_text>We can also use *negative indices* to conveniently access from the "right".<jupyter_code>print("l[-1] = {}".format(l[-1]))
print("l[-2] = {}".format(l[-2]))<jupyter_output>l[-1] = c
l[-2] = b<jupyter_text>In Python, it's easy to take subsets of lists and strings (i.e. substrings) using **slicing**. Given a list (string) named `var`, we can take a subset of elements (characters) ranging from `low` to `high-1` with the syntax `var[low:high]`. Run the following code and observe what happens.<jupyter_code>var = [0, 1, 2, 3, 4, 5]
# var = '012345'
print('The second element:', var[1])
print('The second and third elements:', var[1:3])
print('The second element onwards:', var[1:])
print('Everything before the second element:', var[:1])
print('Everything before the second TO LAST element:', var[:-2])<jupyter_output>The second element: 1
The second and third elements: [1, 2]
The second element onwards: [1, 2, 3, 4, 5]
Everything before the second element: [0]
Everything before the second TO LAST element: [0, 1, 2, 3]<jupyter_text>__q5__ Take the middleUsing your understanding of python indexing & slicing, select the *middle three elements* of the following list.<jupyter_code>###
# TASK: Take the middle values
# TODO: Use python slicing to take the middle three values
###
test = [-2, -1, 0, +1, +2]

# -- WRITE YOUR CODE BELOW -----
# solution-begin
test[1:-1]
# solution-end<jupyter_output><empty_output><jupyter_text>ModulesBy default, python provides a relatively small set of functionality. To carry out more specialized operations, we can *import module*. A module is a set of tools that help us carry out some tasks. For instance, the following module will help us control the execution of python code.<jupyter_code># Import a module first
from time import sleep

# Pause code execution for 5 seconds
sleep(5)

# Print some output to signal code has reached this point
print('Cell is finished running!')<jupyter_output>Cell is finished running!<jupyter_text>In the first line of the above Code cell, we imported a function `sleep()` from a module. A function takes some number of inputs (possibly zero, like `sleep()`), and returns some number of outputs (possibly zero, like `sleep()`). The inputs to a function are called *arguments*, and are put inside the parentheses, if needed.Python modules, even when installed on your machine, must be explicitly imported before they can be used. Some modules, like `time`, are bundled with Python (we might call these "standard modules"), while others must be installed separately from an external source ("third-party modules") before importing.In later parts of the workshop, we will use some highly-specialized modules to carry out materials informatics tasks. If we don't import a *specific* object from a module, then we need to explicitly reference the module in order to use its contents. For instance, we could write the above as the following.<jupyter_code>import time
time.sleep(5)
print("Cell is finished running!")<jupyter_output>Cell is finished running!<jupyter_text>We'll see this a lot later when we start working with the `numpy` and `pandas` modules, where we tend to use *canonical* aliases:```import numpy as npimport pandas as pdX = np.array([[1, 2, 3]])df = pd.DataFrame(    data = X,    columns = ["X"])```  __q6__ Do the mathImport the functions `sin, cos` and the constant `pi` from the `math` module, and use them to complete the following code.<jupyter_code>###
# TASK: Compute some trigonometric operations
# TODO: Import sin, cos, pi from the math module
# TODO: Convert degrees to radians
# TODO: Use sin() and cos() to complete the code below
###

from math import sin, cos, pi
angle_degrees = 15
# task-begin
# -- WRITE YOUR CODE BELOW -----
my_sine = 0  # sin(angle_radians)
my_cosine = 0  # cos(angle_radians)
result = my_sine**2 + my_cosine**2
# task-end
# solution-begin
angle_radians = angle_degrees / 180 * pi
my_sine = sin(angle_radians)
my_cosine = cos(angle_radians)

result = my_sine**2 + my_cosine**2
# solution-end
# -- PRINT THE ANSWER ----
result<jupyter_output><empty_output><jupyter_text>Conditionals and logicConditionals and logic allow us to adapt a program to the data.  Python conditional statements are `if`, `elif`, and `else`, and they follow a particular syntax:<jupyter_code>if True:
    print("True")
else:
    print("Not printed....")<jupyter_output>True<jupyter_text>Note that we **must** either indent or provide four spaces to all lines falling under a conditional.```if True:pass  This will throw an error!if True:    print("This will work!")``` The reason for this is because python uses *whitespace* in the same way C/C++ uses braces, or MATLAB uses `end` statements. Proponents of python argue that the structure of the language *enforces* readability.- An `if` statement will execute only if its argument evaluates to `True`; we can use this to select particular actions.- An `else` statement must follow an `if` or `elif` statement; it will evaluate if none of the other conditionals are triggered.- An `elif` statement is a python-specific construct, and must follow an `if` or another `elif` statement. An `elif` is like an `if`, but only evaluates if the preceding conditionals are not met.Comparisons for equality are similar to other programming languages (`=`, `>`).Logic can be done using *keywords* `not`, `and`, and `or`.<jupyter_code># Uncomment and pick a number!
# n = ???
# solution-begin
n = 0
# solution-end

if n > 0:
    print('Positive.')
elif not n >= 0:
    print('Negative.')
else:
    print('Zero.')<jupyter_output>Zero.<jupyter_text>FunctionsFunctions are defined in Python using the `def` keyword. We specify the arguments by putting variable names in the parentheses `()`. We can specify a value to provide as output by using the `return` keyword, as demonstrated below:```pythondef fcn_example(x, y=1):    return x + y```Like with conditional statements, we must end a function definition line with a colon `:`, and start the following lines indented:```python  This will throw an error: Missing colondef fcn_wrong1(x)    return x + 3  This will throw an error: Missing indentationdef fcn_wrong2(x):return x + 4```We can then "call" (or "execute") that function by typing the function name and including the function inputs in parentheses.```pythonfcn_example(1)fcn_example(1, y=2)```The inputs `x` and `y` are called *arguments*. Arguments without default values must be provided (such as `x`), while arguments with default values are called *keyword arguments* and do not need to be specified when calling the function (such as `y`). Providing a keyword argument (`fcn_example(1, y=2)`) allows us to override the default value.  __q7__ Define a functionDefine a function named `fcn_q7` that takes a single input, adds `2`, and returns the result.<jupyter_code>## TASK: Define a function `fcn_q7` that adds `2` to the input

# task-begin
## Define your function here
# task-end
# solution-begin
def fcn_q7(x):
    return x + 2
# solution-end

# No need to edit this; use this to check your work
assert \
    fcn_q7(2) == 4, \
    "Incorrect value"<jupyter_output><empty_output><jupyter_text>Getting helpIf there is a python function that is mysterious to you, you can always consult the *documentation*. You can call the python built-in `help()` function on an object to display its documentation. The following documentation for `list` points to a number of functions that operate on lists that we did not cover; for instance, the `list.sort()` method is *extremely useful* for data science.<jupyter_code>help(list.sort)<jupyter_output>Help on method_descriptor:

sort(self, /, *, key=None, reverse=False)
    Sort the list in ascending order and return None.
    
    The sort is in-place (i.e. the list itself is modified) and stable (i.e. the
    order of two equal elements is maintained).
    
    If a key function is given, apply it once to each list item and sort them,
    ascending or descending, according to their function values.
    
    The reverse flag can be set to sort in descending order.<jupyter_text>There is a caveat here, which is that `help()` will generally provide *very technical details* on the object in question, which is useful for reference, but often unhelpful when first learning. If you find a particular python object *very mysterious*, try to formulate your question and type it into Google. **Learning how to find useful information about programming concepts is one of the key skills of learning to program well.**```{admonition} Jupyter help shortcutNote that in Jupyter, there's another way to access documentation on a function. If you move your cursor over a function, click there, and press `Shift + Tab`, this will open a documentation panel for easy reference!```  __q8__ Consult the documentationConsult the documentation for `list.sort()` to reverse-sort the following list.*Hint*: You will need to set a certain keyword argument to complete this task.<jupyter_code>## TASK: Reverse-sort the following list
l_rev = [5, 3, 1, 4, 2]

l_rev.sort()
# solution-begin
l_rev.sort(reverse=True)
# solution-end

l_rev<jupyter_output><empty_output><jupyter_text>End NotesThe following notes are not required, but you may find these additional Jupyter tips helpsul.  Save, close, quit, and re-open this notebookYou can save by clicking the floppy-disk icon in the top-left of this notebook pane, and can close the notebook by clicking the "X" of its tab. Re-open by double-clicking on `e-data00-notebooks-assignment.ipynb` in the left-hand navigation pane.You should find that everything is exactly as you had left it! You will have to re-run the cells to load variables, but having the output saved can be helpful, particularly since Jupyter saves figures.  Additional notebook tips* Save your notebooks frequently! Use the "**Save**" icon at the top or the **Ctrl+S**/**Cmd+S** shortcut.* Use **Shift+Enter** to run a cell and move on to the next cell. This allows you to sequentially execute all the cells in your notebook.* **Restart & Clear Output**. This option is found under the *Kernel* menu. It is extremely helpful to reset all code cell executions, stale package imports, and variable values.* Take the time to explore other menu options and tinker around. There's a ton of [keyboard shortcuts](https://www.cheatography.com/weidadeyue/cheat-sheets/jupyter-notebook/pdf_bw/) and [nifty customizations](https://towardsdatascience.com/how-to-effortlessly-optimize-jupyter-notebooks-e864162a06ee) you can do with notebooks.  Jupyter magic functions[Magic functions](https://ipython.readthedocs.io/en/stable/interactive/magics.html) are a set of commands exclusively for Jupyter notebooks that are part of what makes notebooks so powerful. All magic commands start with a `%` sign, and we give examples of some below.<jupyter_code>import numpy as np

# This command will output how long it takes the following inline command to run
%time np.mean(np.random.randint(low=1, high=7, size=1000))

# You'll see this command later; it allows Jupyter to render plots
# as output in the notebook instead of opening a new window.
%matplotlib inline

# This shows which variables are in your environment.
# Can be packages, functions, variables, and more!
%who

# You'll see this command later; it allows Jupyter to automatically
# pick up on changes in referenced code without reloading the kernel.
%load_ext autoreload

# This command will list your system environment variables.
%env<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08715938783128033, 'fraction_numerical': 0.008305337812616649, 'mean_word_length': 4.821021184139055, 'pattern_counts': {'":': 0, '<': 76, '<?xml version=': 0, '>': 79, 'https://': 10, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '248356', 'license': 'MIT', 'n_tokens_mistral': 6058, 'n_tokens_neox': 5778, 'n_words': 2979, 'path': 'exercises/e-data00-notebooks-master.ipynb', 'repo_name': 'cstakeda/evc-course', 'size': '21455'}