<jupyter_start><jupyter_text>What to expect from the Python lessons- Get you started with python through a little project- Showcase relevant use cases of python for exploratory data analysis - Provide you with exercises during and after the lessons so that you practice and experience python- Provide you with good reads to develop further the wonderful craft of programming  Coding project  introductionWe will work with hypothetical clynical trial data about a drug that aims to reduce arthritis inflammation.The data analysis goal is to assess the effectiveness of the treament and inform the clynical trial with data visualization taking advantage of interactive notebooks.More details on this later ;)  Software Carpentry - Python part 112:45 - Introduction to Jupyter notebooks  13:00 - Python fundamentals  13:45 - Exercises in Breakout Room  **14:00 - Coffee break**  14:15 - Loading data  14:45 - Visualizing data  **15:30 - Coffee break**  15:45 - Repeating actions  16:30 - Exercises in Breakout Room  16:50 - Wrap up    Quick tour to Anaconda (Individual edition)"**Your data science toolkit**: With over 25 million users worldwide, the open-source Individual Edition (Distribution) is the easiest way to perform Python/R data science and machine learning on a single machine. Developed for solo practitioners, it is the toolkit that equips you to work with thousands of open-source packages and libraries."https://www.anaconda.com/products/individual  Anaconda installation comes with:* Latest version of Python* JupyterLab, Jupyter notebooks and several other IDE options* An easy-to-install collection of high performance Python libraries* Conda, tool for managing packages and environments* Collection of open source packages and tools to install over 1.5k additional packages* Anaconda Navigator is a graphical user interface to the conda package and environment manager and a free scientific-computing environment.<jupyter_code>display.Image("../../img/conda.png")<jupyter_output><empty_output><jupyter_text>Let's open up a jupyter notebook! Method 1: Anaconda Navigator<jupyter_code>display.Image("../../img/anaconda-navigator-notebook-launch.png")<jupyter_output><empty_output><jupyter_text>Method 2: Terminal  Why are we using notebooks instead of an IDE like Spyder- Storytelling with documented and re-executable analysis- Great for exploring libraries, commands and code snippets- Persistence of code snippets you want to reuse and try compared to a terminal or ipython.<jupyter_code># This is a code snippet
def say_hello(your_name):    # define say_hello function
    print('Hello ' + your_name)

say_hello('Jose') # Here we call the function<jupyter_output>Hello Jose<jupyter_text>Dont confuse the interactive notebook documents  with the "jupyter notebook" application- The first one is a document format that you can find with an `.ipynb` file extension- The second is an application where you can use such documents- Interactive notebooks or `ipynb`s are very related to `.ipython`an interactive shell. Open notebook example: [Gravitational Wave Open Science Center](https://mybinder.org/v2/gh/losc-tutorial/quickview/master?filepath=index.ipynb)  1. Python fundamentals  Lets get started with python and see what it can do - Open your jupyter notebook application - Go to `day_1/` directory and open the `exercises.ipynb` notebook.  How to use the `exercises.ipynb`- Lets write our first code snippet in a cell- Lets execute code (pressing run or pressing `Shift + Enter`)- Lets write some markdown in another cell<jupyter_code># Write your first code snippet
# Write your first markdown cell<jupyter_output><empty_output><jupyter_text>Things you should know about python for Today* Easy to code, It is a very easy to learn language.* It has a lot of reusable code that allows you to do powerful things (libraries)* High-Level, when we write python code you dont have to remember systems architecture* Open source and highly extendable!* Indentation and spaces make a difference<jupyter_code># In this snippet we demonstrate how indentation works in python
names = ['Curie', 'Darwin', 'Turing']


for name in names:
print(name)

# ..... To be explained with live coding
# if name == 'Curie':
# print('Curie is a lady')
# else:
# print(name + ' is not a lady')<jupyter_output><empty_output><jupyter_text>Variables and data types in pythonCreate variables to describe patient characteristics:- Patient identifier- Patient name- Age- Weight- Specificatin of inflammations status (is inflammated, is not inflammated)<jupyter_code># Demonstrate multiple assignment with two patient names
patient_1_name, patient_2_name = "<NAME>", "Wendy"
print(patient_1_name, patient_2_name)

# Demonstrate how to use print to interact with data in memory

# Demonstrate string formatting to have better interaction with the terminal
# Demonstrate variables valid and invalid names 
# Examples 1_patient vs patient_1
## patient name = "<NAME>"   # This will not work
## 1_patient_name = "<NAME>" # This will not work
patient_1_name = "<NAME>"

# Trying data types

# Try integer type with patient_1 age
patient_1_age = 40

# Try floating points with patient's weight
patient_1_weight = 90.5

# a patient_id string
patient_1_id = '001' # or patient_1_id = 1 

# Boolean, does patient "x" has had inflammation?
patient_1_is_inflammated = True

# Get information about data types using a built in function
print(type(patient_1_name))
print(type(patient_1_age))
print(type(patient_1_weight))


# Use other builtin functions
print(len(patient_1_name))<jupyter_output><empty_output><jupyter_text>Combining everything we have learned into a patients dataset- We will create a list of patients- And start populating and manipulating that list with python<jupyter_code># Working with lists data types
# First lets make an empty list
patients = [] 
# Lets populate the list with our prvious values assigned to the patients variables
patients.append(patient_2_name) # Here we append patient 2 to the list
patients.insert(0, patient_1_name) # here we demonstrate how we insert an item in a particular position of the index

# Grouping more data points per patient, we only have names
# Place 0 in the list index is id, place 1, is name, place 3 is age, place 4 is weight, place 5 is_inflammated
patient_1 = ['001', patient_1_name, 40, True ] # This show cases how a list can hold different data types

# A patients list with different data points per patient
# Demonstrate a two dimensional list: [[], [], []]
patients = [[0, patient_1_name, 40, 100, True],
            [1, patient_2_name, 60, 50, False],
            [2, 'Alice', 40, True]]
patients
# Organizing the same data using dictionaries
patient_1 = {
    "id": '001',
    "name": "<NAME>",
    "Age" : 40,
    "Inflammation": True
}

# Lets replace our first item in the patients previous list with our new data type
patients[0] = patient_1
patients
# Introduce slicing here<jupyter_output><empty_output><jupyter_text>What have we done so far?- We know how to assign different data types to different values- We know how to compose more complex data types by aggregating the basic data types or primitive data types- We can use the notebook now to play around and try code snippets- We know how to use `print()` python builtin function to interact display results as outputs  Lets put everything we have learned together into a little app to enter patients data into a list- We will introduce 4 patients to the patients list- Try to stick to the same names I am using...<jupyter_code># We can use input like this
# patient_name = input()
# Lets clear our list again and start using the input() to populate the list
patients = []
# Create a patient id
patient_id = len(patients) 
patients.append([patient_id])
    
# Enter patients name and add the entry to patients list
print('Enter patients name:')
name = input()
patients[patient_id].append(name)

# Enter patients age and add the entry to patients list
print('Enter patients age:')
age = input()
patients[patient_id].append(age)

# Enter patients weight and add the entry to patients list
print('Enter patients weight:')
weight = input()
patients[patient_id].append(weight)

# Specify if patient was inflammated
print ('Enter if the patient was inflammated:')
is_inflammated = input()
patients[patient_id].append(is_inflammated)
patients
# We can recreate our list in case it disappears from memory
patients = [[0, '<NAME>', '40', '80', 'True'],
           [1, '<NAME>', '30', '60', 'False'],
           [2, '<NAME>', '37', '70', 'False'],
           [3, '<NAME>', '60', '80', 'True']]
# Save data to a file with python to csv format
# There are more efficient ways to do this but this is better for demonstration purposes
import csv

# open the file in the write mode
f = open('./my_patients.csv', 'w', encoding='UTF8')

# create the csv writer
writer = csv.writer(f)

# write a row to the csv file
writer.writerows(patients)

# close the file
f.close()

# Now we have a persisted dataset in our directory
# Loading data with pandas 
import pandas as pd
df=pd.read_csv('./my_patients.csv', sep=',',header=None)
df<jupyter_output><empty_output><jupyter_text>Checking data types to store data according to our original specifications<jupyter_code># Lets check the types and see if the the data complies with requirements we have defined upfront
type(patients[0][4])
## type(patients[1][2])

# Another way of checking
type(patients[0][0]) == int

# Do it with a for loop to inspect the data
for item in patients[0]:
    print(item, ' is of type', type(item))<jupyter_output>0  is of type <class 'int'>
Peter Pan  is of type <class 'str'>
40  is of type <class 'str'>
80  is of type <class 'str'>
True  is of type <class 'str'><jupyter_text>Here we have a problem of types that dont match what we defined at the beginning- Age should be an integer- Weight should be a float- And is_inflammated should be a boolean typeHold this issue, we will come back to it later ;)  Breakout Session 1  Exercise 1 - Slicing stringsA section of an array is called a _slice_. We can take slices of character strings as well:<jupyter_code>element = "oxygen"
print('first three characters:', element[0:3])
print('last three characters:', element[3:6])<jupyter_output>first three characters: oxy
last three characters: gen<jupyter_text>What is the value of `element[4]`? What about `element[4:]`? Or `element[:]`?What is `element[-1]`? What is `element[-3:]`?<jupyter_code># Your solution<jupyter_output><empty_output><jupyter_text>Excercise 2 - Slicing lists with stepsSo far we’ve seen how to use slicing to take single blocks of successive entries from a sequence. But what if we want to take a subset of entries that aren’t next to each other in the sequence? You can achieve this by providing a third argument to the range within the brackets, called the step size. The full sytax for creating slices is `[begin:end:step]`, although you most often find a short-hand notation as we've seen in Exercise 1. The example below shows how you can take every third entry in a list:<jupyter_code>primes = [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37]
subset = primes[0:12:3]
print('subset', subset)<jupyter_output>subset [2, 7, 17, 29]<jupyter_text>Given the following list of months:<jupyter_code>months = ['jan', 'feb', 'mar', 'apr', 'may', 'jun', 'jul', 'aug', 'sep', 'oct', 'nov', 'dec']<jupyter_output><empty_output><jupyter_text>Questions1. What slice of `months` will produce the following output `['jan', 'mar', 'may', 'jul', 'sep', nov']`?1. Given the short-hand notation we used for the character string in Exercise 1 (i.e. `element[:2] == 'element[0:2]`), can you find the short-hand notation for question 1? What do you find easier to read?1. Using the step size parameter, can you think of a way to reverse the list?<jupyter_code># Your solution<jupyter_output><empty_output><jupyter_text>Back to our code project: analysing patient data Arthritis InflammationWe are studying inflammation in patients who have been given a new treatment for arthritis.There are 60 patients, who had their inflammation levels recorded for 40 days. We want to analyze these recordings to study the effect of the new arthritis treatment.To see how the treatment is affecting the patients in general, we would like to:1. Calculate the average inflammation per day across all patients.1. Plot the result to discuss and share with colleagues.<jupyter_code>display.Image("../../img/lesson-overview.png")<jupyter_output><empty_output><jupyter_text>What if we need the maximum inflammation for each patient over all days (as in the next diagram on the left) or the average for each day (as in the diagram on the right)?<jupyter_code>display.Image("../../img/python-operations-across-axes.png")<jupyter_output><empty_output><jupyter_text>2. Analysing patient dataLets have a look at the dataset first<jupyter_code># NumPy is a Python library used for operations with matrices and arrays
# Numpy comes pre-installed with anaconda
import numpy
data = numpy.loadtxt(fname='../../day_1/data/inflammation-01.csv', delimiter=',')
print(data) # Shows small part of the dataset

# test type of value holded by data variable
print(type(data))
print(data.dtype) # Highlight OOP
print(data.shape)
# Indexing an array
print('first element in data:', data[0, 0])

# Slicing data
print(data[0:4, 0:10])

subset = data[:3, -4:] # we can use -4 to call the index from the end
print(subset)<jupyter_output>first element in data: 0.0
[[0. 0. 1. 3. 1. 2. 4. 7. 8. 3.]
 [0. 1. 2. 1. 2. 1. 3. 2. 2. 6.]
 [0. 1. 1. 3. 3. 2. 6. 2. 5. 9.]
 [0. 0. 2. 0. 4. 2. 2. 1. 6. 7.]]
[[2. 3. 0. 0.]
 [1. 1. 0. 1.]
 [2. 2. 1. 1.]]<jupyter_text>Talk about libraries and OOP a bit- For Today the most important to know about object oriented programing, is that is a way of structuring code in a certain way.- Python is OOP and it is particularly noticeable when you are reusing libraries like numpy.<jupyter_code># Using methods fromt he class numpy
print(numpy.mean(data))
# Useful statistics
maxval, minval, stdval, meanval = numpy.max(data), numpy.min(data), numpy.std(data), numpy.mean(data)

print('maximum inflammation:', maxval)
print('mininum inflammation:', minval)
print('standard deviation:', stdval)
print('mean:', meanval)<jupyter_output>maximum inflammation: 20.0
mininum inflammation: 0.0
standard deviation: 4.613833197118566
mean: 6.14875<jupyter_text>When analysing data, though, we often want to look at variations in statistical values, such as the maximum inflammation per patient or the average inflammation per day.<jupyter_code># Mean per day over all patients
print(numpy.mean(data, axis=0))

print(numpy.mean(data, axis=0).shape)
# The average inflammation per patient across all days
print(numpy.mean(data, axis=1))

# As a quick check, we can look at the shape
print(numpy.mean(data, axis=1).shape)
import matplotlib.pyplot as plt

# plt.plot(data,"*");
plt.subplots(3,3,figsize=(10,10))
plt.subplots_adjust(wspace=0.5,hspace=0.5)

# Use subplots to show inflamation per patient across days
# Histogram
for i in numpy.arange(1,10):
    plt.subplot(3,3,i)
    plt.hist(data[i,:],label="patient_"+str(i));
    plt.xlabel("Inflamation values")
    plt.ylabel(" Frequency")
    plt.legend()<jupyter_output><empty_output><jupyter_text>Using pandas to show how libraries build on top of each other to add valuePandas is built on top of NumPy, which means the Python pandas package depends on the NumPy package and also pandas intended with many other 3rd party libraries.<jupyter_code># Use pandas to show how to handle array data 
# Explain that pandas are based on numpy arrays
import pandas as pd

df = pd.DataFrame(data = data)
df.head() # A method to show only one part of the dataframe<jupyter_output><empty_output><jupyter_text>Key points* Import a library into a program using import libraryname.* Use the `numpy` library to work with arrays in Python.* Use array `[x, y]` to select a single element from a 2D array.* Use `numpy.mean(array)`, `numpy.max(array)`, and `numpy.min(array)` to calculate simple statistics.* Use `numpy.mean(array, axis=0)` or `numpy.mean(array, axis=1)` to calculate statistics across the specified axis.\ Questions?  3. Visualizing Tabular Data<jupyter_code>import matplotlib.pyplot

# Plot entire dataset as a heatmap

plt.xlabel("Days" )
plt.ylabel("Patients")

image=matplotlib.pyplot.imshow(data,aspect='auto' )
plt.colorbar(image)

# Let's make our lives easier with an alias
import matplotlib.pyplot as plt
# Plot the mean inflammation per day over all patients
mean_inflammation = numpy.mean(data, axis=0)
mean_plot = plt.plot(mean_inflammation)
plt.show(mean_plot)<jupyter_output><empty_output><jupyter_text>This data looks suspicious! I would not expect to find a sharp peak in an average of the dataset. Very unlikely that the inflammation of all patients spikes on day 18. Let's look at two other statistics: max and min<jupyter_code>max_plot = plt.plot(numpy.max(data, axis=0))
plt.show(max_plot)
min_plot = plt.plot(numpy.min(data, axis=0))
plt.show(min_plot)
# Setup subplots
import numpy
import matplotlib.pyplot as plt

# Import data
data = numpy.loadtxt(fname='../../day_1/data/inflammation-01.csv', delimiter=',')

# Setup figure
fig = plt.figure(figsize=(10.0, 3.0))

# Add subplots
axes1 = fig.add_subplot(1, 3, 1) # (nrows, ncols, pos)
axes2 = fig.add_subplot(1, 3, 2)
axes3 = fig.add_subplot(1, 3, 3)

# Add labels
axes1.set_ylabel('mean')
axes2.set_ylabel('max')
axes3.set_ylabel('min')

axes1.plot(numpy.mean(data, axis=0))
axes2.plot(numpy.max(data, axis=0))
axes3.plot(numpy.min(data, axis=0))

plt.savefig('inflammation.png')
plt.show()<jupyter_output><empty_output><jupyter_text>Try to do this on your own later after the workshop:Create `inflammation_analysis.ipynb` using markdown cells.Let's organize our analysis in a new notebook.Use the bullet points bellow to create a structure for your notebook.**Steps:**1. Set up subplots2. Load data3. Plot data4. Add labels5. Save the figure  Key points* Use the `pyplot` module from the `matplotlib` library for creating simple visualizations.* Use an alias when importing lengthy library names, e.g. `import matplotlib.pyplot as plt`* Create subplots and add labels Questions?  4. Repeating Actions with Loops<jupyter_code>odds = [1, 3, 5, 7]
print(odds[0])
print(odds[1])
print(odds[2])
print(odds[3])
# Use a for loop
odds = [1, 3, 5, 7, 9]
for num in odds:
    print(num)<jupyter_output><empty_output><jupyter_text>Bad approach1. **Not scalable**. Four entries might be doable, hundreds will not!1. **Difficult to maintain**. If we want to change the print statement, we would need to change it four times. DRY -> Don't Repeat Yourself1. **Fragile**. If odds changes size, we would need to update the print statement.<jupyter_code># The name of the variable is not critical, aim for meaning and readability
for banana in odds:
    print(banana)<jupyter_output><empty_output><jupyter_text>Looping over multiple data filesAs a final piece to processing our inflammation data, we need a way to get a list of all the files in our data directory whose names start with `inflammation-` and end with `.csv`. The following library will help us to achieve this:<jupyter_code>import glob
glob.glob('../data/inflammation*.csv')
import glob
import numpy
import matplotlib.pyplot as plt

# Find filenames
filenames = sorted(glob.glob('../../day_1/data/inflammation*.csv'))
# Reduce number of plots
filenames = filenames[0:3]

for filename in filenames:

    # Import data
    data = numpy.loadtxt(fname=filename, delimiter=',')

    # Setup figure
    fig = plt.figure(figsize=(10.0, 3.0))

    # Add subplots
    axes1 = fig.add_subplot(1, 3, 1) # (nrows, ncols, pos)
    axes2 = fig.add_subplot(1, 3, 2)
    axes3 = fig.add_subplot(1, 3, 3)

    # Add labels
    axes1.set_ylabel('mean')
    axes2.set_ylabel('max')
    axes3.set_ylabel('min')

    axes1.plot(numpy.mean(data, axis=0))
    axes2.plot(numpy.max(data, axis=0))
    axes3.plot(numpy.min(data, axis=0))

    plt.savefig('inflammation.png')
    plt.show()<jupyter_output><empty_output><jupyter_text>Key points* Use `for variable in sequence` to process the elements of a sequence one at a time.* The body of a `for` loop must be indented.* Use `glob.glob(pattern)` to create a list of files whose names match a pattern.* Use `*` in a pattern to match zero or more characters, and `?` to match any single character. Questions?  Breakout Session 2  Exercise 3 - Change in inflammationThe patient data is longitudinal in the sense that each row represents a series of observations relating to one individual. This means that the change in inflammation over time is a meaningful concept.The `numpy.diff()` function takes an array and returns the differences between two successive values. Let’s use it to examine the changes each day across the first week of patient 3 from our inflammation dataset:<jupyter_code># Load data
import numpy
data = numpy.loadtxt(fname='../data/inflammation-01.csv', delimiter=',')
patient3_week1 = data[3, :7]
print(patient3_week1)
patient3_week1 = data[3, :7]
print(patient3_week1)<jupyter_output><empty_output><jupyter_text>Calling `numpy.diff(patient3_week1)` would do the following calculations<jupyter_code>numpy.diff(patient3_week1)<jupyter_output><empty_output><jupyter_text>Questions1. When calling `numpy.diff()` with a multi-dimensional array, an axis argument may be passed to the function to specify which axis to process. When applying `numpy.diff()` to our 2D inflammation array data, which axis would we specify?<jupyter_code># Solution
numpy.diff(data, axis=1)<jupyter_output><empty_output><jupyter_text>Exercise 4 - Plotting differencesPlot the difference between the average inflammations reported in the first and second datasets (stored in `inflammation-01.csv` and `inflammation-02.csv`, correspondingly), i.e., the difference between the leftmost plots of the first two figures we have plotted so far. Steps:1. Import libraries1. Import data1. Calculate difference1. Create and annotate figure<jupyter_code># Solution
import glob
import numpy
import matplotlib.pyplot

filenames = sorted(glob.glob('../data/inflammation*.csv'))

data0 = numpy.loadtxt(fname=filenames[0], delimiter=',')
data1 = numpy.loadtxt(fname=filenames[1], delimiter=',')

fig = matplotlib.pyplot.figure(figsize=(10.0, 3.0))

matplotlib.pyplot.ylabel('Difference in average')
matplotlib.pyplot.plot(numpy.mean(data0, axis=0) - numpy.mean(data1, axis=0))

fig.tight_layout()
matplotlib.pyplot.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0951262738147984, 'fraction_numerical': 0.020248116969428444, 'mean_word_length': 5.03987155472304, 'pattern_counts': {'":': 3, '<': 125, '<?xml version=': 0, '>': 126, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '29', 'dir': 'jupyter-structured-clean-dedup', 'id': '411141', 'license': 'CC0-1.0', 'n_tokens_mistral': 7212, 'n_tokens_neox': 6716, 'n_words': 2794, 'path': 'Notes_for_lesson/review/day_1_review_lecture.ipynb', 'repo_name': 'jurra/software_carpentries', 'size': '22605'}