<jupyter_start><jupyter_text>[(previous)](03%20-%20-%20-%20Python%20-%20intermediate.ipynb) | [(index)](00%20-%20-%20-%20Introduction%20-%20to%20-%20Python.ipynb) | [(next)](05%20-%20-%20-%20Introduction%20-%20to%20-%20data%20-%20as%20-%20a%20-%20science.ipynb) Python tools for data analysis    Learning outcomes:                Identify concepts in ethical reasoning which may influence our analysis and results from data.        Learn and apply a basic set of methods from the core data analysis libraries of Numpy, Pandas and Matplotlib.    Data has become the most important language of our era, informing everything from intelligence in automated machines, to predictive analytics in medical diagnostics. The plunging cost and easy accessibility of the raw requirements for such systems - data, software, distributed computing, and sensors - are driving the adoption and growth of data-driven decision-making. As it becomes ever-easier to collect data about individuals and systems, a diverse range of professionals - who have never been trained for such requirements - grapple with inadequate analytic and data management skills, as well as the ethical risks arising from the possession and consequences of such data and tools.Before we go on with the technical training, consider the following on the ethics of the data we use. EthicsComputers cannot make decisions. Their output is an absolute function of the data provided as input, and the algorithms applied to analyse that input. The aid of computers in decision-making does not override human responsibility and accountability.It should be expected that both data and algorithms should stand up to scrutiny so as to justify any and all decisions made as a result of their output. "Computer says no," is not an unquestionable statement.Our actions - as data scientists - are intended to persuade people to act or think other than the way they currently do based on nothing more than the strength of our analysis, and informed by data.The process by which we examine and explain why what we consider to be right or wrong, is considered right or wrong in matters of human conduct, belongs to the study of ethics.Case-study: _Polygal_ was a gel made from beet and apple pectin. Administered to a severely wounded patient, it was supposed to reduce bleeding. To test this hypothesis, [Sigmund Rascher](https://en.wikipedia.org/wiki/Sigmund_Rascher) administered a tablet to human subjects who were then shot or - without anesthesia - had their limbs amputated.During the Second World War, and under the direction of senior Nazi officers, medical experiments of quite unusual violence were conducted on prisoners of war and civilians regarded by the Nazi regime as sub-human. After the war, twenty medical doctors were tried for war crimes and crimes against humanity at the [Doctor's Trial](https://en.wikipedia.org/wiki/Doctors%27_trial) held in Nuremberg from 1946 to 1949.In 1947 <NAME> - Deputy Reich Health Leader, a high-ranking Nazi scientist - was acquitted of war crimes on the strength of intervention by the United States. Within two months, he was being debriefed by the US military who wished to learn everything he knew about biological warfare.Do you feel the US was "right" or "wrong" to offer Blome immunity from prosecution in exchange for what he knew?There were numerous experiments conducted by the Nazis that raise ethical dilemmas, including: immersing prisoners in freezing water to observe the result and test hypothermia revival techniques; high altitude pressure and decompression experiments; sulfanilamide tests for treating gangrene and other bacterial infections.Do you feel it would be "right" or "wrong" to use these data in your research and analysis?    Further reading.Whereas everything else we do can describe human behaviour as it _is_, ethics provides a theoretical framework to describe the world as it _should_, or _should not_ be. It gives us full range to describe an ideal outcome, and to consider all that we know and do not know which may impede or confound our desired result.Case-study: A Nigerian man travels to a conference in the United States. After one of the sessions, he goes to the bathroom to wash his hands. The electronic automated soap dispensor does not recognise his hands beneath the sensor. A white American sees his confusion and places his hands beneath the device. Soap is dispensed. The Nigerian man tries again. It still does not recognise him.How would something like this happen? Is it an ethical concern?Further reading.When we consider ethical outcomes, we use the terms _good_ or _bad_ to describe judgements about people or things, and we use _right_ or _wrong_ to refer to the outcome of specific actions. Understand, though, that - while right and wrong may sometimes be obvious - we are often stuck in ethical dilemmas.How we consider whether an action is right or wrong comes down to the tension between what was intended by an action, and what the consequences of that action were. Are only intensions important? Or should we only consider outcomes? And how absolutely do you want to judge this chain: the _right_ motivation, leading to the _right_ intention, performing the _right_ action, resulting in only _good_ consequences. How do we evaluate this against what it may be impossible to know at the time, even if that information will become available after a decision is made?We also need to consider competing interests in good and bad outcomes. A good outcome for the individual making the decision may be a bad decision for numerous others. Conversely, an altruistic person may act only for the benefit of others even to their own detriment.Ethical problems do not always require a call to facts to justify a particular decision, but they do have a number of characteristics: - _Public_: the process by which we arrive at an ethical choice is known to all participants; - _Informal_: the process cannot always be codified into law like a legal system; - _Rational_: despite the informality, the logic used must be accessible and defensible; - _Impartial_: any decision must not favour any group or person;Rather than imposing a specific set of rules to be obeyed, ethics provides a framework in which we may consider whether what we are setting out to achieve conforms to our values, and whether the process by which we arrive at our decision can be validated and inspected by others.No matter how sophisticated our automated machines become, unless our intention is to construct a society "of machines, for machines", people will always be needed to decide on what ethical considerations must be taken into account.There are limits to what analysis can achieve, and it is up to the individuals producing that analysis to ensure that any assumptions, doubts, and requirements are documented along with their results. Critically, it is also each individual's personal responsibility to raise any concerns with the source data used in the analysis, including whether personal data are being used legitimately, or whether the source data are at all trustworthy, as well as the algorithms used to process those data and produce a result. Data analysisThis will be a very brief introduction to some tools used in data analysis in Python. This will not provide insight into the approaches to performing analysis, which is left to self-study, or to modules elsewhere in this series. Numpy arraysData analysis often involves performing operations on large lists of data. Numpy is a powerful suite of tools permitting you to work quickly and easily with complete data lists. We refer to these lists as arrays, and - if you are familiar with the term from mathematics - you can think of these as matrix methods.By convention, we import Numpy as np; `import numpy as np`.We're also going to want to be generating a lot of lists of random floats for these exercises, and that's tedious to write. Let's get Python to do this for us using the `random` module.<jupyter_code>import numpy as np
import random

def generate_float_list(lwr, upr, num):
    """
    Return a list of num random decimal floats ranged between lwr and upr.
    
    Range(lwr, upr) creates a list of every integer between lwr and upr.
    random.sample takes num integers from the range list, chosen randomly.
    """
    int_list = random.sample(range(lwr, upr), num)
    return [x/100 for x in int_list]

# Create two lists
height = generate_float_list(100, 220, 10)
weight = generate_float_list(5000, 20000, 10)

# Convert these to Numpy arrays
np_height = np.array(height)
np_weight = np.array(weight)

print(np_height)
print(np_weight)<jupyter_output>[1.54 1.75 1.43 2.03 1.51 1.59 1.19 1.72 1.13 2.09]
[ 70.08 166.31 170.51 174.34  89.29  69.13 137.76  96.66 123.97  95.73]<jupyter_text>There is a useful timer function built in to Jupyter Notebook. Start any line of code with `%time` and you'll get output on how long the code took to run.This is important when working with data-intensive operations where you want to squeeze out every drop of efficiency by optimising your code.We can now perform operations directly on all the values in these Numpy arrays. Here are two simple methods to use.    Syntax                Element-wise calculations: you can treat Numpy arrays as you would individual floats or integers. Note, they must either have the same shape (i.e. number or elements), or you can perform bitwise operations (operate on each item in the array) with a single float or int        Filtering: You can quickly filter Numpy arrays by performing boolean operations, e.g. `np_array[np_array > num]`, or, for a purely boolean response, `np_array > num`<jupyter_code># Calculate body-mass index based on the heights and weights in our arrays
# Time the calculation ... it won't be long
%time bmi = np_weight / np_height ** 2

print(bmi)

# Any BMI > 35 is considered severely obese. Let's see who in our sample is at risk.

# Get a boolean response
print(bmi > 35)

# Or print only BMI values above 35
print(bmi[bmi > 35])<jupyter_output>Wall time: 0 ns
[29.54967111 54.30530612 83.38305052 42.30629231 39.16056313 27.34464618
 97.28126545 32.67306652 97.08669434 21.91570706]
[False  True  True  True  True False  True False  True False]
[54.30530612 83.38305052 42.30629231 39.16056313 97.28126545 97.08669434]<jupyter_text>PandasWe briefly experimented with Pandas back in [Built-in modules](03 - Python intermediate.ipynbBuilt-in-modules).The description given for Pandas there was:_**pandas** is a Python package providing fast, flexible, and expressive data structures designed to make working with "relational" or "labeled" data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, **real world** data analysis in Python. Additionally, it has the broader goal of becoming **the most powerful and flexible open source data analysis / manipulation tool available in any language**._Pandas is developed by [Wes McKinney](http://wesmckinney.com/) and has a marvelous and active development community. Wes prefers pandas to be written in the lower-case (I'll alternate).Underneath Pandas is Numpy, so they are closely related and tightly integrated. Pandas allows you to manipulate data either as a `Series` (similarly to Numpy, but with added features), or in a tabular form with rows of values and named columns (similar to the way you may think of an Excel spreadsheet).This tabular form is known as a `DataFrame`. Pandas works well with Jupyter Notebook and you can output nicely formated dataframes (just make sure the last line of your code block is the name of the dataframe).The convention is to import pandas as pd, `import pandas as pd`.The following tutorial is taken direclty from the '10 minutes to pandas' section of the [Pandas documentation](https://pandas.pydata.org/pandas-docs/stable/10min.html). Note, this isn't the complete tutorial, and you can continue there. Object creationCreate a `Series` by passing a list of values, and letting pandas create a default integer index.<jupyter_code>import pandas as pd
import numpy as np

s = pd.Series([1,3,5,np.nan,6,8])
s<jupyter_output><empty_output><jupyter_text>Note that `np.nan` is Numpy's default way of presenting a value as "not-a-number". For instance, divide-by-zero returns `np.nan`. This means you can perform complex operations relatively safely and sort out the damage afterwards.Create a DataFrame by passing a numpy array, with a datetime index and labeled columns.<jupyter_code># Create a date range starting at an ISO-formatted date (YYYYMMDD)
dates = pd.date_range('20130101', periods=6)
dates
# Create a dataframe using the date range we created above as the index
df = pd.DataFrame(np.random.randn(6,4), index=dates, columns=list('ABCD'))
df<jupyter_output><empty_output><jupyter_text>We can also mix text and numeric data with an automatically-generated index.<jupyter_code>dict = {"country": ["Brazil", "Russia", "India", "China", "South Africa"],
       "capital": ["Brasilia", "Moscow", "New Dehli", "Beijing", "Pretoria"],
       "area": [8.516, 17.10, 3.286, 9.597, 1.221],
       "population": [200.4, 143.5, 1252, 1357, 52.98] }

brics = pd.DataFrame(dict)
brics<jupyter_output><empty_output><jupyter_text>The numbers down the left-hand side of the table are called the index. This permits you to reference a specific row. However, Pandas permits you to set your own index, as we did where we set a date range index. You could set one of the existing columns as an index (as long as it consists of unique values) or you could set a new custom index.<jupyter_code># Set the ISO two-letter country codes as the index
brics.index = ["BR", "RU", "IN", "CH", "SA"]

brics<jupyter_output><empty_output><jupyter_text>Viewing dataPandas can work with exceptionally large datasets, including millions of rows. Presenting that takes up space and, if you only want to see what your data looks like (since, most of the time, you can work with it symbolically), then that can be painful. Fortunately, pandas comes with a number of ways of viewing and reviewing your data.    Syntax                See the top and bottom rows of your dataframe with `df.head()` or `df.tail(num)` where `num` is an integer number of rows        See the index, columns and underlying numpy data with `df.index`, `df.columns` and `df.values`        Get a quick statistical summary of your data with `df.describe()`        Transpose your data with `df.T`        Sort by an axis with `df.sort_index(axis=1, ascending=False)` where `axis=1` refers to columns, and `axis=0` refers to rows        Sort by values with `df.sort_values(by=column)`<jupyter_code># Head
df.head()
# Tail
df.tail(3)
# Index
df.index
# Values
df.values
# Statistical summary
df.describe()
# Transpose
df.T
# Sort by an axis
df.sort_index(axis=1, ascending=False)
# Sort by values
df.sort_values(by="B")<jupyter_output><empty_output><jupyter_text>SelectionsOne of the first steps in data analysis is simply to filter your data and get at slices you're most interested in. Pandas has numerous approaches to quickly get only what you want.    Syntax                Select a single column by addressing the dataframe as you would a dictionary, with `df[column]` or, if the column name is a single word, with `df.column`. This returns a series        Select a slice in the way you would a Python list, with `df[]`, e.g. `df[:3]`, or by slicing the indices, `df["20130102":"20130104"]`        Use `.loc` to select by specific labels, such as:                    Get a cross-section based on a label, with e.g. `df.loc[index[0]]`            Get on multi-axis by a label, with `df.loc[:, ["A", "B"]]` where the first `:` indicates the slice of rows, and the second list `["A", "B"]` indicates the list of columns                As you would with Numpy, you can get a boolean-based selection, with e.g. `df[df.A > num]`    There are a _lot_ more ways to filter and access data, as well as methods to set data in your dataframes, but this will be enough for now.<jupyter_code># By column
df.A
# By slice
df["20130102":"20130104"]
# Cross-section
df.loc[dates[0]]
# Multi-axis
df.loc[:, ["A", "B"]]
# Boolean indexing
df[df.A > 0]<jupyter_output><empty_output><jupyter_text>MatplotlibIn this last section, you get to meet _Matplotlib_, a fairly ubiquitous and powerful Python plotting library. Jupyter Notebook has some "magic" we can use in the line `%matplotlib inline` which permits us to draw charts directly in this notebook.Matplotlib, Numpy and Pandas form the three most important and ubiquitous tools in data analysis.Note that this is the merest slither of an introduction to what you can do with these libraries.<jupyter_code>import matplotlib.pyplot as plt
# This bit of magic code will allow your Matplotlib plots to be shown directly in your Jupyter Notebook.
%matplotlib inline

# Produce a random timeseries
ts = pd.Series(np.random.randn(1000), index=pd.date_range('1/1/2000', periods=1000))

# Get the cumulative sum of the random numbers generated to mimic a historic data series
ts = ts.cumsum()

# And magically plot
ts.plot()
# And do the same thing with a dataframe
df = pd.DataFrame(np.random.randn(1000, 4), index=ts.index,
                  columns=['A', 'B', 'C', 'D'])

df = df.cumsum()

# And plot, this time creating a figure and adding a plot and legend to it
plt.figure()
df.plot()
plt.legend(loc='best')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06237113402061856, 'fraction_numerical': 0.02663230240549828, 'mean_word_length': 4.490880503144654, 'pattern_counts': {'":': 6, '<': 36, '<?xml version=': 0, '>': 43, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '377533', 'license': 'MIT', 'n_tokens_mistral': 4914, 'n_tokens_neox': 4430, 'n_words': 2508, 'path': '04 - Python tools for data analysis.ipynb', 'repo_name': 'whythawk/intro-to-python', 'size': '17464'}