<jupyter_start><jupyter_text>Data collection project tutorials: Intro to python, jupyter notebook, pandas and data manipulation This is a Jupyter notebook. We had a small mention to them in the FAIR data workshop, but now we're going to use them as a tool to learn the very fundamentals of python and to grasp the unmeasurable importance of programing when we work with data.   Sweet introduction to chaos We will begin where everybody has begun with programming, with the *"Hello world"* exercise. We simply want python to **print** a piece of text, also known as **string**.<jupyter_code>print("Hello world")<jupyter_output>Hello world<jupyter_text>Notice the qutoation marks ("\_\_\_\_") around the text? That's our way of telling python that this is text. Anything that we put between quotation marks becomes a piece of text and can be printed. Example:<jupyter_code>print("Quarantine sucks")
print("wololo")
print("Is the lab open?")<jupyter_output>Quarantine sucks
wololo
Is the lab open?<jupyter_text>What happens if we don't use them?<jupyter_code>print(wololo)<jupyter_output><empty_output><jupyter_text>This error happens because python believes that wololo is a **variable**. A variable is an object that contains information, which can be for instance a string:<jupyter_code>my_first_variable = "Python is fun"
print(my_first_variable)<jupyter_output>Python is fun<jupyter_text>But python can do much more than printing text :)  Data types and simple manipulations The three most commont types of data in python are: strings, floats and integers.<jupyter_code>string_var = 'bacon'
float_var = 3.14
int_var = 42
potato = '5443'<jupyter_output><empty_output><jupyter_text>And we can ask python which kind of data each variable contains with the command **type()**.<jupyter_code>print(type(string_var))
print(type(float_var))
print(type(int_var))
print(type(potato))
print(float(potato))<jupyter_output>5443.0<jupyter_text>Some operations with these variables are infective, some others are incompatible, and some others are possible after a transformation:<jupyter_code>print(float_var+float_var)
print(float_var + 1.2)
print(string_var + string_var)
print(string_var + ' is delicious')
print(string_var + ' costs ' + str(float_var) + ' euros in my local supermarket')  # Float transformed into string
print(int_var + int_var)
print(int_var + float_var)  # Infective!
print(str(int_var)+str(int_var))
print(int_var + potato)<jupyter_output>6.28
4.34
baconbacon
bacon is delicious
bacon costs 3.14 euros in my local supermarket
84
45.14
4242<jupyter_text>We would need two sessions only to show every fancy way in which you can operate with variables, but for the sake of time, we'll go forward to data structures now.  Data structures  Lists The most basic data structure is a **list**. A list is an array of data elements stored in the same object. Lists are definded with square brakets **\[ \]** in python's syntax. These elements can be of any kind, even variables and other lists:<jupyter_code>my_list = ['ham', 'eggs', 42, float_var, False, [1,2,3,4], 3.14]
print(my_list)<jupyter_output>['ham', 'eggs', 42, 3.14, False, [1, 2, 3, 4], 3.14]<jupyter_text>Lists can be indexed, which means that we can choose which position to get the information from (keep in mind, python counts starting from 0):<jupyter_code>print(my_list[0]) 
print(my_list[1])
print(my_list[-1])
print(my_list[-2])
print(my_list[-2][2])<jupyter_output>ham
eggs
3.14
[1, 2, 3, 4]
3<jupyter_text>Dictionaries A very pythonic data structure is the **dictionary**, which keeps unindexed (unsorted) values, which can be found with their key. Dictionaries are defined with curly brakets **{ }** in python's syntax.<jupyter_code>my_car = {"brand": "Ford",
  "model": "Mustang",
  "year": 1964}
print(my_car)<jupyter_output>{'brand': 'Ford', 'model': 'Mustang', 'year': 1964}<jupyter_text>So if we want to know the model of my car we'll do the following:<jupyter_code>print(my_car['model'])<jupyter_output>Mustang<jupyter_text>But I forgot to mention that I upgraded it in 2010!!! Let me just put that in...<jupyter_code>my_car['upgraded'] = [2010, 2017, 2020]
print(my_car)
my_car['upgraded'][-1]<jupyter_output><empty_output><jupyter_text>Perfect So why all this? Explanation? Because the syntax of the **dataframes** that we are interested in behave almost identically!  Pandas dataframes  Introduction to dataframes<jupyter_code>import pandas as pd
import io

download = False

if download:
    url="https://gist.githubusercontent.com/curran/a08a1080b88344b0c8a7/raw/639388c2cbc2120a14dcf466e85730eb8be498bb/iris.csv"
    s=requests.get(url).content
    df=pd.read_csv(io.StringIO(s.decode('utf-8')))
else:
    df = pd.read_csv('./iris.csv')

df<jupyter_output><empty_output><jupyter_text>Subset only species<jupyter_code>df['species']<jupyter_output><empty_output><jupyter_text>How many unique species do we have?<jupyter_code>unique_species = df['species'].unique()
print(unique_species)<jupyter_output>['setosa' 'versicolor' 'virginica']<jupyter_text>Since we are here, let's do a mini intermezzo to explain how to iterate through an indexed object, like our prevoius output:<jupyter_code>for word in unique_species:  # Takes each element (species) in an indexed object (unique_species)
    print(word + ' are beautiful flowers')  # And does something with this element. In this case tell the whole world how beautiful they are
    for a, letter in enumerate(word):
        print(a, word[a], letter)<jupyter_output>setosa are beautiful flowers
0 s s
1 e e
2 t t
3 o o
4 s s
5 a a
versicolor are beautiful flowers
0 v v
1 e e
2 r r
3 s s
4 i i
5 c c
6 o o
7 l l
8 o o
9 r r
virginica are beautiful flowers
0 v v
1 i i
2 r r
3 g g
4 i i
5 n n
6 i i
7 c c
8 a a<jupyter_text>Basic data selection Let's say that for some reason, I don't like wide sepals, so I want to know which of these my entries have a sepal no bigger than 3 cm<jupyter_code>small_sepal_df = df[df['sepal_width']<= 3.0]
small_sepal_df<jupyter_output><empty_output><jupyter_text>Or maybe what I don't like is the ratio of "chubby sepals", so I want to select those entries with a sepal_length/sepal_width of at least 2:<jupyter_code>slim_df = df[df['sepal_length']/df['sepal_width']>=2.0]
slim_df<jupyter_output><empty_output><jupyter_text>Make and plot our own dataframes What's the percentage of species in each of these dataframes?<jupyter_code>df_species_perc = df['species'].value_counts(normalize=True) * 100
small_sepal_df_species_perc = small_sepal_df['species'].value_counts(normalize=True) * 100
slim_df_species_perc = slim_df['species'].value_counts(normalize=True) * 100
percentages_df = pd.DataFrame({'complete':df_species_perc, 'thin': small_sepal_df_species_perc, 'slim':slim_df_species_perc})
percentages_df
import matplotlib.pyplot as plt
import numpy as np

print(percentages_df)
dataframes_labels = percentages_df.index  # directly from the dataframe

# Set distances between bars and their width
x = np.arange(len(dataframes_labels))  # the label locations
width = 0.25  # the width of the bars

# Do the actual plotting
fig, ax = plt.subplots()
rects1 = ax.bar(x - width, percentages_df['complete'], width, label='complete')
rects2 = ax.bar(x, percentages_df['thin'], width, label='thin')
rects3 = ax.bar(x + width, percentages_df['slim'], width, label='slim')

# Add some text for labels, title and custom x-axis tick labels, etc.
ax.set_ylabel('Percentages')
ax.set_title('Percentages of species per dataframe')
ax.set_xticks(x)
ax.set_xticklabels(dataframes_labels)
fig.legend()
plt.show()<jupyter_output>complete       thin       slim
setosa      33.333333   9.638554        NaN
versicolor  33.333333  50.602410  46.428571
virginica   33.333333  39.759036  53.571429<jupyter_text>Or maybe you prefer it transposed?<jupyter_code>transposed_df = percentages_df.T
print(transposed_df)

dataframes_labels = transposed_df.index  # directly from the dataframe

# Set distances between bars and their width
x = np.arange(len(dataframes_labels))  # the label locations
width = 0.25  # the width of the bars

# Do the actual plotting
fig, ax = plt.subplots()
rects1 = ax.bar(x - width, transposed_df['setosa'], width, label='setosa')
rects2 = ax.bar(x, transposed_df['versicolor'], width, label='versicolor')
rects3 = ax.bar(x + width, transposed_df['virginica'], width, label='virginica')

# Add some text for labels, title and custom x-axis tick labels, etc.
ax.set_ylabel('Percentages')
ax.set_title('Percentages of species per dataframe')
ax.set_xticks(x)
ax.set_xticklabels(dataframes_labels)
fig.legend()
plt.show()<jupyter_output>setosa  versicolor  virginica
complete  33.333333   33.333333  33.333333
thin       9.638554   50.602410  39.759036
slim            NaN   46.428571  53.571429<jupyter_text>More than plotting! So now we know that we can very easily operate, select and plot data by using dataframes. But these are still things that can be done easily in other tools, so let's do something cooler. I want to know if there's a correlation between the different dimensions of the plants. Let's do it from our original dataframe!<jupyter_code>df.corr()  # This is it, really, this is how to make the operation. 
import seaborn as sns 
ax = sns.heatmap(
    df.corr(), 
    vmin=-1, vmax=1, center=0,
    cmap=sns.diverging_palette(20, 220, n=200),
    square=True
)
ax.set_xticklabels(
    ax.get_xticklabels(),
    rotation=45,
    horizontalalignment='right'
)
ax.set_title('Global correlations');<jupyter_output><empty_output><jupyter_text>What if I want to know it for each species?!?!<jupyter_code>for specie in df['species'].unique():
    temp_df = df[df['species'] == specie]
    temp_df.corr()
    
#     print(temp_df)
    print(temp_df.corr())
    ax = sns.heatmap(
    temp_df.corr(), 
    vmin=-1, vmax=1, center=0,
    cmap=sns.diverging_palette(20, 220, n=200),
    square=True
    )
    ax.set_xticklabels(
    ax.get_xticklabels(),
    rotation=45,
    horizontalalignment='right'
    )
    ax.set_title('{0} correlations'.format(specie))
    plt.show()
    plt.close();<jupyter_output>sepal_length  sepal_width  petal_length  petal_width
sepal_length      1.000000     0.746780      0.263874     0.279092
sepal_width       0.746780     1.000000      0.176695     0.279973
petal_length      0.263874     0.176695      1.000000     0.306308
petal_width       0.279092     0.279973      0.306308     1.000000
              sepal_length  sepal_width  petal_length  petal_width
sepal_length      1.000000     0.525911      0.754049     0.546461
sepal_width       0.525911     1.000000      0.560522     0.663999
petal_length      0.754049     0.560522      1.000000     0.786668
petal_width       0.546461     0.663999      0.786668     1.000000
              sepal_length  sepal_width  petal_length  petal_width
sepal_length      1.000000     0.457228      0.864225     0.281108
sepal_width       0.457228     1.000000      0.401045     0.537728
petal_length      0.864225     0.401045      1.000000     0.322108
petal_width       0.281108     0.537728      0.322108     1.00[...]
{'fraction_non_alphanumeric': 0.09806754560231172, 'fraction_numerical': 0.06077298175907531, 'mean_word_length': 4.699948533196088, 'pattern_counts': {'":': 3, '<': 76, '<?xml version=': 0, '>': 76, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '560066', 'license': 'Apache-2.0', 'n_tokens_mistral': 4238, 'n_tokens_neox': 3765, 'n_words': 1166, 'path': '1_intro/1_intro_data_manipulation.ipynb', 'repo_name': 'PiranyaStyle/tutorials_data_project_rnact', 'size': '11074'}