<jupyter_start><jupyter_text>Pandas basicsIn this notebook we will **learn** how to work with the two main data types in `pandas`: `DataFrame` and `Series`.  Data structures (`pandas`)  `Series`In `pandas`, series are the building blocks of dataframes.Think of a series as a column in a table. A series collects *observations* about a given *variable*.<jupyter_code>from random import random
import pandas as pd
import numpy as np
from pandas import Series, DataFrame<jupyter_output><empty_output><jupyter_text>Numerical series<jupyter_code># let's create a series containing 100 random numbers
# ranging between 0 and 1

s = pd.Series([random() for n in range(0, 10)])<jupyter_output><empty_output><jupyter_text>Each observation in the series has an **index** as well as a set of **values**: they can be accessed via the omonymous properties:<jupyter_code>s.index
list(s.index)
s.values<jupyter_output><empty_output><jupyter_text>The `head()` and `tail()` methods allows for looking at the begininning and end of a series:<jupyter_code>s.head()
s.tail()<jupyter_output><empty_output><jupyter_text>The `value_counts()` method returns a count of distinct values within a series. Is there any number in `s` that occurs twice?<jupyter_code># a `Series` can be easily cast into a list

list(s.value_counts()).count(2)<jupyter_output><empty_output><jupyter_text>Another way of verifying this:<jupyter_code>s.is_unique
s.min()
s.max()
s.mean()
s.median()<jupyter_output><empty_output><jupyter_text>Datetime series<jupyter_code>from random import randint
from datetime import date
# let's generate a list of random dates
# in the range 1900-1950

dates = [
    date(
        year,
        randint(1, 12),
        randint(1, 28) # try replacing with 31 and see what happens
    )
    for year in range(1900,1950)
]
s1 = pd.Series(dates)
s1
type(s1[1])
s1 = Series(pd.to_datetime(dates))
type(s1[1])
s1[1].day_name()
s1.min()
s1.max()
s1.mean()<jupyter_output><empty_output><jupyter_text>`DataFrame`What is a `pandas.DataFrame`? Think of it as an in-memory spreadsheet that you can analyse and manipulate programmatically.A `DataFrame` is a collection of `Series` having the same length and whose indexes are in sync. A *collection* means that each column of a dataframe is a series Let's create a toy `DataFrame` by hand.<jupyter_code>dates = [
    date(
        year,
        randint(1, 12),
        randint(1, 28) # try replacing with 31 and see what happens
    )
    for year in range(1980,1990)
]
counts = [
    randint(0, 10000)
    for i in range(0, 10)
]
event_types = ["fire", "flood", "car_crash", "plane_crash"]
events = [
    np.random.choice(event_types)
    for i in range(0, 10)
]
assert len(events) == len(counts) == len(dates)
toy_df = pd.DataFrame({
    "date": dates,
    "count": counts,
    "event": events
})
toy_df<jupyter_output><empty_output><jupyter_text>**Try out**: what happens if you change the length of either of the two lists? Try e.g. passing 20 dates instead of 10.<jupyter_code># a df is a collection of series
# each column is a series

type(toy_df.date)<jupyter_output><empty_output><jupyter_text>Data manipulation in `pandas`  Data typesString, datetimes (see above), categorical data.In `pandas`, categories behave very much like string, yet they lead to better performances (faster operations, optimized storage). Bottom-up approach:<jupyter_code># transforms a Series with strings into categories

toy_df.event.astype('category')<jupyter_output><empty_output><jupyter_text>Exploring a dataframeExploring a dataframe: df.head(), df.tail(), df.info(). The method `info()` gives you information about a dataframe:- how much space does it take in memory?- what is the datatype of each column?- how many records are there?- how many `null` values does each column contain (!)?<jupyter_code>toy_df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 10 entries, 0 to 9
Data columns (total 3 columns):
 #   Column  Non-Null Count  Dtype 
---  ------  --------------  ----- 
 0   date    10 non-null     object
 1   count   10 non-null     int64 
 2   event   10 non-null     object
dtypes: int64(1), object(2)
memory usage: 368.0+ bytes<jupyter_text>Alternatively, if you need to know only the number of columns and rows you can use the `.shape` property.It returns a tuple with 1) number of rows, 2) number of columns.<jupyter_code>toy_df.shape<jupyter_output><empty_output><jupyter_text>`head()` prints by first five rows of a dataframe:<jupyter_code>toy_df.head()<jupyter_output><empty_output><jupyter_text>But the number of lines displayed is a parameter that can be changed:<jupyter_code>toy_df.head(2)<jupyter_output><empty_output><jupyter_text>`tail()` does the opposite, i.e. prints the last n rows in the dataframe:<jupyter_code>toy_df.tail()<jupyter_output><empty_output><jupyter_text>Adding columns Let's go back to our toy dataframe:<jupyter_code>toy_df.head()<jupyter_output><empty_output><jupyter_text>Using the column selector with the name of a column that does not exist yet will add the effect of setting the values of all rows in that column to the value specified.<jupyter_code>toy_df['country'] = "UK"
toy_df.head(3)<jupyter_output><empty_output><jupyter_text>But if the column already exists, its value is reset:<jupyter_code>toy_df['country'] = "USA"
toy_df.head(3)<jupyter_output><empty_output><jupyter_text>Removing columns The double square bracket notation ``[[...]]`` returns a dataframe having only the columns specified inside the inner brackets. This said, removing a column is done by unselecting it:<jupyter_code># here we removed the column country 

toy_df2 = toy_df[['date', 'count', 'event']]
# it worked!

toy_df2.head()<jupyter_output><empty_output><jupyter_text>Setting a column as index<jupyter_code>toy_df.set_index('date')
toy_df.head(3)
toy_df.set_index('date', inplace=True)
toy_df.head(3)<jupyter_output><empty_output><jupyter_text>**Q**: can you explain the effect of the `inplace` parameter by looking at the cells above?  Accessing data .loc, .iloc, slicing, iteration over rows<jupyter_code>toy_df.head(3)<jupyter_output><empty_output><jupyter_text>Label-based indexing<jupyter_code>toy_df.loc[date(1980,1,1):date(1982,1,1)]<jupyter_output><empty_output><jupyter_text>Integer-based indexing<jupyter_code># select a single row, the first one

toy_df.iloc[0]
# select  a range of rows by index

toy_df.iloc[[1,3,-1]]
# select  a range of rows with slicing

toy_df.iloc[0:5]
toy_df.index<jupyter_output><empty_output><jupyter_text>Iterating over rows<jupyter_code>for n, row in toy_df.iterrows():
    print(n)
for n, row in toy_df.iterrows():
    print(n, row.event)<jupyter_output>1980-01-23 plane_crash
1981-04-15 plane_crash
1982-11-12 flood
1983-11-22 plane_crash
1984-11-04 fire
1985-06-19 flood
1986-09-14 plane_crash
1987-01-05 fire
1988-03-11 flood
1989-10-10 car_crash
{'fraction_non_alphanumeric': 0.11191966234900305, 'fraction_numerical': 0.03041769756949498, 'mean_word_length': 5.446529080675422, 'pattern_counts': {'":': 3, '<': 96, '<?xml version=': 0, '>': 96, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '105589', 'license': 'MIT', 'n_tokens_mistral': 2533, 'n_tokens_neox': 2349, 'n_words': 719, 'path': 'notebooks/3_PandasBasics.ipynb', 'repo_name': 'Giovanni1085/UvA_CDH_2020', 'size': '6871'}