<jupyter_start><jupyter_text>A thorough guide to Pandas DataFrame *Data frames* are the central concept in Pandas; they rely on *Pandas Series*. Pandas seriesA Pandas Series, `pandas.Series`, is a one-dimensional `numpy.ndarray` with axis labels (including time series):<jupyter_code>import pandas as pd

s = pd.Series([0, 1, 2, 3, 4, 5 , 6 , 8 , 9])
s<jupyter_output><empty_output><jupyter_text>A *data frame* is a two-dimensional, size-mutable, and potentially heterogeneous tabular data structure akin to SQL table or Excel spreadsheet. It constituted of columns and rows. Columns and rows are separate `pandas.Series` objects. A rows is also qualified as *observation*, or *entry*. Creating a data frame<jupyter_code>import numpy as np

df = pd.DataFrame([("falcon", 389.0),
                   ("parrot", 24.0),
                   ("golden eagle", 320),
                   ("lion", 80.5),
                   ("monkey", np.nan),
                   ("cow", 40),
                   ("zebra", 65),],
                  columns=("animal", "max_speed"))
df<jupyter_output><empty_output><jupyter_text>Data frame indices By default, indices correspond to those of the recorded rows. They can be named:<jupyter_code>import numpy as np

df = pd.DataFrame([("bird", 389.0),
                   ("bird", 24.0),
                   ("bird", 320),
                   ("mammal", 80.5),
                   ("mammal", np.nan),
                   ("mammal", 40),
                   ("mammal", 65),],
                  index=["falcon", 'parrot', "golden eagle", "lion", "monkey", "cow", "zebra"],
                  columns=("cls", "max_speed"))
df<jupyter_output><empty_output><jupyter_text>Accessing the indices (`pandas.DataFrame.index`) `pandas.DataFrame.index` contains the index, i.e. row labels, of the DataFrame:<jupyter_code>df.index<jupyter_output><empty_output><jupyter_text>Turning the indices into a Numpy array:<jupyter_code>df.index.values<jupyter_output><empty_output><jupyter_text>Iterating over indices:<jupyter_code>for i in df.index:
    print(i)<jupyter_output>falcon
parrot
golden eagle
lion
monkey
cow
zebra<jupyter_text>Updating data frame indices<jupyter_code>df2 = pd.DataFrame(df.values, 
                   index=["{} ({})".format(i, df["cls"][i][0]) for i in df.index],
                   columns=df.columns)
df2<jupyter_output><empty_output><jupyter_text>Data frame columns<jupyter_code>df<jupyter_output><empty_output><jupyter_text>A data frame is traversed along (iterated over) its column indices, i.e. axis $1$'s indices.<jupyter_code>for c in df:
    print(c)<jupyter_output>cls
max_speed<jupyter_text>Accessing a specific column:<jupyter_code>df.cls<jupyter_output><empty_output><jupyter_text>or:<jupyter_code>df["cls"]<jupyter_output><empty_output><jupyter_text>Accessing the column indices: `pandas.DataFrame.columns`<jupyter_code>df.columns<jupyter_output><empty_output><jupyter_text>Iterating over columns: `items()` and `iteritems()`<jupyter_code>df<jupyter_output><empty_output><jupyter_text>`pandas.DataFrame.items()` and `pandas.DataFrame.iteritems()`, which have the same behaviour, iterate over `(column name, Series)` pairs, where the `Series` object corresponds to the series of values of column `column name`:<jupyter_code>for c in df.iteritems():
    print(c, '\n')<jupyter_output>('cls', falcon            bird
parrot            bird
golden eagle      bird
lion            mammal
monkey          mammal
cow             mammal
zebra           mammal
Name: cls, dtype: object) 

('max_speed', falcon          389.0
parrot           24.0
golden eagle    320.0
lion             80.5
monkey            NaN
cow              40.0
zebra            65.0
Name: max_speed, dtype: float64)<jupyter_text>For each column, the indices and their corresponding values in each column are displayed.  Creating a new column<jupyter_code>df["weight"] = [1, 4, 5.5, 160, 37, 1000, 400]
df<jupyter_output><empty_output><jupyter_text>Selecting a subset of columns<jupyter_code>df[["cls", "weight"]]<jupyter_output><empty_output><jupyter_text>DataFrame rows As seen above a data frame is traversed along its columns. However, it is sliced along its rows:<jupyter_code>df[:2]<jupyter_output><empty_output><jupyter_text>It cannot be indexed by means of integers through the indexing operators `[]`, i.e. selection by position, though:<jupyter_code>try:
    df[0]
except:
    print("Error: A DataFrame cannot be accessed via any integer.")<jupyter_output>Error: A DataFrame cannot be accessed via any integer.<jupyter_text>Row indices (`df.index`) As seen above, `df.index` returns the rows' indices:<jupyter_code>df.index<jupyter_output><empty_output><jupyter_text>Iterating over rows: `iterrows()` `pandas.DataFrame.iterrows()` iterates over DataFrame rows by returning `(index, Series)` pairs:<jupyter_code>for r in df.iterrows():
    print(r, '\n')<jupyter_output>('falcon', cls           bird
max_speed    389.0
weight         1.0
Name: falcon, dtype: object) 

('parrot', cls          bird
max_speed    24.0
weight        4.0
Name: parrot, dtype: object) 

('golden eagle', cls           bird
max_speed    320.0
weight         5.5
Name: golden eagle, dtype: object) 

('lion', cls          mammal
max_speed      80.5
weight        160.0
Name: lion, dtype: object) 

('monkey', cls          mammal
max_speed       NaN
weight         37.0
Name: monkey, dtype: object) 

('cow', cls          mammal
max_speed      40.0
weight       1000.0
Name: cow, dtype: object) 

('zebra', cls          mammal
max_speed      65.0
weight        400.0
Name: zebra, dtype: object)<jupyter_text>Comparison with `iteritems()`:<jupyter_code>for c in df.iteritems():
    print(c, '\n')<jupyter_output>('cls', falcon            bird
parrot            bird
golden eagle      bird
lion            mammal
monkey          mammal
cow             mammal
zebra           mammal
Name: cls, dtype: object) 

('max_speed', falcon          389.0
parrot           24.0
golden eagle    320.0
lion             80.5
monkey            NaN
cow              40.0
zebra            65.0
Name: max_speed, dtype: float64) 

('weight', falcon             1.0
parrot             4.0
golden eagle       5.5
lion             160.0
monkey            37.0
cow             1000.0
zebra            400.0
Name: weight, dtype: float64)<jupyter_text>Accessing DataFrame elements Accessing first a column, then a row by using the indexing operators `[]`:<jupyter_code>df["max_speed"]["parrot"]<jupyter_output><empty_output><jupyter_text>Accessing rows via their index labels: `pandas.DataFrame.loc` `pandas.DataFrame.loc` allows accessing a group of rows and columns by their index labels or a boolean array.Accessing a given row and column with `loc`:<jupyter_code>df.loc["parrot", "max_speed"]<jupyter_output><empty_output><jupyter_text>identical to:<jupyter_code>df.loc["parrot"]["max_speed"]<jupyter_output><empty_output><jupyter_text>Accessing a group of rows and columns with `loc`:<jupyter_code>df.loc[["parrot", "falcon"], ["cls", "max_speed"]]<jupyter_output><empty_output><jupyter_text>or<jupyter_code>df.loc[("parrot", "falcon"), ("cls", "max_speed")]
df.loc[["parrot", "falcon"]][["cls", "max_speed"]]<jupyter_output><empty_output><jupyter_text>Accessing a row for all columns with `loc`:<jupyter_code>df.loc["parrot"]<jupyter_output><empty_output><jupyter_text>`pandas.DataFrame.at` `pandas.DataFrame.at` accesses a single value for a row-column label pair. It is similar to `pandas.DataFrame.loc` in that it provides a label-based look-up, although `loc` may access a group of rows and columns. `at` is favoured when it is only needed to get or set a single value in a `DataFrame` or `Series`.<jupyter_code>df.at["falcon", "max_speed"] = 388.999
df.at["parrot", "max_speed"] = 23.999<jupyter_output><empty_output><jupyter_text>Accessing rows via their location-based: `pandas.DataFrame.iloc` `pandas.DataFrame.iloc` provides purely integer location-based indexing for selection by position along the rows, contrary to `pandas.DataFrame.loc` that accesses elements by their index labels.<jupyter_code>df.iloc[0]<jupyter_output><empty_output><jupyter_text>Accessing a row for a subset of columns with `iloc`:<jupyter_code>df.iloc[0, [0,1]]<jupyter_output><empty_output><jupyter_text>Data frame's values: `pandas.DataFrame.values` Returns a NumPy representation of the DataFrame.<jupyter_code>df.values<jupyter_output><empty_output><jupyter_text>Filtering and sorting Selecting all the birds, i.e. all the rows that correspond to the class of animal "bird":<jupyter_code>df[df["cls"] == "bird"]<jupyter_output><empty_output><jupyter_text>Selecting all the birds that fly at more than 300 km/h:<jupyter_code>df[(df["cls"] == "bird") & (df.max_speed >= 300)]<jupyter_output><empty_output><jupyter_text>Do not forget the parentheses for each condition.  DataFrame description  Data frame dimensionality: `pandas.DataFrame.shape`<jupyter_code>df.shape<jupyter_output><empty_output><jupyter_text>`pandas.DataFrame.describe()` Generates descriptive statistics such as the count, central tendency, dispersion and the shape of the dataset's distribution (quantiles, min, max, etc.). To obtain the statistics of a categorial variable, the `include` shall be set to `all`.<jupyter_code>df.describe(include="all")<jupyter_output><empty_output><jupyter_text>`n` first rows: `pandas.DataFrame.head(n=5)` Returns the first `n` rows; useful for quickly testing if the object contains the right type of data.<jupyter_code>df.head()<jupyter_output><empty_output><jupyter_text>`pandas.DataFrame.value_counts` `pandas.DataFrame.value_counts` returns the counts of a same row in the data frame:<jupyter_code>df.value_counts()
df.loc["other zebra"] = ("mammal", 65, 400)
df
df.value_counts()<jupyter_output><empty_output><jupyter_text>`pandas.DataFrame.round()` Rounds a DataFrame to a variable number of decimal places:<jupyter_code>df.round(2)<jupyter_output><empty_output><jupyter_text>`pandas.Series.str` and `pandas.Index.str` `pandas.Series.str`, resp. `pandas.Index.str`, returns a set of vectorised string functions for string `Series`, resp. string `Index`. Example:<jupyter_code>df.cls
df.cls.str.len()
df.cls.str.len().max()
df.cls.str.len().mean()
df.cls.str.len().std()
df.cls.str.replace("bird", "beautiful bird")<jupyter_output><empty_output><jupyter_text>With `Index`:<jupyter_code>df.index.str.len()<jupyter_output><empty_output><jupyter_text>Loading data  Loading data from a CSV file: `pandas.DataFrame.read_csv(sep=',')`<jupyter_code>df_titles = pd.read_csv("https://github.com/mothguib/data/raw/main/title_conference.csv", sep=',')
df_titles.head()<jupyter_output><empty_output><jupyter_text>Loading structured data: `pandas.DataFrame.from_records()` Let us create a text with several sentences:<jupyter_code>sentences = """<NAME> began building Monticello at the\
 age of 26.\n"""
sentences += """Construction was done mostly by local masons and\
 carpenters.\n"""
sentences += "He moved into the South Pavilion in 1770.\n"
sentences += """Turning Monticello into a neoclassical masterpiece\
 was Jefferson's obsession."""

print(sentences)<jupyter_output><NAME> began building Monticello at the age of 26.
Construction was done mostly by local masons and carpenters.
He moved into the South Pavilion in 1770.
Turning Monticello into a neoclassical masterpiece was Jefferson's obsession.<jupyter_text>Now, create a bag of words for each sentence:<jupyter_code>corpus = {}

for i, sent in enumerate(sentences.splitlines()):
    corpus["sent{}".format(i)] = dict([(tok, 1) for tok in sent.split()])<jupyter_output><empty_output><jupyter_text>`corpus`'s content:<jupyter_code>corpus<jupyter_output><empty_output><jupyter_text>Data are wrapped up in a Pandas DataFrame by using `pandas.DataFrame.from_records()`, which creates a data frame from a structured ndarray, sequence of tuples or dicts, or DataFrame.<jupyter_code>data = pd.DataFrame.from_records(corpus)
data<jupyter_output><empty_output><jupyter_text>The data frame must be transposed because the first axis in the dictionary `corpus` makes up the columns:<jupyter_code>data = data.T
data<jupyter_output><empty_output><jupyter_text>Determining the rows containing a NaN value `pandas.DataFrame.isnull` or `pandas.DataFrame.isna` detect missing values, also named "not available" or "NA", by returning a boolean same-sized object indicating if the values are NA, i.e. `NaN`or `None`.<jupyter_code>data.isnull()<jupyter_output><empty_output><jupyter_text>`pandas.DataFrame.any` detects if there is at least one true value along a series or a data frame axis<jupyter_code>data.isnull().any(axis=1)
na_rows = data[data.isnull().any(axis=1)].head(n=15)
na_rows<jupyter_output><empty_output><jupyter_text>Filling NA values with `pandas.DataFrame.fillna()`:<jupyter_code>data = data.fillna(0).astype(int)<jupyter_output><empty_output><jupyter_text>Now in one line:<jupyter_code>data = pd.DataFrame.from_records(corpus).T.fillna(0).astype(int)
data.head()
data.head()[data.columns[:15]]<jupyter_output><empty_output><jupyter_text>Considering that `corpus` is a dictionary, `DataFrame`'s constructor could simply have been used in this case:<jupyter_code>data = pd.DataFrame(corpus).T.fillna(0).astype(int)
data[data.columns[:15]]<jupyter_output><empty_output><jupyter_text>`pandas.from_dict` can also be used:<jupyter_code>data = pd.DataFrame.from_dict(corpus).T.fillna(0).astype(int)
data[data.columns[:15]]<jupyter_output><empty_output><jupyter_text>Column reindexing with `pandas.DataFrame.reindex`:<jupyter_code>data = data.reindex(sorted(data.columns), axis=1)
data[data.columns[:15]]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12257922535211267, 'fraction_numerical': 0.017092136150234742, 'mean_word_length': 4.793880152996175, 'pattern_counts': {'":': 1, '<': 208, '<?xml version=': 0, '>': 209, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '53', 'dir': 'jupyter-structured-clean-dedup', 'id': '260481', 'license': 'CC0-1.0', 'n_tokens_mistral': 4872, 'n_tokens_neox': 4623, 'n_words': 1158, 'path': 'Pandas_key_concepts.ipynb', 'repo_name': 'mothguib/datanal', 'size': '13652'}