<jupyter_start><jupyter_text>Python Pandas TutorialPandas is great and a defacto library for data mungling in Python. In this tutorial, we will take a look at the basic operations that a data scientist needs to use Pandas. See official page for more information: https://pandas.pydata.org/  Import Pandas and Numpy<jupyter_code>import pandas as pd
import numpy as np<jupyter_output><empty_output><jupyter_text>Data TypesBefore we delve into the details, let's first understand the Pandas Data Types, namely Series and DataFrames.  SeriesSeries are the foundational block of Pandas. You can think of it as labeled (indexed) arrays<jupyter_code># Create Series
x = pd.Series(np.random.randn(5))
x
# Create Series with index (if no index is provided, then numbered from 1 to n)
x = pd.Series(np.random.randn(5), index=['k', 'l', 'm', 'n', 'o'])
x
# You can provide a dictionary to create series as well (with key as indices)
d = {
    'k': 1,
    'l': 2,
    'm': 3,
    'n': 7
}
y = pd.Series(d)
y
# Series acts like ndarray in numpy so you can slice and do operations
print("From second element to end:\n{}\n".format(y[1:]))
print("Except for the last element:\n{}\n".format(y[:-1]))
print("Elements at indices:\n{}\n".format(y[[2,3]]))
# You can even do operations, if an index does not exist in one operand, it will be NaN, which indicates not a number
# NaN basically means missing element in pandas
x + y<jupyter_output><empty_output><jupyter_text>DataFramesDataFrames are the foundational block of Pandas. It is most important piece of Pandas that you need to master.One can see a dataframe as a table (e.g. pivot table in Excel or a database table), organized as rows and columns.Below let us see how DataFrames can be created and used<jupyter_code># Create an empty DataFrame with columns specified
df = pd.DataFrame(columns=['a', 'b', 'c'])
print("Empty DF:\n{}\n".format(df))

# Create from Series
s = pd.Series(np.random.rand(5), index=['a', 'b', 'c', 'd', 'e'])
df = pd.DataFrame(s, columns=['N'])
print("DF from created from Series:\n{}\n".format(df))

# Create from Dictionary (keys will be column names, values will be rows)
d = {'a': np.random.rand(5), 'b': np.random.rand(5), 'c': np.random.rand(5)}
df = pd.DataFrame(d)
print("DF from created from a Dictionary:\n{}\n".format(df))
# Create a sample dataframe
d = {'a': np.random.rand(5), 'b': np.random.rand(5), 'c': np.random.rand(5)}
df = pd.DataFrame(d)
print("Original:\n{}\n".format(df))

# Column Selection
col_a = df['a']
print("Column A has type: {} and content:\n{}\n".format(type(col_a), col_a))

# Column Addition (just treat df like a dictionary)
col_d = np.random.rand(5)
df['d'] = col_d
print("New df with column d:\n{}\n".format(df))

# You can insert column to specific location with insert function
col_e = np.random.rand(5)
df.insert(1, 'e', col_e)
print("New df with column e:\n{}\n".format(df))

# Delete a column
del df['d']
del df['e']
print("Original:\n{}\n".format(df))
# Create a sample dataframe
d = {'a': np.random.rand(5), 'b': np.random.rand(5), 'c': np.random.rand(5)}
df = pd.DataFrame(d)
df.index = ['e', 'f', 'g', 'h', 'j']
print("Original:\n{}\n".format(df))

# Select row by index label
row = df.loc['f']
print("Row of type:{} content:\n{}\n".format(type(row), row))


# Select row by integer location
row = df.iloc[3]
print("Row of type:{} content:\n{}\n".format(type(row), row))

# Select multiple rows (result is a dataframe)
df_selected = df[2:4]
print("Sliced with indices:\n{}\n".format(df_selected))

# Select multiple rows with a boolean vector
vec = [True, False, False, True, False]
df_selected = df[vec]
print("Sliced with boolean vector:\n{}\n".format(df_selected))
# You can do any operations with df and it will be auto broadcasted to all elements
d = {'a': np.random.rand(3), 'b': np.random.rand(3)}
df = pd.DataFrame(d, index=['e', 'f', 'g'])
print("Original df:\n{}\n".format(df))
print("Original df:\n{}\n".format(df + 3))
print("Original df:\n{}\n".format(df * 2))
# You can print details of a dataframe with info() method
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Index: 3 entries, e to g
Data columns (total 2 columns):
a    3 non-null float64
b    3 non-null float64
dtypes: float64(2)
memory usage: 72.0+ bytes
{'fraction_non_alphanumeric': 0.15957696827262044, 'fraction_numerical': 0.010575793184488837, 'mean_word_length': 4.83013698630137, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '611372', 'license': 'MIT', 'n_tokens_mistral': 1490, 'n_tokens_neox': 1438, 'n_words': 520, 'path': 'data-wrangling/pandas.ipynb', 'repo_name': 'bulutmf/data-science-toolbox', 'size': '4255'}