<jupyter_start><jupyter_text>PandasPandas is a Python package providing fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real world data analysis in Python. Additionally, it has the broader goal of becoming the most powerful and flexible open source data analysis/manipulation tool available in any language. It is already well on its way toward this goal.**Note:** - Pandas is essential high level data manipulation tool, built on Numpy- Documentation [here](http://pandas.pydata.org/pandas-docs/stable/index.html)- 10 minutes tutorial [here](http://pandas.pydata.org/pandas-docs/stable/10min.html)  DataFrame - tabular data structure with rows and columns<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline

# DataFrame is one of Pandas' most important data structures
# Let's build DataFrame from a dictionary

# data lists
student_names = ['John', 'Arnold', 'Donald']
age           = [23, 46, 31]
salary        = [30000, 60000, 40000]

# data dict
data_dict = {'student_names': student_names, 'age': age, 'salary': salary} 

# and finally DataFrame
data_frame = pd.DataFrame(data_dict)
pd.re
type(data_frame), data_frame.shape, data_frame.columns
# csv to DataFrame (dataset from here https://www.kaggle.com/pavansubhasht/ibm-hr-analytics-attrition-dataset/home)
df_ibm = pd.read_csv('../Datasets/ibm_hr_analytics_employee_attrition_performance.csv')
df_ibm.head()
df_ibm.tail(4)
# get general info about data
df_ibm.info()
# select few columns for analysis
df_ibm_subset = df_ibm[['MonthlyIncome', 'Gender', 'Age', 'Attrition', 'EmployeeNumber']].copy()

# let's make non-integer index based on employee number
df_ibm_subset['MonthlyIncome'] = df_ibm_subset['MonthlyIncome'].astype('str')
df_ibm_subset.set_index('MonthlyIncome', inplace=True)
# square brackets selection (columns selection, slicing for rows)
df_ibm_subset[1:5]
# label-based selection (row, column, row and column access)
df_ibm_subset.loc[['2426', '2911'], ['Gender', 'Age']]
# integer-based selection 
df_ibm_subset.iloc[:5, :]
# filtering pandas Data Frame
ind = df_ibm_subset['EmployeeNumber'] < 10
df_ibm_subset[ind]
# few more ways to filter 
df_ibm_subset[np.logical_and(df_ibm_subset.Age > 35, df_ibm_subset.EmployeeNumber < 10)]

df_ibm_subset[(df_ibm_subset.Age > 35) & (df_ibm_subset.EmployeeNumber < 10)] # &, |, ~  
# looping over Data Frame
for ind, row in df_ibm_subset[:5].iterrows():
    print(row['Age'], row['Attrition'])
for ind, row in df_ibm_subset[:5].iterrows():
    df_ibm_subset.loc[ind, 'EmployeeNumberSquared'] = row['EmployeeNumber']**2
    
df_ibm_subset['EmployeeNumberSquared'] = df_ibm_subset['EmployeeNumber'].apply(lambda x: your_function(x)**2)
df_ibm_subset[:10]<jupyter_output><empty_output><jupyter_text>Ploting with Pandas<jupyter_code>plot_1 = df_ibm.Age.plot(kind='hist', bins=20, color='g', legend=True)
plot_2 = df_ibm.HourlyRate.plot(kind='hist', bins=20, color='r', legend=True)
plot_3 = df_ibm.JobRole.value_counts().plot(kind='barh', color='m', legend=True)
# save figure
fig = plot_1.get_figure()
fig.savefig("age_histogram.png")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12295081967213115, 'fraction_numerical': 0.018518518518518517, 'mean_word_length': 6.241758241758242, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 11, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '219206', 'license': 'MIT', 'n_tokens_mistral': 1175, 'n_tokens_neox': 1104, 'n_words': 289, 'path': 'Modules/1-Preparation/notebooks/intro_to_pandas.ipynb', 'repo_name': 'open-dse/into_to_ds_with_python', 'size': '3294'}