<jupyter_start><jupyter_text>Pandas BasicsPandas is a software library in python for data manipulation and analysis. The name is derived from the term "panel data".The two primary data structures of pandas are Series (1-dimensional) and DataFrame (2-dimensional).  Series:***pandas.Series(data=None, index=None, dtype=None, name=None, copy=False, fastpath=False)***Its a one dimentional ndarray with axis labels (including time series).Lets create a series and do some basic operations.<jupyter_code>import pandas as pd

list_1 = [1,2,3,4,5]
tuple_1 = (1,2,3,4,5)

series_1 = pd.Series(data=list_1)
series_2 = pd.Series(data=tuple_1)

print("List Series:")
print(series_1)
print("\nTuple Series:\n")
print(series_2)<jupyter_output>List Series:
0    1
1    2
2    3
3    4
4    5
dtype: int64

Tuple Series:

0    1
1    2
2    3
3    4
4    5
dtype: int64<jupyter_text>Dataframe:***pandas.DataFrame(data=None, index=None, columns=None, dtype=None, copy=False)***Its a two-dimensional mutable, heterogeneous tabular data structure with labeled axes (rows and columns).Lets create a dataframe and do some basic operations.<jupyter_code>df_dict_1 = [{'a': 1, 'b': 2, 'c': 3, 'd': 4},
             {'a': 10, 'b': 20, 'c': 30, 'd': 40},
              {'a': 100, 'b': 200, 'c': 300, 'd': 400},
                {'a': 1000, 'b': 2000, 'c': 3000, 'd': 4000 }]

df_1 = pd.DataFrame(df_dict_1)

df_1<jupyter_output><empty_output><jupyter_text>Selecting rows and columns of a dataframe:loc and iloc are widely used to fetch rows and columns from a dataframe.iloc is purely integer-location based indexing for selection by position. While selection through iloc start and (stop-1) are fetched. Below are some of the allowed inputs for iloc:1. An integer, (series.iloc[5])2. A list or array of integers, (series.iloc[[0,1]])3. A slice object with ints, (series.iloc[1:5])4. A boolean array, (series.iloc[True,False,True])<jupyter_code># selection using an integer
print("using integers")
print(df_1.iloc[0]) # this fetches all columns of row 1
#to fetch specific columns use comma"," (left side of comma specify rows index and right side columns index)
print("\nspecific columns using integers\n")
print(df_1.iloc[0,1])
# selection using an array of integers
print("\nspecific columns using arrays of integers\n")
print(df_1.iloc[[0,1],[2,3]])
# slice object with ints
print("\nslice\n")
print(df_1.iloc[0:2,1:3])
# some more examples
print(df_1.iloc[0:2,[0,3]])
print("\n\n")
print(df_1.iloc[[0,2],0:3])
print("\n\n")
print(df_1['a'])
print("\n\n")
print(df_1[['a','b']])<jupyter_output>a   d
0   1   4
1  10  40



     a    b    c
0    1    2    3
2  100  200  300



0       1
1      10
2     100
3    1000
Name: a, dtype: int64



      a     b
0     1     2
1    10    20
2   100   200
3  1000  2000<jupyter_text>loc is purely label-location based indexer for selection by label. While selection through loc both the start and the stop are fetched. Below are some of the allowed inputs for loc:1. A single label, e.g. (series.loc[5]) or (series.loc['label']), (note that 5 is interpreted as a label of the index, and never as an integer position along the index).2. A list or array of labels, (series.loc['a', 'b', 'c']).3. A slice object with labels, (series.loc['a':'f']).4. A boolean array of the same length as the axis being sliced, (series.loc[True, False, True].)<jupyter_code># selection using a single label
print(df_1.loc[:,'a']) # this fetches all rows of column 'a'
# selection using array of labels
print(df_1.loc[:,['a','b']]) # this fetches all rows of column 'a' and 'b'
# selecting a slice
print(df_1.loc[2:3,'a':'c']) # this fetches row 2 to 3 and columns 'a' to 'c'<jupyter_output>a     b     c
2   100   200   300
3  1000  2000  3000<jupyter_text>Adding new column (Single)<jupyter_code>df_1['diff'] = df_1['a'] - df_1['b']
df_1
df_1['new_diff'] = df_1.a - df_1.c
df_1<jupyter_output><empty_output><jupyter_text>Adding new column (Multiple)<jupyter_code>df_1 = df_1.assign(add_mul1=df_1['a'] + df_1['b'], add_mul2=df_1['a'] + df_1['c'])
df_1<jupyter_output><empty_output><jupyter_text>Renaming columns<jupyter_code>df_1 = df_1.rename(columns={'diff':'renamed_1','new_diff':'renamed_2','add_mul1':'renamed_3','add_mul2':'renamed_4'})
df_1<jupyter_output><empty_output><jupyter_text>Delete a column<jupyter_code>del df_1['renamed_4']
df_1
# Missing value treatment

import pandas as pd
url = 'https://raw.githubusercontent.com/mkmritunjay/machineLearning/master/Car_sales.csv'
car_sales = pd.read_csv(url)
car_sales.head(10)
car_sales.describe()
car_sales.info()
car_sales["four_year_resale_value"].isna()
# get the missing data count
car_sales.isnull().sum()
# Filling missing data
# Forward fill
car_sales.fillna(method='ffill') # replicates last filled row to all bottom missing rows
# backward fill
car_sales.fillna(method='bfill') # replicates last filled row from bottom to all its above missing rows
# Linear interpolation
car_sales.interpolate()
# pchip interpolation
car_sales.interpolate(method='pchip')
# Group by
df_grpby = car_sales.groupby("Vehicle_type")
df_grpby
# Commonly used functions of group by
df_grpby.first()
df_grpby.last()
df_grpby['Sales_in_thousands'].max()
df_grpby['Sales_in_thousands'].min()
df_grpby['Sales_in_thousands'].mean()
# iterating groups
df_grpby.groups
for v_type, grpby_df in df_grpby:
  print(v_type)
  print(grpby_df)
# Get information about a specific group
df_grpby.get_group("Car")
# Get detailed view of the group
df_grpby.describe()
# check length of the group
len(df_grpby.get_group("Car"))
len(df_grpby) # no of groups
# Group by with multi-index
#car_sales.columns
df_mi = car_sales.set_index(['Vehicle_type','Model'])
df_mi
# aggregation
df_grpby.size()
df_grpby['Sales_in_thousands'].mean()
# multiple aggregation at once
import numpy as np
df_grpby.agg([np.sum, np.mean, np.std])
# renaming columns for aggregation function
df_grpby.agg([np.sum, np.mean, np.std]).rename(columns={"sum":"Total", "mean":"Average", "std":"Standard Deviation"})
# named aggregation, custom agg function
# Transform
df_g2 = car_sales.groupby("Vehicle_type")['Sales_in_thousands'].sum().rename("Total_sales").reset_index()
df_g2
df_merge = car_sales.merge(df_g2)
df_merge["%"] = df_merge["Sales_in_thousands"] / df_merge["Total_sales"]
df_merge.head(10)
# using transform()
df_merge["Total_s"] = car_sales.groupby("Vehicle_type")['Sales_in_thousands'].transform('sum')
df_merge["Mean_sales"] = car_sales.groupby("Vehicle_type")['Sales_in_thousands'].transform('mean')
df_merge.head(10)
#windows functions
import pandas as pd
import numpy as np

price = np.random.randn(10,1)
days = pd.date_range('1/1/2020', periods=10)

df = pd.DataFrame(price, index=days, columns=['StockPrice'])
df
df['prev_price'] = df['StockPrice'].shift(1)
df
df['daily_change'] = df['StockPrice'] / df['prev_price'] - 1
df
df['expanding_mean'] = df['daily_change'].expanding().mean()
df
df['rolling_mean'] = df['daily_change'].rolling(3).mean()
df
# filter
df_filter = car_sales.filter(items=['Vehicle_type','Sales_in_thousands'])
df_filter
df_regex = car_sales.filter(regex='e$', axis=1)
df_regex
# join / merge / concat
# join = left, right, inner, outer 
import pandas as pd
df1 = pd.DataFrame({'Key':['X','Y'],
                    'A':['A1','A2'],
                    'B':['B1','B2']}).set_index('Key')
df1
df2 = pd.DataFrame({'Key':['X','Z'],
                    'A':['A1','A2'],
                    'C':['C1','C2']}).set_index('Key')
df2
df1.join(df2, lsuffix='_df1', rsuffix='_df2', how='left')
df1.join(df2, lsuffix='_df1', rsuffix='_df2', how='right')
df1.join(df2, lsuffix='_df1', rsuffix='_df2', how='inner')
df1.join(df2, lsuffix='_df1', rsuffix='_df2', how='outer')
# merge
df1.merge(df2, on='A')
df1.merge(df2, on='A',left_index=True)
df1.merge(df2, on='A',left_index=True, how='left')
df1.merge(df2, on='A',left_index=True, how='right')
df1.merge(df2, on='A',left_index=True, how='right', right_index=True)
# concat
pd.concat([df1,df2])
pd.concat([df1,df2], join='inner', axis=1)
pd.concat([df1,df2], join='inner', axis=0)
# apply
import pandas as pd
import numpy as np
temp = np.arange(90,150)
df = pd.DataFrame(temp,columns=['Fahrenheit'])
df
def convertToCelsius(ftemp):
  return ((ftemp - 32) * 5/9)
df['Celsius'] = df['Fahrenheit'].apply(convertToCelsius)
df
# Data Frame reshape
import pandas as pd
df = pd.DataFrame({'EmpName':['a','b','c'],
                   'EmpId':[1,2,3],
                   'EmpTitle':['t1','t2','t3']})
df
df_melted = pd.melt(df,id_vars=['EmpId'], value_vars=['EmpName','EmpTitle'])
df_melted
df_melted2 = pd.melt(df,id_vars=['EmpId','EmpName'], value_vars=['EmpTitle'])
df_melted2
df_melted3 = pd.melt(df,id_vars=['EmpId'], value_vars=['EmpTitle'])
df_melted3
# stack
df_stack = df.stack()
df_stack
# unstack
df_unstack = df_stack.unstack()
df_unstack
df<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15794825443452717, 'fraction_numerical': 0.038639701728618234, 'mean_word_length': 5.295874822190612, 'pattern_counts': {'":': 3, '<': 30, '<?xml version=': 0, '>': 30, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '566084', 'license': 'MIT', 'n_tokens_mistral': 3506, 'n_tokens_neox': 3307, 'n_words': 824, 'path': 'pandas_basics.ipynb', 'repo_name': 'mkmritunjay/machineLearning', 'size': '8851'}