<jupyter_start><jupyter_text>**Introduction to Pandas**In this section of the course we will learn how to use pandas for data analysis. You can think of pandas as an extremely powerful version of Excel, with a lot more features. In this section of the course, you should go through the notebooks in this order:  **What is Pandas?**Pandas is a software library written for the Python programming language for data manipulation and analysis. Python with Pandas is used in a wide range of fields including academic and commercial domains including finance, economics, Statistics, analytics, etc. **Key Features of Pandas*** Fast and efficient DataFrame object with default and customized indexing.* Data alignment and integrated handling of missing data.* Reshaping and pivoting of date sets.* Columns from a data structure can be deleted or inserted.* Group by data for aggregation and transformations.* High performance merging and joining of data.* Time Series functionality.  **Series**The first main data type we will learn about for pandas is the Series data type. Let's import Pandas and explore the Series object.A Pandas Series is like a column in a table.It is a one-dimensional array holding data of any type.A Series is very similar to a NumPy array (in fact it is built on top of the NumPy array object). *What differentiates the NumPy array from a Series, is that a Series can have axis labels, meaning it can be indexed by a label, instead of just a number location. It also doesn't need to hold numeric data, it can hold any arbitrary Python Object.*Let's explore this concept through some examples: **pandas.Series( data, index, dtype, copy)**where* data= data takes various forms like ndarray, list, constants* index= Index values must be unique and hashable, same length as data. Default np.arrange(n) if no index is passed.* dtype= dtype is for data type. If None, data type will be inferred* copy= Copy data. Default False  **Creating a Series**You can convert a list,numpy array, or dictionary to a Series:<jupyter_code>import pandas as pd
import numpy as np
labels = ['a','b','c']
my_list = [10,20,30]
arr = np.array([10,20,30])
d = {'a':10,'b':20,'c':30}
pd.Series(data=my_list)
pd.Series(data=my_list,index=labels)
#pd.Series(my_list,labels)
#pd.Series(arr)
#pd.Series(arr,labels)
#pd.Series(d)<jupyter_output><empty_output><jupyter_text>**Data in a Series**A pandas Series can hold a variety of object types:<jupyter_code>pd.Series(data=labels)
# Even functions (although unlikely that you will use this)
pd.Series([sum,print,len])<jupyter_output><empty_output><jupyter_text>**Using an Index**The key to using a Series is understanding its index. Pandas makes use of these index names or numbers by allowing for fast look ups of information (works like a hash table or dictionary).Let's see some examples of how to grab information from a Series. Let us create two sereis, ser1 and ser2:<jupyter_code>ser1 = pd.Series([1,2,3,4],index = ['USA', 'Germany','USSR', 'Japan'])    
ser1
ser2 = pd.Series([1,2,5,4],index = ['USA', 'Germany','Italy', 'Japan'])   
ser2
ser1['USA']
ser1 + ser2<jupyter_output><empty_output><jupyter_text>**DataFrames**A Data frame is a two-dimensional data structure, i.e., data is aligned in a tabular fashion in rows and columns.Let's use pandas to explore this topic! **pandas.dataframes**pandas.DataFrame( data, index, columns, dtype, copy)where * data= data takes various forms like ndarray, series, map, lists, dict, constants and also another DataFrame.* index= For the row labels, the Index to be used for the resulting frame is Optional Default np.arange(n) if no index is passed.* columns= For column labels, the optional default syntax is - np.arange(n). This is only true if no index is passed.* dtype= Data type of each column.* copy= This command (or whatever it is) is used for copying of data, if the default is False.<jupyter_code>import pandas as pd
data = [1,2,3,4,5]
df = pd.DataFrame(data)
df
data = [{'a': 1, 'b': 2},{'a': 5, 'b': 10, 'c': 20}]
df = pd.DataFrame(data)
df<jupyter_output><empty_output><jupyter_text>Here NaN means Not a Number.  **Selection and Indexing**Let's learn the various methods to grab data from a DataFrame<jupyter_code>from numpy.random import randn
np.random.seed(101)
df = pd.DataFrame(randn(5,4),index='A B C D E'.split(),columns='W X Y Z'.split())
df
df['W']
# Pass a list of column names
df[['W','Z']]
# SQL Syntax (NOT RECOMMENDED!)
df.W<jupyter_output><empty_output><jupyter_text>DataFrame Columns are just Series<jupyter_code>type(df['W'])<jupyter_output><empty_output><jupyter_text>**Creating a new column:**<jupyter_code>df['new'] = df['W'] + df['Y']
df<jupyter_output><empty_output><jupyter_text>** Removing Columns**<jupyter_code>df.drop('new',axis=1)                                  
# Not inplace unless specified!
df  
df.drop('new',axis=1,inplace=True)
df<jupyter_output><empty_output><jupyter_text>Can also drop rows this way:<jupyter_code>df.drop('E',axis=0)<jupyter_output><empty_output><jupyter_text>** Selecting Rows**<jupyter_code>df.loc['A']<jupyter_output><empty_output><jupyter_text>Or select based off of position instead of label<jupyter_code>df.iloc[2]<jupyter_output><empty_output><jupyter_text>** Selecting subset of rows and columns **<jupyter_code>df.loc['B','Y']
df.loc[['A','B'],['W','Y']]<jupyter_output><empty_output><jupyter_text>**Conditional Selection**An important feature of pandas is conditional selection using bracket notation, very similar to numpy:<jupyter_code>df
df>0
#df[df>0]
#df[df['W']>0]
df[df['W']>0]['Y']
#df[df['W']>0][['Y','X']]<jupyter_output><empty_output><jupyter_text>For two conditions you can use | and & with parenthesis:<jupyter_code>df[(df['W']>0) & (df['Y'] > 1)]<jupyter_output><empty_output><jupyter_text>**More Index Details**Let's discuss some more features of indexing, including resetting the index or setting it something else. We'll also talk about index hierarchy!<jupyter_code>df
# Reset to default 0,1...n index
df.reset_index()
newind = 'CA NY WY OR CO'.split()
df['States'] = newind
df
df.set_index('States')
df
df.set_index('States',inplace=True)
df<jupyter_output><empty_output><jupyter_text>**Multi-Index and Index Hierarchy**Let us go over how to work with Multi-Index, first we'll create a quick example of what a Multi-Indexed DataFrame would look like:<jupyter_code># Index Levels
outside = ['G1','G1','G1','G2','G2','G2']
inside = [1,2,3,1,2,3]
hier_index = list(zip(outside,inside))
hier_index = pd.MultiIndex.from_tuples(hier_index)
hier_index
df = pd.DataFrame(np.random.randn(6,2),index=hier_index,columns=['A','B'])
df<jupyter_output><empty_output><jupyter_text>Now let's show how to index this! For index hierarchy we use df.loc[], if this was on the columns axis, you would just use normal bracket notation df[]. Calling one level of the index returns the sub-dataframe:<jupyter_code>df.loc['G1']
df.loc['G1'].loc[1]
df.index.names
df.index.names = ['Group','Num']
df
df.xs('G1')
df.xs(['G1',1])
df.xs(1,level='Num')<jupyter_output><empty_output><jupyter_text>**Missing Data**Let's show a few convenient methods to deal with Missing Data in pandas:<jupyter_code>import numpy as np
import pandas as pd
df = pd.DataFrame({'A':[1,2,np.nan],
                  'B':[5,np.nan,np.nan],
                  'C':[1,2,3]})
df
df.dropna()
df.dropna(axis=1)
df.dropna(thresh=2)
df.fillna(value='FILL VALUE')
df['A'].fillna(value=df['A'].mean())<jupyter_output><empty_output><jupyter_text>**GroupBy**The groupby method allows you to group rows of data together and call aggregate functions<jupyter_code>import pandas as pd
# Create dataframe
data = {'Company':['GOOG','GOOG','MSFT','MSFT','FB','FB'],
       'Person':['Sam','Charlie','Amy','Vanessa','Carl','Sarah'],
       'Sales':[200,120,340,124,243,350]}
df = pd.DataFrame(data)
df<jupyter_output><empty_output><jupyter_text>** Now you can use the .groupby() method to group rows together based off of a column name. For instance let's group based off of Company. This will create a DataFrameGroupBy object:**<jupyter_code>df.groupby('Company')<jupyter_output><empty_output><jupyter_text>You can save this object as a new variable:<jupyter_code>by_comp = df.groupby("Company")<jupyter_output><empty_output><jupyter_text>And then call aggregate methods off the object:<jupyter_code>by_comp.mean()
df.groupby('Company').mean()<jupyter_output><empty_output><jupyter_text>More examples of aggregate methods:<jupyter_code>by_comp.std()
by_comp.min()
by_comp.max()
by_comp.count()
by_comp.describe()
by_comp.describe().transpose()
by_comp.describe().transpose()['GOOG']<jupyter_output><empty_output><jupyter_text>**Merging,Joining,and Concatenating**There are 3 main ways of combining DataFrames together: Merging, Joining and Concatenating. In this lecture we will discuss these 3 methods with examples. **Example DataFrames**<jupyter_code>import pandas as pd
df1 = pd.DataFrame({'A': ['A0', 'A1', 'A2', 'A3'],
                        'B': ['B0', 'B1', 'B2', 'B3'],
                        'C': ['C0', 'C1', 'C2', 'C3'],
                        'D': ['D0', 'D1', 'D2', 'D3']},
                        index=[0, 1, 2, 3])
df2 = pd.DataFrame({'A': ['A4', 'A5', 'A6', 'A7'],
                        'B': ['B4', 'B5', 'B6', 'B7'],
                        'C': ['C4', 'C5', 'C6', 'C7'],
                        'D': ['D4', 'D5', 'D6', 'D7']},
                         index=[4, 5, 6, 7]) 
df3 = pd.DataFrame({'A': ['A8', 'A9', 'A10', 'A11'],
                        'B': ['B8', 'B9', 'B10', 'B11'],
                        'C': ['C8', 'C9', 'C10', 'C11'],
                        'D': ['D8', 'D9', 'D10', 'D11']},
                        index=[8, 9, 10, 11])
df1
df2
df3<jupyter_output><empty_output><jupyter_text>**Concatenation**Concatenation basically glues together DataFrames. Keep in mind that dimensions should match along the axis you are concatenating on. You can use pd.concat and pass in a list of DataFrames to concatenate together:<jupyter_code>pd.concat([df1,df2,df3])
pd.concat([df1,df2,df3],axis=1)<jupyter_output><empty_output><jupyter_text>**Example DataFrames**<jupyter_code>left = pd.DataFrame({'key': ['K0', 'K1', 'K2', 'K3'],
                     'A': ['A0', 'A1', 'A2', 'A3'],
                     'B': ['B0', 'B1', 'B2', 'B3']})
   
right = pd.DataFrame({'key': ['K0', 'K1', 'K2', 'K3'],
                          'C': ['C0', 'C1', 'C2', 'C3'],
                          'D': ['D0', 'D1', 'D2', 'D3']})    
left
right<jupyter_output><empty_output><jupyter_text>**Merging**The merge function allows you to merge DataFrames together using a similar logic as merging SQL Tables together. For example:<jupyter_code>pd.merge(left,right,how='inner',on='key')<jupyter_output><empty_output><jupyter_text>Or to show a more complicated example:<jupyter_code>left = pd.DataFrame({'key1': ['K0', 'K0', 'K1', 'K2'],
                     'key2': ['K0', 'K1', 'K0', 'K1'],
                        'A': ['A0', 'A1', 'A2', 'A3'],
                        'B': ['B0', 'B1', 'B2', 'B3']})
    
right = pd.DataFrame({'key1': ['K0', 'K1', 'K1', 'K2'],
                               'key2': ['K0', 'K0', 'K0', 'K0'],
                                  'C': ['C0', 'C1', 'C2', 'C3'],
                                  'D': ['D0', 'D1', 'D2', 'D3']})
pd.merge(left, right, on=['key1', 'key2'])
pd.merge(left, right, how='outer', on=['key1', 'key2'])
pd.merge(left, right, how='right', on=['key1', 'key2'])
pd.merge(left, right, how='left', on=['key1', 'key2'])<jupyter_output><empty_output><jupyter_text>**Joining**Joining is a convenient method for combining the columns of two potentially differently-indexed DataFrames into a single result DataFrame.<jupyter_code>left = pd.DataFrame({'A': ['A0', 'A1', 'A2'],
                     'B': ['B0', 'B1', 'B2']},
                      index=['K0', 'K1', 'K2']) 

right = pd.DataFrame({'C': ['C0', 'C2', 'C3'],
                    'D': ['D0', 'D2', 'D3']},
                      index=['K0', 'K2', 'K3'])
left.join(right)
left.join(right, how='inner')<jupyter_output><empty_output><jupyter_text>**Operations**There are lots of operations with pandas that will be really useful to you, but don't fall into any distinct category. Let's show them here in this lecture:<jupyter_code>import pandas as pd
df = pd.DataFrame({'col1':[1,2,3,4],'col2':[444,555,666,444],'col3':['abc','def','ghi','xyz']})
df.head()<jupyter_output><empty_output><jupyter_text>**Info on Unique Values**<jupyter_code>df['col2'].unique()
df['col2'].nunique()
df['col2'].value_counts()<jupyter_output><empty_output><jupyter_text>**Selecting Data**<jupyter_code>#Select from DataFrame using criteria from multiple columns
newdf = df[(df['col1']>2) & (df['col2']==444)]
newdf<jupyter_output><empty_output><jupyter_text>**Applying Functions**<jupyter_code>def times2(x):
    return x*2
df['col1'].apply(times2)
df['col3'].apply(len)
df['col1'].sum()<jupyter_output><empty_output><jupyter_text>** Permanently Removing a Column**<jupyter_code>del df['col1']
df<jupyter_output><empty_output><jupyter_text>** Get column and index names: **<jupyter_code>df.columns
df.index<jupyter_output><empty_output><jupyter_text>** Sorting and Ordering a DataFrame:**<jupyter_code>df
df.sort_values(by='col2') #inplace=False by default<jupyter_output><empty_output><jupyter_text>** Find Null Values or Check for Null Values**<jupyter_code>df.isnull()
# Drop rows with NaN Values
df.dropna()<jupyter_output><empty_output><jupyter_text>** Filling in NaN values with something else: **<jupyter_code>import numpy as np
df = pd.DataFrame({'col1':[1,2,3,np.nan],
                   'col2':[np.nan,555,666,444],
                   'col3':['abc','def','ghi','xyz']})
df.head()
df.fillna('FILL')
data = {'A':['foo','foo','foo','bar','bar','bar'],
     'B':['one','one','two','two','one','one'],
       'C':['x','y','x','y','x','y'],
       'D':[1,3,2,5,4,1]}

df = pd.DataFrame(data)
df
df.pivot_table(values='D',index=['A', 'B'],columns=['C'])<jupyter_output><empty_output><jupyter_text>**Data Input and Output**This notebook is the reference code for getting input and output, pandas can read a variety of file types using its pd.read_ methods. Let's take a look at the most common data types:<jupyter_code>import numpy as np
import pandas as pd<jupyter_output><empty_output><jupyter_text>**CSV**  **CSV Input**<jupyter_code>df = pd.read_csv('data.csv')
df<jupyter_output><empty_output><jupyter_text>`df.to_string()` means entire csv folder will be displayed.<jupyter_code>df = pd.read_csv('data.csv')
df.to_string()<jupyter_output><empty_output><jupyter_text>**CSV Output**<jupyter_code>df.to_csv('data',index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.16253838280450358, 'fraction_numerical': 0.02320027294438758, 'mean_word_length': 4.962571196094386, 'pattern_counts': {'":': 0, '<': 169, '<?xml version=': 0, '>': 177, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '42', 'dir': 'jupyter-structured-clean-dedup', 'id': '256124', 'license': 'MIT', 'n_tokens_mistral': 5245, 'n_tokens_neox': 5050, 'n_words': 1403, 'path': 'Python_Session_5.ipynb', 'repo_name': 'ashupadhyay/Getting-Started-With-Python', 'size': '14655'}