<jupyter_start><jupyter_text>**Data Analysis with Pandas**Pandas provides a range of toolkit for data analysis through the two basic pandas data strucutures: DataFrame and Series. This guide introduces the basic characteristics of the two classes and explores the capacity Pandas provides through these two classes and other powerful functions.  **I. DataFrame**The DataFrame class can efficiently handle data in tabular form, similar to a relational database and an Excel spreadsheet.<jupyter_code>import pandas as pd
# Defining a dataframe with specified column label and index values.
df = pd.DataFrame([400, 200, 90, 300], 
                  columns=['Price'], 
                  index=['a', 'b', 'c', 'd'])

# Index and Columns attribute and object.
df.index
df.columns

# Get value at the specified index
df.loc['c']
df.iloc[1:3]

# Calculate the sum of a column
df.sum()

# Use apply method to calculate in a vectorized fashion
df.apply(lambda x: x ** 2)
df**2 

# Modifying dataframe by adding columns of different object, 
# even df object is possible. The indices will be aligned automatically.
df['put/call'] = (0.3, 0.5, 0.8, 0.2)
df['ticker'] = pd.DataFrame(['TSLA', 'AMD', 'AAPL', 'BABA'], 
                            index=['a', 'c', 'b', 'd'])

"""
Note that appending other data to the pandas dataframe work similarly.
However, appending some data structure such as dictionary, 
incomplete data can results in loss of index information 
and NaN values respectively.
"""

df

# Calculate the mean over the columns specified.
df[['Price','put/call']].mean()
df[['Price','put/call']].std()
import numpy as np

# Set up a normally distributed random number ndarray.
np.random.seed(100)
a = np.random.standard_normal((9,4))

# Creating the dataframe from ndarray object.
df = pd.DataFrame(a)

# This process can be "undone", that is, ndarray can be created from df.
df.values
np.array(df)<jupyter_output><empty_output><jupyter_text>DateTimeIndex object can be generated by date_range function. The frequency parameter that can be taken are as followed:B Business day frequencyC Custom business day frequencyD Calendar day frequency **2. Basic Analytics**<jupyter_code># Get metainformation regarding the data.
df.info()

# Get summary statistics per columns. 
df.describe()

# Calculate differents sums and means.
df.sum()
df.mean()
df.mean(axis=0)

# Row-wise mean
df.mean(axis=1)

# Column-wise cumulative mean.
df.cumsum()<jupyter_output><empty_output><jupyter_text>In general, Pandas DataFrame object also support NumPy Universal functions, similar to applying such operations to an ndarray objects that contain the same type of data.<jupyter_code># Column-wise mean
np.mean(df)

# Element-wise natural logarithm
np.log(df)

# Element-wise square root for the absolute values.
np.sqrt(abs(df))<jupyter_output><empty_output><jupyter_text>3. Basic VisualizationPandas also provide a wrapper for matplotlib specifically designed to support DataFrame object. Such, some basic visualization of data can be performed on pandas.<jupyter_code>from pylab import plt, mpl

# Customize the plotting style with seaborn
plt.style.use('seaborn')
mpl.rcParams['font.family'] = 'serif'

# makes the plot appears in line with the code.
%matplotlib inline

# Plot the mean as line plot
df.cumsum().plot(lw=2.0, figsize=(10, 6))

# Alternatively, we can plot a bar plot with the same data
df.plot.bar(figsize=(10, 6), rot=15)

# or, equivalently: df.plot(kind='bar', figsize=(10, 6))<jupyter_output><empty_output><jupyter_text>**II. The Series Class**Series is another important class in the Pandas packages. It is characterized by its feature of having only a single column of data. Such, it can be obtained from selecting a single column from a multi-column DataFrame object.<jupyter_code>type(df)
sc = pd.Series(np.linspace(0, 15, 7), name='series')
type(sc)

# All main methods for pandas DataFrame work for Series 
sc.plot(lw=2.0, figsize=(10, 6))
sc.mean()<jupyter_output><empty_output><jupyter_text>4. GroupBy OperationGroupBy operation works similar to the GroupBy in SQL<jupyter_code># Randomly construct some dataframe
import numpy as np
np.random.seed(100)
a = np.random.standard_normal((9, 4))
df = pd.DataFrame(a)
df.columns = ['No1', 'No2', 'No3', 'No4']
dates = pd.date_range('2019-1-1', periods=9, freq='M')

df['Quarter']=['Q1', 'Q1', 'Q1', 'Q2', 'Q2', 'Q2', 'Q3', 'Q3', 'Q3']

# Group according to the quarter column.
groups = df.groupby('Quarter')

# Obtain various information about the groups.
groups.size()
groups.mean()
groups.max()

# Aggregating the calculated results. 
groups.aggregate([min, max]).round(2)<jupyter_output><empty_output><jupyter_text>Grouping can also be done with multiple groups, such multiple criteria can be used.<jupyter_code>df['Odd_Even'] = ['Odd', 'Even', 'Odd', 'Even', 'Odd', 'Even', 'Odd', 'Even', 'Odd']
groups = df.groupby(['Quarter', 'Odd_Even'])
groups.size()
groups[['No1', 'No4']].aggregate([sum, np.mean])<jupyter_output><empty_output><jupyter_text>**5. Concatenation, Joining, and Merging**Pandas also allows various approaches to combine two simple data sets into a single dataframe object.<jupyter_code>df1 = pd.DataFrame([1000, 200, 90, 300], 
                  columns=['Price'], 
                  index=['a', 'b', 'c', 'd'])
df1['Tickers'] = ['TSLA', 'AMD', 'AAPL', 'BABA']
df2 = pd.DataFrame([200, 2000, 300, 1000], 
                  columns=['Price'], 
                  index=['a', 'b', 'c', 'd'])
df2['Tickers'] = ['SEA', 'AMZN', 'ZM', 'GOOGL']

df3 = pd.DataFrame([0.3, 0.5, 0.8, 0.2], 
                  columns=['Put/Call'], 
                  index=['a', 'b', 'c', 'd'])
df3['Ratings'] = ['Sell', 'Buy', 'Hold', 'Buy']<jupyter_output><empty_output><jupyter_text>**Concatenation** is basically making rows added from one DataFrame object to the other target one. This can be achieved through various mean, including append() ir pd.concat() function. These two approaches are determined considering how index values want to be addressed.<jupyter_code># Appending data from df2 to df1 as new rows.
df1.append(df2, sort=False)

# Acheive above goal but ignore indices.
df1.append(df2, ignore_index=True, sort=False)

# Similar to the first append operation above.
pd.concat((df1, df2), sort=False)

# Similar to the second append operation above.
pd.concat((df1, df2), ignore_index=True, sort=False)<jupyter_output><empty_output><jupyter_text>**Joining**<jupyter_code># Default left join, only the index values of df2, df2 are relevant respectively.
df1.join(df3)
df3.join(df1)

# Default left join method
df1.join(df3, how='left')

# Reversing the sequence of Dataframe object, with right df being relevant.
df1.join(df3, how='right')

# Inner join only preserves common index values(found in both df)
df1.join(df3, how='inner')

# Outer join preserves all index values(Think of AND, OR)
df1.join(df3, how='outer')<jupyter_output><empty_output><jupyter_text>**Merging**As opposed to join operation executing based on indices of the DataFrame objects, merge operation takes place on columns shared between the two data sets.<jupyter_code>c = pd.Series(['Overvalued', 'Undervalued', 'Undervalued', 'Fair-valued'], index=['a', 'b', 'c', 'd'])
df1['Valuation'] = c
df3['Valuation'] = c

# By default, merging operation merged based on single shared column
pd.merge(df1, df3)

# Outer merge preserves all data rows.
pd.merge(df1, df3, how='outer')

# Other common merging method
pd.merge(df1, df3, left_index=True, right_index=True)
pd.merge(df1, df3, left_on='Price', right_on='Put/Call', how='outer')<jupyter_output>/usr/local/lib/python3.6/dist-packages/pandas/core/reshape/merge.py:1108: UserWarning: You are merging on int and float columns where the float values are not equal to their int representation
  UserWarning,
{'fraction_non_alphanumeric': 0.12769507186858317, 'fraction_numerical': 0.02104722792607803, 'mean_word_length': 4.957951070336391, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 44, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '609902', 'license': 'Apache-2.0', 'n_tokens_mistral': 2615, 'n_tokens_neox': 2474, 'n_words': 857, 'path': '1. Basic Framework/Pandas_Guide.ipynb', 'repo_name': 'conquerv0/Pynance', 'size': '7792'}