<jupyter_start><jupyter_text>The pandas Series and DataFrame<jupyter_code># import pandas
import pandas as pd
# and NumPy
import numpy as np

# Set some Pandas options
pd.set_option('display.notebook_repr_html', False)
pd.set_option('display.max_columns', 8)
pd.set_option('display.max_rows', 8)<jupyter_output><empty_output><jupyter_text>Creating a Series and accessing elements<jupyter_code># create a Series from a NumPy array of random values
np.random.seed(1)
s = pd.Series(np.random.randn(100))
s
# select item with matching label of 2
s[2]
# selected elements at positions 2, 5, and 20
s[[2, 5, 20]]
# slice the Series
s[3:8]
s.head()
s.tail()
s.index
s.values
# specify an index at creation time
s2 = pd.Series([1, 2, 3, 4], index=['a', 'b', 'c', 'd'])
s2
# create a Sereis from a Python dict
s2 = pd.Series({'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5})
s2<jupyter_output><empty_output><jupyter_text>Size, shape, uniqueness and counts of values<jupyter_code># create a series
s = pd.Series([5, 0, 1, 1, 2, 3, 4, 5, 6, np.nan])
len(s)
# reports the shape, which is a tuple with len in the first value
s.shape
# .count() is the number of non NaN values
s.count()
# all unique values
s.unique()
# all unique values and their counts
s.value_counts()<jupyter_output><empty_output><jupyter_text>Alignment via index labels<jupyter_code>s3 = pd.Series([1, 2, 3, 4], index=['a', 'b', 'c', 'd'])
s3
s4 = pd.Series([4, 3, 2, 1], index=['d', 'c', 'b', 'a'])
s4
# add s3 and s4
s3 + s4
# see how different from adding numpy arrays
a1 = np.array([1, 2, 3, 4])
a2 = np.array([4, 3, 2, 1])
a1 + a2<jupyter_output><empty_output><jupyter_text>Creating a DataFrame<jupyter_code># create a DataFrame from a 2-d ndarray
pd.DataFrame(np.array([[10, 11], [20, 21]]))
# create a DataFrame for a list of Series objects
df1 = pd.DataFrame([pd.Series(np.arange(10, 15)), 
                    pd.Series(np.arange(15, 20))])
df1
# what's the shape of this DataFrame
df1.shape  # it is two rows by 5 columns
# specify column names
df = pd.DataFrame(np.array([[10, 11], [20, 21]]), 
                  columns=['a', 'b'])
df
# what are names of the columns?
df.columns
df.columns = ['c1', 'c2']
df
# create a DataFrame with named columns and rows
df = pd.DataFrame(np.array([[0, 1], [2, 3]]), 
                  columns=['c1', 'c2'], 
                  index=['r1', 'r2'])
df
# retrieve the index of the DataFrame
df.index
df.values
# create a DataFrame with two Series objects
# and a dictionary
s1 = pd.Series(np.arange(1, 6, 1))
s2 = pd.Series(np.arange(6, 11, 1))
pd.DataFrame({'c1': s1, 'c2': s2})
# demonstrate alignment during creation
s3 = pd.Series(np.arange(12, 14), index=[1, 2])
pd.DataFrame({'c1': s1, 'c2': s2, 'c3': s3})<jupyter_output><empty_output><jupyter_text>Example data<jupyter_code># read in the data and print the first five rows
# use the Symbol column as the index, and 
# only read in columns in positions 0, 2, 3, 7
sp500 = pd.read_csv("sp500.csv", 
                    index_col='Symbol', 
                    usecols=[0, 2, 3, 7])
# peek at the first 5 rows of the data using .head()
sp500.head()
# examine the index
sp500.index<jupyter_output><empty_output><jupyter_text>Selecting columns of a DataFrame<jupyter_code># get first and second columns (1 and 2) by location
sp500[[1, 2]].head(3)
# just the price column
sp500[[1]].head(3)
# get price column by name
# result is a Series
sp500['Price']
# get Price and Sector columns
# since a list is passed, result is a DataFrame
sp500[['Price', 'Sector']]
# attribute access of column by name
sp500.Price<jupyter_output><empty_output><jupyter_text>Selecting rows of a DataFrame   Slicing using the [] operator<jupyter_code># first five rows
sp500[:3]
# XYL through YUM labels
sp500['XYL':'YUM']<jupyter_output><empty_output><jupyter_text>Selecting rows by index label and location: .loc[] and .iloc[]<jupyter_code># get row with label MMM
# returned as a Series
sp500.loc['MMM']
# rows with label MMM and MSFT
# this is a DataFrame result
sp500.loc[['MMM', 'MSFT']]
# get rows in location 0 and 2
sp500.iloc[[0, 2]]
# get the location of MMM and A in the index
i1 = sp500.index.get_loc('MMM')
i2 = sp500.index.get_loc('A')
i1, i2
# and get the rows
sp500.iloc[[i1, i2]]<jupyter_output><empty_output><jupyter_text>Selecting rows by index label and/or location: .ix[]<jupyter_code># by label
sp500.ix[['MSFT', 'ZTS']]
# by location
sp500.ix[[10, 200, 450]]<jupyter_output><empty_output><jupyter_text>Scalar lookup by label or location using .at[] and .iat[]<jupyter_code># by label in both the index and column
sp500.at['MMM', 'Price']
# by location.  Row 0, column 1
sp500.iat[0, 1]<jupyter_output><empty_output><jupyter_text>Selecting rows using Boolean selection<jupyter_code># what rows have a price < 100?
sp500.Price < 100
# now get the rows with Price < 100
sp500[sp500.Price < 100]
# get only the Price where Price is < 10 and > 0
sp500[(sp500.Price < 10) & (sp500.Price > 0)] [['Price']]<jupyter_output><empty_output><jupyter_text>Arithmetic on a DataFrame<jupyter_code># set the seed to allow replicatable results
np.random.seed(123456)
# create the DataFrame
df = pd.DataFrame(np.random.randn(5, 4), 
                  columns=['A', 'B', 'C', 'D'])
df
# multiply everything by 2
df * 2
df - df.iloc[0]
# get rows 1 through three, and only B, C columns
subframe = df[1:4][['B', 'C']]
# we have extracted a little square in the middle of df
subframe
# demonstrate the alignment of the subtraction
df - subframe
# get the A column
a_col = df['A']
df.sub(a_col, axis=0)<jupyter_output><empty_output><jupyter_text>Reindexing Series and DataFrame objects<jupyter_code># create a series of five random numbers
np.random.seed(1)
s = pd.Series(np.random.randn(5))
s
# now set the index to alpha values
s.index = ['a', 'b', 'c', 'd', 'e']
s
# reindex the copy
s2 = s.reindex(['a', 'c', 'e', 'g'])
# change the value at 'a'
s2['a'] = 0
s2
s['a']
# Series objects with string and integer index types
# with the "same" values will not align
s1 = pd.Series([0, 1, 2], index=[0, 1, 2])
s2 = pd.Series([3, 4, 5], index=['0', '1', '2'])
s1 + s2
# demonstrate treating values at a specific type
s2.index = s2.index.values.astype(int)
s1 + s2
# show reindexing with filling of NaN with a specified value
s2 = s.copy()
s2.reindex(['a', 'f'], fill_value=0)
# a Series to demonstrate reindexing
s3 = pd.Series(['red', 'green', 'blue'], index=[0, 3, 5])
s3
# forward fill (last known value technique)
s3.reindex(np.arange(0,7), method='ffill')
# demonstrate how backwards fill differs
s3.reindex(np.arange(0,7), method='bfill')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14801334546557476, 'fraction_numerical': 0.04943888383378829, 'mean_word_length': 4.641573994867408, 'pattern_counts': {'":': 0, '<': 63, '<?xml version=': 0, '>': 59, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '456459', 'license': 'MIT', 'n_tokens_mistral': 2663, 'n_tokens_neox': 2444, 'n_words': 804, 'path': 'Mastering Pandas for Finance_Code/Chapter02/.ipynb_checkpoints/02_The_pandas_Series_and_DataFrame-checkpoint.ipynb', 'repo_name': 'renc/Mastering-Python-for-Finance-source-codes', 'size': '6594'}