<jupyter_start><jupyter_text>Using pandas Series allows you to create indexed lists or arrays shown below.<jupyter_code>obj = Series([3,6,9,12])
obj<jupyter_output><empty_output><jupyter_text>A Series kind of looks like a column of a spreadsheet.  Using values method like we did for dictionaries allows all of the same functionality in a Series.<jupyter_code>obj.values
obj.index<jupyter_output><empty_output><jupyter_text>We can also create our own indexes for the lists / arrays along with them as shown below<jupyter_code>ww2_cas = Series([8700000, 4300000,3000000,2100000, 400000],index= ['USSR','Germany','China','Japan','USA'])
ww2_cas<jupyter_output><empty_output><jupyter_text>We can then search by index and by value<jupyter_code>ww2_cas['USA']
#Check for conutries with casualties > 4 mill
ww2_cas[ww2_cas > 4000000]
'USSR' in ww2_cas<jupyter_output><empty_output><jupyter_text>We can easily change a Series to a dictionary as shown below<jupyter_code>ww2_dict = ww2_cas.to_dict()
ww2_dict
ww2_series = Series(ww2_dict)
ww2_series<jupyter_output><empty_output><jupyter_text>And we can create new Series from dictionaries indexed with variables that contain whatever we want as shown below<jupyter_code>countries = ['China', 'Germany', 'Japan']
obj2 = Series(ww2_dict, index=countries)
obj2<jupyter_output><empty_output><jupyter_text>We can also check for null values using pandas methods like isnull or notnull.<jupyter_code>pd.isnull(obj2)
pd.notnull(obj2)
ww2_series + obj2
obj2.name = 'World War 2 Casualties'
obj2
obj2.index.name = 'Countries'
obj2<jupyter_output><empty_output><jupyter_text>Data frames are like a spreadsheet in pandas.  It seems much easier to use than numpy's arrays and matrices, but there aren't any mathematical functions available for it.<jupyter_code>import numpy as np
import pandas as pd
from pandas import Series,DataFrame
import webbrowser
website = 'https://en.wikipedia.org/wiki/NFL_win-loss_records'
webbrowser.open(website)<jupyter_output><empty_output><jupyter_text>With pandas data frames you can literally use python's built in webbrowser function to pull up a webpage to pull data to create your data frame.  You just copy any table to your clipboard and use the pd.read_clipboard() statement to capture.  Below I captured the nfl wins losses just like taking a snippet.  Very cool!.<jupyter_code>nfl_frame = pd.read_clipboard()
nfl_frame<jupyter_output><empty_output><jupyter_text>Then you can pull out different values by using the many .methods available including any of the headings as long as they are one word.<jupyter_code>nfl_frame.columns
nfl_frame.Team<jupyter_output><empty_output><jupyter_text>If there are two words in the heading then use the index method  with the exact name in quotes.  Verify by calling with the columns method first.<jupyter_code>nfl_frame['First NFL Season']<jupyter_output><empty_output><jupyter_text>To call multiple columns use the DataFrame() function and pass the name of the data frame you want to use along with a list of columns you want by name in quotes as shown below.<jupyter_code>DataFrame(nfl_frame, columns=['Team','First NFL Season'])<jupyter_output><empty_output><jupyter_text>Here is how we pull rows.<jupyter_code>nfl_frame.head()# to see what you have and you can pass a number to pull more or fewer rows<jupyter_output><empty_output><jupyter_text>In my class, the instructor uses the .ix method, but this method was deprecated in pandas 0.20.1 in 2016 I think.  I had to use loc or iloc instead.  I found loc to be very similar at least superficially and the exact structure of the instructors code worked with loc instead of ix.<jupyter_code>nfl_frame.loc[3]
nfl_frame['Stadium'] = "Levi's Stadium"
nfl_frame
nfl_frame['Stadium'] = np.arange(5)
nfl_frame<jupyter_output><empty_output><jupyter_text>We can even add a Series as shown below<jupyter_code>stadiums =  Series(["Levi's Stadium", ' AT&T Stadium'], index = [4,0])
stadiums
nfl_frame ['Stadium'] = stadiums
nfl_frame<jupyter_output><empty_output><jupyter_text>We can delete by using del as shown below<jupyter_code>del nfl_frame['Stadium']
nfl_frame<jupyter_output><empty_output><jupyter_text>You can also create data frames by passing in a dictionary<jupyter_code>data = {'City': ['SF', 'LA', 'NYC'], 'Population': [837000, 3880000, 8400000]}
city_frame = DataFrame(data)
city_frame<jupyter_output><empty_output><jupyter_text>index objects -<jupyter_code>my_ser = Series([1,2,3,4], index= ['A', 'B', 'C', 'D'])
my_ser
my_index = my_ser.index
my_index
my_index[3]
my_index[2:]<jupyter_output><empty_output><jupyter_text>Indexes are immutable but you can reindex<jupyter_code>from numpy.random import randn
ser1 = my_ser
ser1
ser2 = ser1.reindex(['A', 'B', 'C', 'D', 'E'])
ser2
ser2.reindex(['A','B','C','D','E','F','G'],fill_value=0)
ser3 = Series(['USA', 'Mexico'], index = [0,5])
ser3<jupyter_output><empty_output><jupyter_text>ffill extends the value to the next index change.<jupyter_code>ser3.reindex(range(15), method ='ffill')  
dframe = DataFrame(randn(4).reshape((2,2)),index=['A','B'],columns=['col1','col2'])
dframe
dframe2 = dframe.reindex(['B','C'])
dframe2<jupyter_output><empty_output><jupyter_text>drop entry<jupyter_code>import numpy as np
from pandas import Series,DataFrame
import pandas as pd
ser1 = Series(np.arange(3), index=['a','b','c'])
ser1
ser1.drop('b')
dframe1= DataFrame(np.arange(9).reshape((3,3)), index=['SF', 'LA', 'NY'])
dframe1
dframe1.drop('LA')
dframe1
dframe1.drop(1, axis=1)# the axis is needed to drop the column and the 1 axis is the first one
import pandas as pd
import numpy as np
from pandas import Series, DataFrame
ser1 = Series(np.arange(3), index=['a','b','c'])
ser1 = ser1*2
ser1
ser1['b']
ser1[1]
ser1[ser1>3]
ser1[ser1>3] = 7
ser1
dframe=DataFrame(np.arange(9).reshape((3,3)), index=['NYC', 'LA', 'SF'],
                columns = ['a','b','c'])
dframe
dframe['b']<jupyter_output><empty_output><jupyter_text>Note that you must use two sets of brackets if calling a list of indexed items (one set of brackets for list and one for index).<jupyter_code>dframe[['b','c']]
dframe[dframe['b']>4]
dframe > 4
dframe.loc['LA']
dframe.iloc[1]<jupyter_output><empty_output><jupyter_text>Data alignment  - arithmatic between data frames with different index values<jupyter_code>ser1
ser2 = Series([2,3,4,5],index=['c','d','e','f'])
ser2
ser1+ser2
dframe1=DataFrame(np.arange(4).reshape((2,2)),columns=list('AB'),index=['NYC', 'LA'])
dframe1
dframe2 = DataFrame(np.arange(9).reshape((3,3)),columns=list('ADC'),index=['NYC', 'SF', 'LA']) 
dframe2
dframe1+dframe2
dframe1.add(dframe2,fill_value=0)<jupyter_output><empty_output><jupyter_text>rank and sort<jupyter_code>ser1
ser1.sort_index()<jupyter_output><empty_output><jupyter_text>The instructor had us use the dot method order and it threw and error on me.  I guess this class is so old that the versions have overtaken the instruction.  I looked it up and the sort_values has replaced .order.  I am still learning at a low cost and I know nothing so anything is valuable.  Still, I can't help but feel I could have found and purchased a better starting point.  I must meditate on this later.  I will continue to learn for now.<jupyter_code>ser1.sort_values()
from numpy.random import randn
ser2 = Series(randn(10))
ser2<jupyter_output><empty_output><jupyter_text>The instructor asked us to use a .sort method that threw me and error.  The sort_value dot method is what I needed for today's version.  NOw I know why this class was so cheap :)<jupyter_code>ser2.sort_values()<jupyter_output><empty_output><jupyter_text>Rank method also did not work as the instructor explained.<jupyter_code>ser2.rank()<jupyter_output><empty_output><jupyter_text>summary statistics - numpy dataframe statistical summaries<jupyter_code>arr = np.array([[1,2,np.nan],[np.nan,3,4]])
dframe1 = DataFrame(arr,index=['A','B'],columns=['One', 'Two', 'Three'])
dframe1
dframe1.sum()
dframe1.sum(axis=1)
dframe1
dframe1.min()
dframe1
dframe1.idxmin()
dframe1.cumsum()
dframe1.describe()
from IPython.display import YouTubeVideo
YouTubeVideo('xGbpuFNR1ME')
YouTubeVideo('4EXNedimDMs')
from pandas_datareader import data as pdweb
import datetime

prices = pdweb.get_data_yahoo(['CVX','XOM','BP'],start=datetime.datetime(2010,1,1),end=datetime.datetime(2013,1,1))['Adj Close']
prices.head()
rets = prices.pct_change()
corr = rets.corr
%matplotlib inline
prices.plot()
import seaborn as sns
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Another case in which the instructor used sns.corrplot(rets,annot=False,diag_names = False) and this was depricated long ago in favor of heatmap.  Heatmap does not take all of the same arguments and it took me a bit to find the info.  I've learned that heatmaps rated poorly on a study about human understanding of graphs.  Scatterplots and bar charts were the best.<jupyter_code>sns.heatmap(rets,annot=False)
ser7 = Series(['w','x','g','h','x','f','r','e','q','g','w'])
ser7

ser7.unique()
ser7.value_counts()<jupyter_output><empty_output><jupyter_text>Missing data - how to<jupyter_code>import numpy as np
from pandas import Series, DataFrame
import pandas as pd
data=Series(['one','two',np.nan,'four'])
data<jupyter_output><empty_output><jupyter_text>We can see if we have nulls in our data by using .isnull to return a boolean as shown below.<jupyter_code>data.isnull()<jupyter_output><empty_output><jupyter_text>We can also use .dropna() to remove the null in our Series<jupyter_code>data.dropna()
dframe = DataFrame([[1,2,3],[np.nan, 5,6],[7,np.nan,9],[np.nan,np.nan,np.nan]])
dframe<jupyter_output><empty_output><jupyter_text>In a dataframe, .dropna() will remove any row that has any null in it.  So even if you have a single null, you lose all the rest of your data in that row.<jupyter_code>clean_dframe = dframe.dropna()
clean_dframe<jupyter_output><empty_output><jupyter_text>We can change this behavior by specifying with "how" as shown below.  Note that only row 3 that had all nulls was removed.<jupyter_code>clean_dframe = dframe.dropna(how='all')
clean_dframe<jupyter_output><empty_output><jupyter_text>If we want to drop columns, just include the axis as we've done in previous classes.<jupyter_code>dframe.dropna(axis=1)
dframe7 = DataFrame([[1,2,3,np.nan],[2,np.nan,5,6],[np.nan,7,np.nan,9],[1,np.nan,np.nan,np.nan]])

dframe7<jupyter_output><empty_output><jupyter_text>We can set a threshold for data points with the thresh = pass through in the .dropna method.  In the example below, we set our threshold for 2 which means a row must have at least two valid datapoints in order to stay.  Subsequently, row three was deleted.<jupyter_code>dframe7.dropna(thresh=2)
dframe7.dropna(thresh=3)
dframe7<jupyter_output><empty_output><jupyter_text>As usual .fillna() fills na values with whatever you put in parenthesis.<jupyter_code>dframe7.fillna(1)<jupyter_output><empty_output><jupyter_text>Unless you assign a new variable name, it won't permanently change.<jupyter_code>dframe7<jupyter_output><empty_output><jupyter_text>You can also change nulls in specific columns by passing a dictionary in fillna().  The key represents the column number and the value is what will be swapped with the null<jupyter_code>dframe7.fillna({0:1,1:4,2:5,3:6})<jupyter_output><empty_output><jupyter_text>If you want a change to be permanent with fillna() you can pass the inplace=True statement.  This is permanent.<jupyter_code>dframe7
dframe7.fillna(1,inplace=True)
dframe7<jupyter_output><empty_output><jupyter_text>No more nulls. index hierarchy<jupyter_code>import numpy as np
from pandas import Series, DataFrame
import pandas as pd
from numpy.random import randn
ser = Series(randn(6),index = [[1,1,1,2,2,2,],['a','b','c','a','b','c']])
ser
ser.index
ser[1]
ser[2]
ser[:,'a']
dframe = ser.unstack()
dframe
dframe2= DataFrame(np.arange(16).reshape(4,4),index=[['a','a','b','b'],[1,2,1,2]],
                   columns=[['NYC','NYC','LA','SF'],['cold','hot','hot','cold']])
dframe2
dframe2.index.names=['Index_1','Index_2']
dframe2.columns.names=['Cities','Temp']
dframe2
dframe2.swaplevel('Cities','Temp',axis=1)<jupyter_output><empty_output><jupyter_text>INstructor did .sortlevel that has been deprecated.  .sortindex() or .sortvalue() are current options<jupyter_code>dframe2.sort_index()
dframe2.sort_values(by=['Index_2'])
dframe2.sum(level='Temp',axis=1)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12304214435653157, 'fraction_numerical': 0.024786048764734378, 'mean_word_length': 7.197882197220384, 'pattern_counts': {'":': 0, '<': 169, '<?xml version=': 0, '>': 175, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '42', 'dir': 'jupyter-structured-clean-dedup', 'id': '613715', 'license': 'Unlicense', 'n_tokens_mistral': 4492, 'n_tokens_neox': 4329, 'n_words': 1206, 'path': 'Arrays notes II Pandas.ipynb', 'repo_name': 'Luzyviento/Class_Notebook', 'size': '12386'}