<jupyter_start><jupyter_text>Following Notebook is part of notes made from Data Analysis To get started with pandas, you will need to get comfortable with its two workhorsedata structures: Series and DataFrame. While they are not a universal solution forevery problem, they provide a solid, easy-to-use basis for most applications.  Series A Series is a one-dimensional array-like object containing a sequence of values (ofsimilar types to NumPy types) and an associated array of data labels, called its index.The simplest Series is formed from only an array of data:<jupyter_code>import pandas as pd
import numpy as np
obj = pd.Series([4, 7, -5, 3])
obj
obj.values
obj.index # like range(4)<jupyter_output><empty_output><jupyter_text>Often it will be desirable to create a Series with an index identifying each data pointwith a label:<jupyter_code>obj2 = pd.Series([4, 7, -5, 3], index=['d', 'b', 'a', 'c'])
obj2.index<jupyter_output><empty_output><jupyter_text>Compared with NumPy arrays, you can use labels in the index when selecting singlevalues or a set of values:<jupyter_code>obj2['a']
obj2['d'] = 6
obj2[['c', 'a', 'd']]<jupyter_output><empty_output><jupyter_text>Here ['c', 'a', 'd'] is interpreted as a list of indices, even though it containsstrings instead of integers.Using NumPy functions or NumPy-like operations, such as filtering with a booleanarray, scalar multiplication, or applying math functions, will preserve the index-valuelink:<jupyter_code>obj2[obj2 > 0]
obj2 * 2
np.exp(obj2)<jupyter_output><empty_output><jupyter_text>Another way to think about a Series is as a fixed-length, ordered dict, as it is a map‐ping of index values to data values. It can be used in many contexts where you mightuse a dict:<jupyter_code>'b' in obj2
'e' in obj2<jupyter_output><empty_output><jupyter_text>Should you have data contained in a Python dict, you can create a Series from it bypassing the dict:<jupyter_code>sdata = {'Ohio': 35000, 'Texas': 71000, 'Oregon': 16000, 'Utah': 5000}
obj3 = pd.Series(sdata)
obj3<jupyter_output><empty_output><jupyter_text>When you are only passing a dict, the index in the resulting Series will have the dict’skeys in sorted order. You can override this by passing the dict keys in the order youwant them to appear in the resulting Series:<jupyter_code>states = ['California', 'Ohio', 'Oregon', 'Texas']
obj4 = pd.Series(sdata, index=states)
obj4
pd.isnull(obj4)
pd.notnull(obj4)<jupyter_output><empty_output><jupyter_text>Series also has these as instance methods:<jupyter_code>obj4.isnull()<jupyter_output><empty_output><jupyter_text>A useful Series feature for many applications is that it automatically aligns by indexlabel in arithmetic operations:<jupyter_code>obj3
obj4
obj3 + obj4
obj4.name = 'population'
obj4.index.name = 'state'
obj4<jupyter_output><empty_output><jupyter_text>A Series’s index can be altered in-place by assignment:<jupyter_code>obj
obj.index = ['Bob', 'Steve', 'Jeff', 'Ryan']
obj<jupyter_output><empty_output><jupyter_text>DataFrame A DataFrame represents a rectangular table of data and contains an ordered collec‐tion of columns, each of which can be a different value type (numeric, string,boolean, etc.). The DataFrame has both a row and column index; it can be thought ofas a dict of Series all sharing the same index. Under the hood, the data is stored as oneor more two-dimensional blocks rather than a list, dict, or some other collection ofone-dimensional arrays. The exact details of DataFrame’s internals are outside thescope of this book.<jupyter_code>data = {'state': ['Ohio', 'Ohio', 'Ohio', 'Nevada', 'Nevada', 'Nevada'],
'year': [2000, 2001, 2002, 2001, 2002, 2003],
'pop': [1.5, 1.7, 3.6, 2.4, 2.9, 3.2]}
frame = pd.DataFrame(data)
frame<jupyter_output><empty_output><jupyter_text>For large DataFrames, the head method selects only the first five rows:<jupyter_code>frame.head()<jupyter_output><empty_output><jupyter_text>If you specify a sequence of columns, the DataFrame’s columns will be arranged inthat order:<jupyter_code>pd.DataFrame(data, columns=['year', 'state', 'pop'])<jupyter_output><empty_output><jupyter_text>If you pass a column that isn’t contained in the dict, it will appear with missing valuesin the result:<jupyter_code>frame2 = pd.DataFrame(data, columns=['year', 'state', 'pop', 'debt'],
index=['one', 'two', 'three', 'four','five', 'six'])
frame2
frame2.columns<jupyter_output><empty_output><jupyter_text>A column in a DataFrame can be retrieved as a Series either by dict-like notation orby attribute:<jupyter_code>frame2['state']
frame2.year<jupyter_output><empty_output><jupyter_text>Note that the returned Series have the same index as the DataFrame, and their nameattribute has been appropriately set.Rows can also be retrieved by position or name with the special loc attribute (muchmore on this later):<jupyter_code>frame2.loc['three']<jupyter_output><empty_output><jupyter_text>Columns can be modified by assignment. For example, the empty 'debt' columncould be assigned a scalar value or an array of values:<jupyter_code>frame2['debt'] = 16.5
frame2
frame2['debt'] = np.arange(6.)
frame2<jupyter_output><empty_output><jupyter_text>When you are assigning lists or arrays to a column, the value’s length must match thelength of the DataFrame. If you assign a Series, its labels will be realigned exactly tothe DataFrame’s index, inserting missing values in any holes:<jupyter_code>val = pd.Series([-1.2, -1.5, -1.7], index=['two', 'four', 'five'])
frame2['debt'] = val
frame2<jupyter_output><empty_output><jupyter_text>Assigning a column that doesn’t exist will create a new column. The del keyword willdelete columns as with a dict.As an example of del, I first add a new column of boolean values where the statecolumn equals 'Ohio':<jupyter_code>frame2['eastern'] = frame2.state == 'Ohio'
frame2<jupyter_output><empty_output><jupyter_text>New columns cannot be created with the frame2.eastern syntax. New columns cannot be created with the frame2.eastern syntax.<jupyter_code>del frame2['eastern']
frame2.columns<jupyter_output><empty_output><jupyter_text>The column returned from indexing a DataFrame is a view on the underlying data, not a copy. Thus, any in-place modifications to the Series will be reflected in the DataFrame. The column can be explicitly copied with the Series’s copy method. Another common form of data is a nested dict of dicts:<jupyter_code>pop = {'Nevada': {2001: 2.4, 2002: 2.9}, 'Ohio': {2000: 1.5, 2001: 1.7, 2002: 3.6}}<jupyter_output><empty_output><jupyter_text>If the nested dict is passed to the DataFrame, pandas will interpret the outer dict keysas the columns and the inner keys as the row indices:<jupyter_code>frame3 = pd.DataFrame(pop)
frame3<jupyter_output><empty_output><jupyter_text>You can transpose the DataFrame (swap rows and columns) with similar syntax to aNumPy array:<jupyter_code>frame3.T<jupyter_output><empty_output><jupyter_text>The keys in the inner dicts are combined and sorted to form the index in the result.This isn’t true if an explicit index is specified:<jupyter_code>pd.DataFrame(pop, index=[2001, 2002, 2003])
pdata = {'Ohio': frame3['Ohio'][:-1],
'Nevada': frame3['Nevada'][:2]}
pd.DataFrame(pdata)<jupyter_output><empty_output><jupyter_text>If a DataFrame’s index and columns have their name attributes set, these will also bedisplayed:<jupyter_code>frame3.index.name = 'year'; frame3.columns.name = 'state'
frame3<jupyter_output><empty_output><jupyter_text>As with Series, the values attribute returns the data contained in the DataFrame as atwo-dimensional ndarray:<jupyter_code>frame3.values<jupyter_output><empty_output><jupyter_text>If the DataFrame’s columns are different dtypes, the dtype of the values array will bechosen to accommodate all of the columns:<jupyter_code>frame2.values<jupyter_output><empty_output><jupyter_text>Index Objects pandas’s Index objects are responsible for holding the axis labels and other metadata(like the axis name or names). Any array or other sequence of labels you use whenconstructing a Series or DataFrame is internally converted to an Index:<jupyter_code>obj = pd.Series(range(3), index=['a', 'b', 'c'])
index = obj.index
index
index[1:]<jupyter_output><empty_output><jupyter_text>Index objects are immutable and thus can’t be modified by the user:<jupyter_code>index[1] = 'd'<jupyter_output><empty_output><jupyter_text>Immutability makes it safer to share Index objects among data structures:<jupyter_code>labels = pd.Index(np.arange(3))
labels
obj2 = pd.Series([1.5, -2.5, 0], index=labels)
obj2<jupyter_output><empty_output><jupyter_text>Some users will not often take advantage of the capabilities pro‐vided by indexes, but because some operations will yield resultscontaining indexed data, it’s important to understand how theywork. In addition to being array-like, an Index also behaves like a fixed-size set:<jupyter_code>obj2.index is labels
dup_labels = pd.Index(['foo', 'foo', 'bar', 'bar'])
dup_labels<jupyter_output><empty_output><jupyter_text>Essential Functionality This section will walk you through the fundamental mechanics of interacting with thedata contained in a Series or DataFrame. In the chapters to come, we will delve moredeeply into data analysis and manipulation topics using pandas. This book is notintended to serve as exhaustive documentation for the pandas library; instead, we’llfocus on the most important features, leaving the less common (i.e., more esoteric)things for you to explore on your own.<jupyter_code>obj = pd.Series([4.5, 7.2, -5.3, 3.6], index=['d', 'b', 'a', 'c'])
obj<jupyter_output><empty_output><jupyter_text>Calling reindex on this Series rearranges the data according to the new index, intro‐ducing missing values if any index values were not already present:<jupyter_code>obj2 = obj.reindex(['a', 'b', 'c', 'd', 'e'])
obj2<jupyter_output><empty_output><jupyter_text>For ordered data like time series, it may be desirable to do some interpolation or fill‐ing of values when reindexing. The method option allows us to do this, using amethod such as ffill, which forward-fills the values:<jupyter_code>obj3 = pd.Series(['blue', 'purple', 'yellow'], index=[0, 2, 4])
obj3
obj3.reindex(range(6), method='ffill')<jupyter_output><empty_output><jupyter_text>With DataFrame, reindex can alter either the (row) index, columns, or both. Whenpassed only a sequence, it reindexes the rows in the result:<jupyter_code>frame = pd.DataFrame(np.arange(9).reshape((3, 3)),
index=['a', 'c', 'd'],
columns=['Ohio', 'Texas', 'California'])
frame
frame2 = frame.reindex(['a', 'b', 'c', 'd'])<jupyter_output><empty_output><jupyter_text>The columns can be reindexed with the columns keyword:<jupyter_code>states = ['Texas', 'Utah', 'California']
frame.reindex(columns=states)<jupyter_output><empty_output><jupyter_text>As we’ll explore in more detail, you can reindex more succinctly by label-indexingwith loc, and many users prefer to use it exclusively:<jupyter_code>frame.loc[['a', 'b', 'c', 'd'], states]<jupyter_output>C:\Users\piotr\AppData\Local\Continuum\anaconda3\lib\site-packages\ipykernel_launcher.py:1: FutureWarning: 
Passing list-likes to .loc or [] with any missing label will raise
KeyError in the future, you can use .reindex() as an alternative.

See the documentation here:
https://pandas.pydata.org/pandas-docs/stable/indexing.html#deprecate-loc-reindex-listlike
  """Entry point for launching an IPython kernel.
{'fraction_non_alphanumeric': 0.10321845128474963, 'fraction_numerical': 0.01841620626151013, 'mean_word_length': 6.653691275167785, 'pattern_counts': {'":': 0, '<': 148, '<?xml version=': 0, '>': 149, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '37', 'dir': 'jupyter-structured-clean-dedup', 'id': '401955', 'license': 'MIT', 'n_tokens_mistral': 3748, 'n_tokens_neox': 3572, 'n_words': 1285, 'path': 'Pandas/pandas_notes.ipynb', 'repo_name': 'piszewc/python-deep-learning-data-science', 'size': '11403'}