<jupyter_start><jupyter_text>Pandas From Head() To Tail()  Customarily, we import as follows:<jupyter_code>import numpy as np

import pandas as pd<jupyter_output><empty_output><jupyter_text>Here is a basic tenet to keep in mind: **data alignment is intrinsic.** The link between labels and data will not be broken unless done so explicitly by you.We’ll give a brief intro to the data structures, then consider all of the broad categories of functionality and methods in separate sections.  Series`Series` is a one-dimensional labeled array capable of holding any data type (integers, strings, floating point numbers, Python objects, etc.). The axis labels are collectively referred to as the **index**. The basic method to create a Series is to call:<jupyter_code>>>> s = pd.Series(data, index=index)<jupyter_output><empty_output><jupyter_text>Here, data can be many different things:* a Python dict* an ndarray* a scalar value (like 5)The passed index is a list of axis labels.  Creating a Series by passing a list of values, letting pandas create a default integer index:<jupyter_code>s = pd.Series([1, 3, 5, np.nan, 6, 8])

s<jupyter_output><empty_output><jupyter_text>DataFrame`DataFrame` is a 2-dimensional labeled data structure with columns of potentially different types. You can think of it like a spreadsheet or SQL table, or a dict of Series objects. It is generally the most commonly used pandas object. Like Series, DataFrame accepts many different kinds of input:Dict of 1D ndarrays, lists, dicts, or Series* 2-D numpy.ndarray* Structured or record ndarray* A Series* Another DataFrameAlong with the data, you can optionally pass index (row labels) and columns (column labels) arguments. If you pass an index and / or columns, you are guaranteeing the index and / or columns of the resulting DataFrame. Thus, a dict of Series plus a specific index will discard all data not matching up to the passed index.If axis labels are not passed, they will be constructed from the input data based on common sense rules.Creating a DataFrame by passing a NumPy array, with a datetime index and labeled columns:<jupyter_code>dates = pd.date_range('20210101', periods=6,)

print(dates)
list('ABCD')
np.random.randn(6, 4)
df = pd.DataFrame(np.random.randn(6, 4), index=dates, columns=list('ABCD'))
df<jupyter_output><empty_output><jupyter_text>Creating a `DataFrame` by passing a `dict` of objects that can be converted to series-like.<jupyter_code>df2 = pd.DataFrame({'A': 1.,
                    'B': pd.Timestamp('20130102'),
                    'C': pd.Series(1, index=list(range(4)), dtype='float32'),
                    'D': np.array([3] * 4, dtype='int32'),
                    'E': pd.Categorical(["test", "train", "test", "train"]),
                    'F': 'foo'})
df2<jupyter_output><empty_output><jupyter_text>The columns of the resulting `DataFrame` have different `dtypes`.<jupyter_code>df2.dtypes<jupyter_output><empty_output><jupyter_text>If you’re using IPython, tab completion for column names (as well as public attributes) is automatically enabled. Here’s a subset of the attributes that will be completed:<jupyter_code>df2.<TAB>  # noqa: E225, E999
df2.A                  df2.bool
df2.abs                df2.boxplot
df2.add                df2.C
df2.add_prefix         df2.clip
df2.add_suffix         df2.columns
df2.align              df2.copy
df2.all                df2.count
df2.any                df2.combine
df2.append             df2.D
df2.apply              df2.describe
df2.applymap           df2.diff
df2.B                  df2.duplicated<jupyter_output><empty_output><jupyter_text>> As you can see, the columns A, B, C, and D are automatically tab completed. E and F are there as well; the rest of the attributes have been truncated for brevity.  Viewing dataHere is how to view the top and bottom rows of the frame:<jupyter_code>df = pd.DataFrame({'A': 1.,
                    'B': pd.Timestamp('20130102'),
                    'C': pd.Series(1, index=list(range(20)), dtype='float32'),
                    'D': np.array([3,1,2,5] * 5, dtype='int32'),
                    'E': pd.Categorical(["test", "train", "test", "train"]*5),
                    'F': 'foo'})
df
df.head()
df.tail()<jupyter_output><empty_output><jupyter_text>Display the `index`, `columns`:<jupyter_code>df.index
df.columns<jupyter_output><empty_output><jupyter_text>`DataFrame.to_numpy()` gives a NumPy representation of the underlying data. Note that this can be an expensive operation when your DataFrame has columns with different data types, which comes down to a fundamental difference between pandas and NumPy: **NumPy arrays have one dtype for the entire array, while pandas DataFrames have one dtype per column.** When you call `DataFrame.to_numpy()`, pandas will find the NumPy dtype that can hold all of the dtypes in the DataFrame. This may end up being object, which requires casting every value to a Python object. > For df, our DataFrame of all floating-point values, DataFrame.to_numpy() is fast and doesn’t require copying data.<jupyter_code>df = pd.DataFrame(np.random.randn(6, 4), index=dates, columns=list('ABCD'))
df
df.to_numpy()<jupyter_output><empty_output><jupyter_text>> For df2, the DataFrame with multiple dtypes, DataFrame.to_numpy() is relatively expensive.<jupyter_code>df2 = pd.DataFrame({'A': 1.,
                    'B': pd.Timestamp('20130102'),
                    'C': pd.Series(1, index=list(range(4)), dtype='float32'),
                    'D': np.array([3] * 4, dtype='int32'),
                    'E': pd.Categorical(["test", "train", "test", "train"]),
                    'F': 'foo'})
df2
df2.to_numpy()<jupyter_output><empty_output><jupyter_text>> DataFrame.to_numpy() does not include the index or column labels in the output. `describe()` shows a quick statistic summary of your data:<jupyter_code>df.describe()<jupyter_output><empty_output><jupyter_text>Transposing your data:<jupyter_code>df.T<jupyter_output><empty_output><jupyter_text>Sorting by an axis:<jupyter_code>df.sort_index(axis=1, ascending=False)<jupyter_output><empty_output><jupyter_text>Sorting by values:<jupyter_code>df.sort_values(by=['A'])<jupyter_output><empty_output><jupyter_text>Selection ---**Note:**  While standard Python / Numpy expressions for selecting and setting are intuitive and come in handy for interactive work, for production code, we recommend the optimized pandas data access methods, `.at`, `.iat`, `.loc` and `.iloc`.---See the indexing documentation [Indexing and Selecting Data](https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.htmlindexing) and [MultiIndex / Advanced Indexing](https://pandas.pydata.org/pandas-docs/stable/user_guide/advanced.htmladvanced).<jupyter_code>df = pd.DataFrame(np.random.randn(6, 4), index=dates, columns=list('ABCD'))
df<jupyter_output><empty_output><jupyter_text>GettingSelecting a single column, which yields a Series, equivalent to `df.A`:<jupyter_code>df.A
df['A']<jupyter_output><empty_output><jupyter_text>Selecting via [], which slices the **rows**.<jupyter_code>df[0:3]<jupyter_output><empty_output><jupyter_text>Selection by labelSee more in [Selection by Label](https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.htmlindexing-label).For getting a cross section using a label:<jupyter_code>dates[0]
df.loc[dates[0]]<jupyter_output><empty_output><jupyter_text>Selecting on a multi-axis by label:<jupyter_code>df.loc[:, ['A', 'B']]<jupyter_output><empty_output><jupyter_text>Showing label slicing, both endpoints are included:<jupyter_code>df.loc['20210102':'20210104', ['A', 'B']]<jupyter_output><empty_output><jupyter_text>Reduction in the dimensions of the returned object:<jupyter_code>df.loc['20210102', ['A', 'B']]<jupyter_output><empty_output><jupyter_text>For getting a scalar value:<jupyter_code>df.loc[dates[0], 'A']<jupyter_output><empty_output><jupyter_text>For getting fast access to a scalar (equivalent to the prior method):<jupyter_code>df.at[dates[0], 'A']<jupyter_output><empty_output><jupyter_text>Selection by positionSee more in [Selection by Position](https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.htmlindexing-integer).Select via the position of the passed integers:<jupyter_code>df.iloc[3]<jupyter_output><empty_output><jupyter_text>By integer slices, acting similar to numpy/python:<jupyter_code>df.iloc[3:5, 0:2]<jupyter_output><empty_output><jupyter_text>By lists of integer position locations, similar to the numpy/python style:<jupyter_code>df.iloc[[1, 2, 4], [0, 2]]<jupyter_output><empty_output><jupyter_text>For slicing rows explicitly:<jupyter_code>df.iloc[1:3, :]<jupyter_output><empty_output><jupyter_text>For slicing columns explicitly:<jupyter_code>df.iloc[:, 1:3]<jupyter_output><empty_output><jupyter_text>For getting a value explicitly:<jupyter_code>df.iloc[1, 1]<jupyter_output><empty_output><jupyter_text>For getting fast access to a scalar (equivalent to the prior method):<jupyter_code>df.iat[1, 1]<jupyter_output><empty_output><jupyter_text>Boolean indexingUsing a single column’s values to select data.<jupyter_code>df[df['A'] > 0]<jupyter_output><empty_output><jupyter_text>Selecting values from a DataFrame where a boolean condition is met.<jupyter_code>df
df[df > 0]<jupyter_output><empty_output><jupyter_text>Using the `isin()` method for filtering:<jupyter_code>df2 = df.copy()
df2['E'] = ['one', 'one', 'two', 'three', 'four', 'three']
df2
df2[df2['E'].isin(['two', 'four'])]<jupyter_output><empty_output><jupyter_text>SettingSetting a new column automatically aligns the data by the indexes.<jupyter_code>df = pd.DataFrame(np.random.randn(6, 4), index=dates, columns=list('ABCD'))
df
s1 = pd.Series([1, 2, 3, 4, 5, 6], index=pd.date_range('20210102', periods=6))
s1
df['F'] = s1
df<jupyter_output><empty_output><jupyter_text>Setting values by label:<jupyter_code>df.at[dates[0], 'A'] = 0
df<jupyter_output><empty_output><jupyter_text>Setting values by position:<jupyter_code>df.iat[0, 1] = 0
df<jupyter_output><empty_output><jupyter_text>Setting by assigning with a NumPy array:<jupyter_code>df.loc[:, 'D'] = np.array([5] * len(df))
df<jupyter_output><empty_output><jupyter_text>The result of the prior setting operations. A `where` operation with setting.<jupyter_code>df2 = df.copy()
df2[df2 > 0] = -df2
df2<jupyter_output><empty_output><jupyter_text>Missing datapandas primarily uses the value `np.nan` to represent missing data. It is by default not included in computations. See the [Missing Data section](https://pandas.pydata.org/pandas-docs/stable/user_guide/missing_data.htmlmissing-data).Reindexing allows you to change/add/delete the index on a specified axis. This returns a copy of the data.<jupyter_code>df = pd.DataFrame(np.random.randn(6, 4), index=dates, columns=list('ABCD'))
df
df1 = df.reindex(index=dates[0:4], columns=list(df.columns) + ['E'])
df1
df1.loc[dates[0]:dates[1], 'E'] = 1
df1<jupyter_output><empty_output><jupyter_text>To drop any rows that have missing data.<jupyter_code>df1.dropna(how='any')<jupyter_output><empty_output><jupyter_text>Filling missing data.<jupyter_code>df1.fillna(value=5)<jupyter_output><empty_output><jupyter_text>To get the boolean mask where values are nan.<jupyter_code>pd.isna(df1)<jupyter_output><empty_output><jupyter_text>OperationsSee the Basic section on [Binary Ops](https://pandas.pydata.org/pandas-docs/stable/user_guide/basics.htmlbasics-binop).<jupyter_code>df = pd.DataFrame(np.random.randn(6, 4), index=dates, columns=list('ABCD'))
df<jupyter_output><empty_output><jupyter_text>StatsOperations in general exclude missing data.Performing a descriptive statistic:<jupyter_code>df.mean()<jupyter_output><empty_output><jupyter_text>Same operation on the other axis:<jupyter_code>df.mean(axis=1)<jupyter_output><empty_output><jupyter_text>Operating with objects that have different dimensionality and need alignment. In addition, pandas automatically broadcasts along the specified dimension.<jupyter_code>s = pd.Series([1, 3, 5, np.nan, 6, 8], index=dates).shift(2)
s
df.sub(s, axis='index')<jupyter_output><empty_output><jupyter_text>ApplyApplying functions to the data:<jupyter_code>df.apply(np.cumsum)
df.apply(lambda x: x.max() - x.min())<jupyter_output><empty_output><jupyter_text>HistogrammingSee more at [Histogramming and Discretization](https://pandas.pydata.org/pandas-docs/stable/user_guide/basics.htmlbasics-discretization).<jupyter_code>s = pd.Series(np.random.randint(0, 7, size=10))
s
s.value_counts()<jupyter_output><empty_output><jupyter_text>String MethodsSeries is equipped with a set of string processing methods in the str attribute that make it easy to operate on each element of the array, as in the code snippet below. Note that pattern-matching in str generally uses [regular expressions](https://docs.python.org/3/library/re.html) by default (and in some cases always uses them). See more at [Vectorized String Methods](https://pandas.pydata.org/pandas-docs/stable/user_guide/text.htmltext-string-methods).<jupyter_code>s = pd.Series(['A', 'B', 'C', 'Aaba', 'Baca', np.nan, 'CABA', 'dog', 'cat'])
s
s.str.lower()<jupyter_output><empty_output><jupyter_text>Merge Concatpandas provides various facilities for easily combining together Series and DataFrame objects with various kinds of set logic for the indexes and relational algebra functionality in the case of join / merge-type operations.[See the Merging section](https://pandas.pydata.org/pandas-docs/stable/user_guide/merging.htmlmerging).Concatenating pandas objects together with `concat()`:<jupyter_code>df = pd.DataFrame(np.random.randn(10, 4))
df
# break it into pieces
pieces = [df[:3], df[3:7], df[7:]]
pieces
pd.concat(pieces)<jupyter_output><empty_output><jupyter_text>JoinSQL style merges. See the [Database style joining](https://pandas.pydata.org/pandas-docs/stable/user_guide/merging.htmlmerging-join) section.<jupyter_code>left = pd.DataFrame({'key': ['foo', 'foo'], 'lval': [1, 2]})
left
right = pd.DataFrame({'key': ['foo', 'foo'], 'rval': [4, 5]})
right
 pd.merge(left, right, on='key')<jupyter_output><empty_output><jupyter_text>Another example that can be given is:<jupyter_code>left = pd.DataFrame({'key': ['foo', 'bar'], 'lval': [1, 2]})
left
right = pd.DataFrame({'key': ['foo', 'bar'], 'rval': [4, 5]})
right
pd.merge(left, right, on='key')<jupyter_output><empty_output><jupyter_text>GroupingBy “group by” we are referring to a process involving one or more of the following steps:* Splitting the data into groups based on some criteria* Applying a function to each group independently* Combining the results into a data structureSee the [Grouping section](https://pandas.pydata.org/pandas-docs/stable/user_guide/groupby.htmlgroupby).<jupyter_code>df = pd.DataFrame({'A': ['foo', 'bar', 'foo', 'bar','foo', 'bar', 'foo', 'foo'],
                  'B': ['one', 'one', 'two', 'three','two', 'two', 'one', 'three'],
                  'C': np.random.randn(8),
                  'D': np.random.randn(8)}) 
df<jupyter_output><empty_output><jupyter_text>Grouping and then applying the `sum()` function to the resulting groups.<jupyter_code>df.groupby('A').sum()<jupyter_output><empty_output><jupyter_text>Grouping by multiple columns forms a hierarchical index, and again we can apply the `sum()` function.<jupyter_code>df.groupby(['A', 'B']).sum()<jupyter_output><empty_output><jupyter_text>ReshapingSee the sections on [Hierarchical Indexing](https://pandas.pydata.org/pandas-docs/stable/user_guide/advanced.htmladvanced-hierarchical) and [Reshaping](https://pandas.pydata.org/pandas-docs/stable/user_guide/reshaping.htmlreshaping-stacking). Stack<jupyter_code>tuples = list(zip(*[['bar', 'bar', 'baz', 'baz','foo', 'foo', 'qux', 'qux'],
                    ['one', 'two', 'one', 'two','one', 'two', 'one', 'two']]))
tuples
index = pd.MultiIndex.from_tuples(tuples, names=['first', 'second'])
index
df = pd.DataFrame(np.random.randn(8, 2), index=index, columns=['A', 'B'])
df
df2 = df[:4]
df2<jupyter_output><empty_output><jupyter_text>The `stack()` method “compresses” a level in the DataFrame’s columns.<jupyter_code>stacked = df2.stack()
stacked<jupyter_output><empty_output><jupyter_text>With a “stacked” DataFrame or Series (having a MultiIndex as the index), the inverse operation of `stack()` is `unstack()`, which by default unstacks the **last level**:<jupyter_code>stacked.unstack()
stacked.unstack(1)
stacked.unstack(0)<jupyter_output><empty_output><jupyter_text>Pivot tablesSee the section on [Pivot Tables](https://pandas.pydata.org/pandas-docs/stable/user_guide/reshaping.htmlreshaping-pivot).<jupyter_code>df = pd.DataFrame({'A': ['one', 'one', 'two', 'three'] * 3,
                   'B': ['A', 'B', 'C'] * 4,
                   'C': ['foo', 'foo', 'foo', 'bar', 'bar', 'bar'] * 2,
                   'D': np.random.randn(12),'E': np.random.randn(12)})
df<jupyter_output><empty_output><jupyter_text>We can produce pivot tables from this data very easily:<jupyter_code>pd.pivot_table(df, values='D', index=['A', 'B'], columns=['C'])<jupyter_output><empty_output><jupyter_text>Time seriespandas has simple, powerful, and efficient functionality for performing resampling operations during frequency conversion (e.g., converting secondly data into 5-minutely data). This is extremely common in, but not limited to, financial applications. See the [Time Series section](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.htmltimeseries).<jupyter_code>rng = pd.date_range('1/1/2012', periods=100, freq='S')
rng
ts = pd.Series(np.random.randint(0, 500, len(rng)), index=rng)
ts
ts.resample('5Min').sum()<jupyter_output><empty_output><jupyter_text>Time zone representation:<jupyter_code>rng = pd.date_range('3/6/2012 00:00', periods=5, freq='D')
rng
ts = pd.Series(np.random.randn(len(rng)), rng)
ts
ts_utc = ts.tz_localize('UTC')
ts_utc<jupyter_output><empty_output><jupyter_text>Converting to another time zone:<jupyter_code>ts_utc.tz_convert('US/Eastern')<jupyter_output><empty_output><jupyter_text>Converting between time span representations:<jupyter_code>rng = pd.date_range('1/1/2012', periods=5, freq='M')
rng
ts = pd.Series(np.random.randn(len(rng)), index=rng)
ts
ps = ts.to_period()
ps
ps.to_timestamp()<jupyter_output><empty_output><jupyter_text>Converting between period and timestamp enables some convenient arithmetic functions to be used. In the following example, we convert a quarterly frequency with year ending in November to 9am of the end of the month following the quarter end:<jupyter_code>prng = pd.period_range('1990Q1', '2000Q4', freq='Q-NOV')
prng
ts = pd.Series(np.random.randn(len(prng)), prng)
ts
ts.index = (prng.asfreq('M', 'e') + 1).asfreq('H', 's') + 9
ts.head()<jupyter_output><empty_output><jupyter_text>Categoricalspandas can include categorical data in a `DataFrame`. For full docs, see the [categorical introduction](https://pandas.pydata.org/pandas-docs/stable/user_guide/categorical.htmlcategorical) and the [API documentation](https://pandas.pydata.org/pandas-docs/stable/reference/arrays.htmlapi-arrays-categorical).<jupyter_code>df = pd.DataFrame({"id": [1, 2, 3, 4, 5, 6],
                   "raw_grade": ['a', 'b', 'b', 'a', 'a', 'e']})
df<jupyter_output><empty_output><jupyter_text>Convert the raw grades to a categorical data type.<jupyter_code>df["grade"] = df["raw_grade"].astype("category")
df["grade"]<jupyter_output><empty_output><jupyter_text>Rename the categories to more meaningful names (assigning to `Series.cat.categories()` is in place!).<jupyter_code>df["grade"].cat.categories = ["very good", "good", "very bad"]
df["grade"]<jupyter_output><empty_output><jupyter_text>Reorder the categories and simultaneously add the missing categories (methods under `Series.cat()` return a new Series by default).<jupyter_code>df["grade"] = df["grade"].cat.set_categories(["very bad", "bad", "medium","good", "very good"])
df["grade"]<jupyter_output><empty_output><jupyter_text>Sorting is per order in the categories, not lexical order.<jupyter_code>df.sort_values(by="grade")<jupyter_output><empty_output><jupyter_text>Grouping by a categorical column also shows empty categories.<jupyter_code>df.groupby("grade").size()<jupyter_output><empty_output><jupyter_text>Getting data in/out CSV[Writing to a csv file](https://pandas.pydata.org/pandas-docs/stable/user_guide/io.htmlio-store-in-csv).<jupyter_code>df.to_csv('foo.csv')<jupyter_output><empty_output><jupyter_text>[Reading from a csv file](https://pandas.pydata.org/pandas-docs/stable/user_guide/io.htmlio-read-csv-table).<jupyter_code>df1 = pd.read_csv('foo.csv')
df1<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1382058509087375, 'fraction_numerical': 0.01569637476917096, 'mean_word_length': 5.971205962059621, 'pattern_counts': {'":': 2, '<': 298, '<?xml version=': 0, '>': 308, 'https://': 20, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '74', 'dir': 'jupyter-structured-clean-dedup', 'id': '290091', 'license': 'MIT', 'n_tokens_mistral': 7272, 'n_tokens_neox': 6967, 'n_words': 1712, 'path': 'Week05/Pandas.ipynb', 'repo_name': 'njiix/py4ds', 'size': '20578'}