<jupyter_start><jupyter_text>We will start with the practice of basic operations in Pandas. It is very important to get familiar with this stuff because we will be using it again and again throughout this course. We will cover an introduction to Pandas, specifically:* Attributes of Pandas objects* Counting values in Series* Altering labels* .dt and .str accessors* Sorting Pandas Basics IOne of the great things about the frequently used Python packages is that their documentation is really good. We can usually easily google anything we want to do in Pandas. We will also be working intensively with the official documentation throughout this module and the course. Attributes of Pandas objects ```Pandas``` objects have a number of attributes enabling us to access metadata:* Shape: gives the axis dimensions of the object, consistent with ndarray* Axis labels:    * Series: index (only axis)    * DataFrame: index (rows) and columns<jupyter_code>df = pd.DataFrame(np.random.randn(8, 3),
                      columns=['A', 'B', 'C'])
df
df.columns = [x.lower() for x in df.columns]
df<jupyter_output><empty_output><jupyter_text>We can think of the Pandas objects (Index, Series, DataFrame) as containers for arrays, which hold the actual data and do the actual computation. To get the actual data inside an Index or Series, use the attribute `.array`.<jupyter_code>df.a.array<jupyter_output><empty_output><jupyter_text>Counting values in SeriesThe `value_counts()` Series method and top-level function computes a histogram of a 1D array of values.<jupyter_code>data = np.random.randint(0, 7, size=50)
data
s = pd.Series(data)
s.value_counts() ## !!<jupyter_output><empty_output><jupyter_text>Similarly, we can get the most frequently occurring value(s) (`mode()`) of the values in a Series or DataFrame.<jupyter_code>s5 = pd.Series([1, 1, 3, 3, 3, 5, 5, 7, 7, 7])
s5.mode()
df5 = pd.DataFrame({"A": np.random.randint(0, 7, size=50),
                       "B": np.random.randint(-10, 15, size=50)})
df5.mode()<jupyter_output><empty_output><jupyter_text>Even though `mode()` can be called on both Series and DataFrame, `value_counts()` can only be used on 1D arrays, therefore, not on DataFrames.  Altering labels  Reindexing ```reindex()``` is the fundamental data alignment method in Pandas. It is used to implement nearly all other features relying on a label-alignment functionality. To reindex means to conform the data to match a given set of labels along a particular axis. This accomplishes several things:* Reorders the existing data to match a new set of labels* Inserts missing value (NA) markers in label locations where no data for that label existed<jupyter_code>s = pd.Series(np.random.randn(5), index=['a', 'b', 'c', 'd', 'e'])
s
s.reindex(['e', 'b', 'f', 'd']) # We can see that we have NaN for the index f. 
                                    # This happens because we didn't have a label f in the original Series.<jupyter_output><empty_output><jupyter_text>With a DataFrame, we can simultaneously reindex the index and columns:<jupyter_code>df = pd.DataFrame({
     'one': pd.Series(np.random.randn(3), index=['a', 'b', 'c']),
     'two': pd.Series(np.random.randn(4), index=['a', 'b', 'c', 'd']),
     'three': pd.Series(np.random.randn(3), index=['b', 'c', 'd'])})
df
df.reindex(index=['c', 'f', 'b'], columns=['three', 'two', 'one']) # reindex both the index and the columns<jupyter_output><empty_output><jupyter_text>We may also use reindex with an axis keyword:<jupyter_code>df.reindex(['c', 'f', 'b'], axis='index')<jupyter_output><empty_output><jupyter_text>Index objects containing the actual axis labels can be shared between objects. So if we have a Series and a DataFrame, the following can be done:<jupyter_code>rs = s.reindex(df.index)
rs<jupyter_output><empty_output><jupyter_text>Dropping labels from an axis A method closely related to reindex is the ```drop()``` function. It removes a set of labels from an axis:<jupyter_code>df
df.drop(['a', 'd'], axis=0)
df.drop(['one'], axis=1)<jupyter_output><empty_output><jupyter_text>Renaming The ```rename()``` method allows us to relabel an axis based on some mapping (a dict or Series) or an arbitrary function.<jupyter_code>s
s.rename(str.upper)<jupyter_output><empty_output><jupyter_text>A ```dict``` or ```Series``` can also be used:<jupyter_code>df.rename(columns={'one': 'foo', 'two': 'bar'},
              index={'a': 'apple', 'b': 'banana', 'd': 'durian'})<jupyter_output><empty_output><jupyter_text>```DataFrame.rename()``` also supports an “axis-style” calling convention, where we specify a single mapper and an axis to apply that mapping to.<jupyter_code>df.rename({'one': 'foo', 'two': 'bar'}, axis='columns')
df.rename({'a': 'apple', 'b': 'banana', 'd': 'durian'}, axis='index')<jupyter_output><empty_output><jupyter_text>```.dt``` and ```.str``` accessors  ```.dt``` ```Series``` has an accessor to succinctly return datetime-like properties for the values of the Series, if it is a datetime/period-like Series. This will return a Series, indexed like an existing Series.<jupyter_code>s = pd.Series(pd.date_range('20130101 09:10:12', periods=4))
s
s.dt.hour
s.dt.second
s.dt.day
s.dt.dayofweek<jupyter_output><empty_output><jupyter_text>We can easily produce timezone-aware transformations:<jupyter_code>stz = s.dt.tz_localize('US/Eastern')
stz
stz.dt.tz # call what timezone it is
s.dt.tz_localize('UTC').dt.tz_convert('US/Eastern')<jupyter_output><empty_output><jupyter_text>```.str``` Series is equipped with a set of **string processing methods** that make it easy to operate on each element of the array. These are accessed via the Series’s str attribute and generally have names matching the equivalent (scalar) built-in string methods<jupyter_code>s = pd.Series(['A', 'B', 'C', 'Aaba', 'Baca', np.nan, 'CABA', 'dog', 'cat'],
                  dtype="string")
s
s.str.lower()<jupyter_output><empty_output><jupyter_text>Using `.str` accessor, we can apply all string functions from standard Python to our Series. SortingThere are three types of sorting in Pandas: 1. Sorting by index labels 2. Sorting by column values 3. Sorting by a combination of both  by index The ```Series.sort_index()``` and ```DataFrame.sort_index()``` methods are used to sort a Pandas object by its index levels.<jupyter_code>df = pd.DataFrame({
        'one': pd.Series(np.random.randn(3), index=['a', 'b', 'c']),
        'two': pd.Series(np.random.randn(4), index=['a', 'b', 'c', 'd']),
        'three': pd.Series(np.random.randn(3), index=['b', 'c', 'd'])})
unsorted_df = df.reindex(index=['a', 'd', 'c', 'b'],
                          columns=['three', 'two', 'one'])
unsorted_df
# Sort DataFrame by index
unsorted_df.sort_index()
unsorted_df.sort_index(ascending=False)
# Sort DataFrame by column names
unsorted_df.sort_index(axis=1)
# Sort Series by index
unsorted_df['three'].sort_index()<jupyter_output><empty_output><jupyter_text>by values The ```Series.sort_values()``` method is used to sort a Series by its values. The ```DataFrame.sort_values()``` method is used to sort a DataFrame by its column or row values.<jupyter_code>df1 = pd.DataFrame({'one': [2, 1, 1, 1],
                        'two': [1, 3, 2, 4],
                        'three': [5, 4, 3, 2]})
df1
# Sort DataFrame by column "two"
df1.sort_values(by='two')
# Sort DataFrame by columns "one" and "two"
df1[['one', 'two', 'three']].sort_values(by=['one', 'two'])<jupyter_output><empty_output><jupyter_text>These methods have a special treatment of NA values via the na_position argument:<jupyter_code>s[2] = np.nan
s.sort_values()
s.sort_values(na_position='first')<jupyter_output><empty_output><jupyter_text>`by` parameter in `sort_values()` method can refer to either columns or index level names.We can use the name of the index to sort by both an index and a column.<jupyter_code># Build MultiIndex
idx = pd.MultiIndex.from_tuples([('a', 1), ('a', 2), ('a', 2),
                                   ('b', 2), ('b', 1), ('b', 1)])
idx.names = ['first', 'second']
# Build DataFrame
df_multi = pd.DataFrame({'A': np.arange(6, 0, -1)},
                            index=idx)
df_multi
# Sort DataFrame by 'second' (index) and 'A' (column)
df_multi.sort_values(by=['second', 'A'])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14178562787321558, 'fraction_numerical': 0.010404064843939027, 'mean_word_length': 4.913447782546495, 'pattern_counts': {'":': 2, '<': 77, '<?xml version=': 0, '>': 77, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '569667', 'license': 'MIT', 'n_tokens_mistral': 2753, 'n_tokens_neox': 2669, 'n_words': 915, 'path': 'Pandas/2 Pandas Basics I.ipynb', 'repo_name': 'robynmundle/data_science', 'size': '8266'}