<jupyter_start><jupyter_text>Hierarchical Indexes on DataFrames and Series<jupyter_code># np.random.seed(12345)
# pd.options.display.max_rows = 20
# plt.rc('figure', figsize=(10, 6))<jupyter_output><empty_output><jupyter_text>Hierarchical Indexing- Hierarchical indexing allows you to have multiple index levels - Despite the name (indexing), it applies to both indexes and columns (i.e., rows or columns)- Each index in a `MutiIndex` Object is referred to as a level<jupyter_code>prog_languages = pd.DataFrame({ "Course":["Python", "Rust", "Python", "Rust"], 
              "Nb_participants": [10, 30, 27, 18]})
prog_languages.groupby("Course").sum()
              
data = pd.Series(np.random.randn(9),
                 index=[['a', 'a', 'a', 'b', 'b', 'c', 'c', 'd', 'd'],
                        [1, 2, 3, 1, 3, 1, 2, 2, 3]])
data
data.index
data[('b', 1)]

data['b', 1]
data['b':'c']

data.loc[['b', 'd']]
data
data.loc[: , 2]
df = pd.DataFrame({"A": np.random.randn(9), "B":np.random.randn(9)}, 
                  index=[['a', 'a', 'a', 'b', 'b', 'c', 'c', 'd', 'd'], [1, 2, 3, 1, 3, 1, 2, 2, 3]])
df 
pd.IndexSlice?
#E.g., data.loc[: , 2] worked on a Series but does not work on a DataFrame

df.loc(axis=0)[pd.IndexSlice[:, 2]]<jupyter_output><empty_output><jupyter_text>Stacked vs. Unstacked Data* Hierarchical indexes represent how the data is often collected * Ex. if variables in a hospital are measured and recorded in different files```python    File 1    patient_ABC LDL 112    patient_ABC HDL 48    patient_CCX  LDL 112    patient_VDM  LDL 112    patient_ABC  VO2 112    patient_CCZ  RER 48    ...```* This format is preferred  when not all the variables are measured in all the patients  * I.e., using columns for the variables would result in a large number of empty cells    Stacking and unstacking a DataFrame* You can convert back and forth between a data frame and a hierarchical index with `stack()` and `unstack()`![](https://www.dropbox.com/s/2u5lk89ga6mkcob/stacking_DF.png?dl=1)<jupyter_code>x
x  = x.stack()
x
x = data.unstack()
x<jupyter_output><empty_output><jupyter_text>Hierarchical indexes on rows and columns* In a `DataFrame`, both axes can have a hierarchical index.* Alway remember the labels are stored under `index` in rows and `columns` for the columns![](https://www.dropbox.com/s/9knv5i5n63nw7l0/index_columns.png?dl=1)<jupyter_code>data = pd.DataFrame(np.arange(12).reshape((4, 3)),
                     index=[['a', 'a', 'b', 'b'], [1, 2, 1, 2]],
                     columns=[['Ohio', 'Ohio', 'Colorado'],
                              ['Green', 'Red', 'Green']])
data
data.index.names = ['key1', 'key2']
data.columns.names = ['state', 'color']
data
print(data.index.get_level_values("key1"))
data.index.get_level_values("key2")

print(data.columns.get_level_values("state"))
data.columns.get_level_values("color")

data['Ohio']
data['Ohio', 'Green']<jupyter_output><empty_output><jupyter_text>Creating and assigning a `MultiIndex` Object* You can create a `MultiIndex` using one of the constructors  * Ex. create form arrays* As long as the dimensions are compatible, you can assign the `MultiIndex` to either the `index` or `column` attribute of a `DataFrame` or a `Series`<jupyter_code>data = pd.DataFrame(np.arange(12).reshape((4, 3)),
                     index=[['a', 'a', 'b', 'b'], [1, 2, 1, 2]])
data.index.names = ["letter", "number"]
data
pd.MultiIndex.from_arrays([['Ohio', 'Ohio', 'Colorado'], ['Green', 'Red', 'Green']],
                       names=['state', 'color'])
data.columns = pd.MultiIndex.from_arrays([['Ohio', 'Ohio', 'Colorado'], ['Green', 'Red', 'Green']],
                       names=['state', 'color'])
data<jupyter_output><empty_output><jupyter_text>Reordering and Sorting Levels- Swap levels takes two levels `numbers` or `names` and returns a new object with the levels interchanged- And as with simple indexes, you can sort a single or a combination of indexes of the `MultiIndex` object<jupyter_code>data.index.names = ['key1', 'key2']
data
data.swaplevel('key1', 'key2')
# You can also pass the level's index, instead of name
data.sort_index(level=1, ascending=False)

data.swaplevel(0, 1).sort_index(level=0, ascending=False)
data.swaplevel(0, 1, 0)
data.swaplevel(0, 1, 1)
data.swaplevel('state', 'color', axis=1)<jupyter_output><empty_output><jupyter_text>Summary Statistics by Level* The summary statistics that we've seen before can also be applied by level* When you apply a function on the level, the remaining levels get `"squashed"`* While you will see such code, this approach is no longer recommended and won't be supported in future versions  * Use `groupby` instead<jupyter_code>data

data.sum(level="state", axis=1)<jupyter_output>/var/folders/6m/188vj34d6nv8qsnxqt6plydw0000gn/T/ipykernel_78605/1335975597.py:1: FutureWarning: Using the level keyword in DataFrame and Series aggregations is deprecated and will be removed in a future version. Use groupby instead. df.sum(level=1) should use df.groupby(level=1).sum().
  data.sum(level="state", axis=1)<jupyter_text>Setting and Resetting and Index* It' common to read in a data frame file and use one of its columns as an index  * Setting a column as an index is done using `set_index()`    * By default, the columns are removed from the `DataFrame`, but you can choose to leave them in   * `reset_index()` does the opposite; moves the index as a new column of the `DataFrame`    * Creates a new index from an `IndexRange()`<jupyter_code>data = pd.DataFrame({'a': range(7), 'b': range(7, 0, -1),
                      'c': ['one', 'one', 'one', 'two', 'two',
                            'two', 'two'],
                      'd': [0, 1, 2, 0, 1, 2, 3]})
data
data2 = data.set_index('b')
data2
data3 = data.set_index(['c', 'd'])
data3
data.set_index(['c', 'd'], drop=False)
data4 = data2.reset_index()
data4
frame4.index<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14988966219657104, 'fraction_numerical': 0.027839076557460533, 'mean_word_length': 4.191189427312775, 'pattern_counts': {'":': 4, '<': 32, '<?xml version=': 0, '>': 32, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '77473', 'license': 'Apache-2.0', 'n_tokens_mistral': 2109, 'n_tokens_neox': 1940, 'n_words': 630, 'path': 'morea/Data_Wrangling_and_Pandas_2/resources/.ipynb_checkpoints/10_hierarchical_indexes-checkpoint.ipynb', 'repo_name': 'mahdi-b/ICS434-Sp2022', 'size': '5891'}